Statistical data feature-based text drawing method and system

Abstract

The invention discloses a text map generation method and system based on statistical data features, and relates to the technical field of advertisement design, and the method comprises the steps: carrying out the statistics of a feature histogram of a target group, and converting the feature histogram into prompt; collecting a training sample; distributing weights for m styles, and converting the m styles into prompt; calculating a loss function; a CLIP model is trained; forming a style feature library of a single style; sending the style feature library of the single style, the content feature library and the first prompt into an SD model for fusion; and inputting the fused image features of each scale into an SD model to generate a result image. According to the method, a new model loss function is designed through a statistical histogram-to-prompt mode, image generation is carried out through fusion of SD convolution features, advertisement style features and advertisement content features, and stable training of the model and accurate generation of the image are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of advertisement design, and in particular to a method and system for generating a text graph based on statistical data features. Background Art

[0002] As advertising costs rise, accurate advertising and improving effective conversion rates are important goals of advertising design. Manual advertising design is labor-intensive and time-consuming, and it is impossible to quickly design different styles and styles to capture the audience's preferences.

[0003] With the development of large-scale text-based image model technology, accurate advertising design has become possible. The text-based image method based on stable difusion (SD) can realize text-to-image generation, but the current mapping result looks stiff and often the generated result is not what you want. This is mainly because the text description and the final image are not completely one-to-one corresponding. The redundancy of images is very high, while the redundancy of language text is very low, so the details of many generated images are not clear enough.

[0004] Therefore, the academic community proposed that controlnet guides the generation of SD images through image conditional features. This method greatly improves the quality of generated images. At present, image generation is still based on manually designed or manually adjusted text image models. In terms of application to advertising generation, this type of model still has the following problems: on the one hand, it is necessary to constantly adjust the prompt and model parameters to achieve the purpose of generating the target image; it is impossible to generate prompts adaptively, and it cannot be applied in some occasions where the speed of advertising image generation is required to be high, and the semi-manual method is costly. On the other hand, the style and content of advertising images need to be designed based on human understanding, rather than based on marketing statistical data, so the designed advertisements are not truly precision marketing. In addition, the generation method based on controlnet requires training a model for each style, which has high training costs and cannot quickly achieve the fusion of multiple styles. Summary of the invention

[0005] The present invention provides a method and system for generating a gram based on statistical data characteristics, so as to overcome at least one technical problem existing in the prior art.

[0006] In one aspect, the present invention provides a method for generating a gram based on statistical data features, comprising:

[0007] Counting the characteristic histogram of the target group, and converting the characteristic histogram into prompts to obtain a first prompt;

[0008] Collecting multiple single-style design images and corresponding style descriptions to obtain multiple training samples; the multiple training samples correspond to n types of style features;

[0009] Select m types of style features from n types of style features and assign weights to the m types of style features;

[0010] Convert the weights corresponding to the m style features into prompts to obtain the second prompt;

[0011] Using the second prompt and the corresponding design image, calculate the loss function;

[0012] Training the CLIP model according to the loss function to obtain a trained CLIP model;

[0013] Using the trained CLIP model, encoding the design images in the training samples to form a style feature library of a single style;

[0014] Send the style feature library, content feature library and the first prompt of a single style into the SD model, fuse the SD convolutional features, content features and style features at the same scale, and obtain the image features after fusion at each scale;

[0015] The image features after fusion at each scale are input into the SD model to generate the result graph.

[0016] Optionally, a feature histogram of the target group is counted and the feature histogram is converted into a prompt, specifically:

[0017] Count the characteristic histogram of the target group and get the style feature statistics {k1, k2, ..., k l}, where l represents the number of style features, k i Indicates the number of people with the corresponding characteristics;

[0018] The feature histogram is normalized to obtain the probability of each feature:

[0019] Arrange the probability of each feature in descending order to obtain the sorted feature probability {q1, q2, ..., q l}, and record the corresponding feature labels;

[0020] Based on the sorted feature probabilities, the feature labels are traversed sequentially to generate a prompt description criterion set containing the feature label sorting index as {the proportion of the "feature label" in the i-th order to the total data is q i}.

[0021] Optionally, the loss function is calculated using the second prompt and the corresponding design image, specifically:

[0022] Send the second prompt to the text encoder of the CLIP model for encoding to obtain text features Tm;

[0023] The design images corresponding to the m style features are sent to the image encoder of the CLIP model for encoding to obtain the corresponding image features {v1, v2, ..., v m};

[0024] According to the weights of m style features, the image features {v1, v2, ..., v m} to fuse and obtain image features Among them, r i represents the weight of the i-th style feature;

[0025] According to the image features and the text features, the loss function is calculated as Loss=loss(Im, Tm).

[0026] Optionally, the design images in the training samples are encoded to form a style feature library of a single style, specifically:

[0027] For each design image with style features, according to the image coding model layer, image features v64, v32, v16, v8 of 64x64, 32x32, 16x16, and 8x8 scales are generated respectively, and the image features v64, v32, v16, and v8 of each scale are stored to form a feature library of a single style.

[0028] Optionally, the style feature library, content feature library and the first prompt of a single style are sent to the SD model, and the SD convolutional features, content features and style features are fused at the same scale, specifically:

[0029] According to the first prompt, the style features of each scale are fused respectively, and the fused style features of each scale are obtained respectively.

[0030] Convolve the fused style features at each scale to obtain the convolved style features;

[0031] At the same scale, the SD convolution features, the content features after convolution, and the style features after convolution are fused to obtain the image features after fusion at each scale: V64 = λ·v64 SD +α·Ik64c+(1-α-λ)·Ic64c, V32=λ·v32 SD+α·Ik32c+(1-α-λ)·Ic32c, V16=λ·v16 SD +α·Ik16c+(1-α-λ)·Ic16c, V8=λ·v8 SD +α·Ik8c+(1-α-λ)·Ic8c; where, v64 SD 、v32 SD 、v16 SD 、v8 SD They are SD convolution features respectively, Ik64c, Ik32c, Ik16c, and Ik8c are style features after convolution respectively, Ic64c, Ic32c, Ic16c, and Ic8c are content features after convolution respectively, and λ and α are hyperparameters.

[0032] On the other hand, the present invention also provides a cultural graph system based on statistical data features, comprising:

[0033] A statistical module, used for counting a characteristic histogram of a target group and converting the characteristic histogram into a prompt to obtain a first prompt;

[0034] A collection module, used to collect multiple design images of a single style and corresponding style descriptions to obtain multiple training samples; the multiple training samples correspond to n types of style features;

[0035] A selection module is used to select m types of style features from n types of style features and assign weights to the m types of style features;

[0036] A conversion module is used to convert the weights corresponding to the m style features into prompts to obtain a second prompt;

[0037] A calculation module, used to calculate a loss function using the second prompt and a corresponding design image;

[0038] A training module, used for training the CLIP model according to the loss function to obtain a trained CLIP model;

[0039] An encoding module, used to encode the design images in the training samples using the trained CLIP model to form a style feature library of a single style;

[0040] A fusion module is used to send a style feature library of a single style, a content feature library and the first prompt into the SD model, fuse the SD convolution features, content features and style features at the same scale, and obtain image features after fusion at each scale;

[0041] The generation module is used to input the image features after fusion at each scale into the SD model to generate the result image.

[0042] Optionally, the statistics module is specifically used to:

[0043] Count the characteristic histogram of the target group and get the style feature statistics {k1, k2, ..., k l}, where l represents the number of style features, k i Indicates the number of people with the corresponding characteristics;

[0044] The feature histogram is normalized to obtain the probability of each feature:

[0045] Arrange the probability of each feature in descending order to obtain the sorted feature probability {q1, q2, ..., q l}, and record the corresponding feature labels;

[0046] Based on the sorted feature probabilities, the feature labels are traversed sequentially to generate a prompt description criterion set containing the feature label sorting index as {the proportion of the "feature label" in the i-th order to the total data is q i}.

[0047] Optionally, the calculation module is specifically used to:

[0048] Send the second prompt to the text encoder of the CLIP model for encoding to obtain text features Tm;

[0049] The design images corresponding to the m style features are sent to the image encoder of the CLIP model for encoding to obtain the corresponding image features {v1, v2, ..., v m};

[0050] According to the weights of m style features, the image features {v1, v2, ..., v m} to fuse and obtain image features Among them, r i represents the weight of the i-th style feature;

[0051] According to the image features and the text features, the loss function is calculated as Loss=loss(Im, Tm).

[0052] Optionally, the encoding module is specifically used to:

[0053] For each design image with style features, according to the image coding model layer, image features v64, v32, v16, v8 of 64x64, 32x32, 16x16, and 8x8 scales are generated respectively, and the image features v64, v32, v16, and v8 of each scale are stored to form a feature library of a single style.

[0054] Optionally, the fusion module is specifically used to:

[0055] According to the first prompt, the style features of each scale are fused respectively, and the fused style features of each scale are obtained respectively.

[0056] Convolve the fused style features at each scale to obtain the convolved style features;

[0057] At the same scale, the SD convolution features, the content features after convolution, and the style features after convolution are fused to obtain the image features after fusion at each scale: V64 = λ·v64 SD +α·Ik64c+(1-α-λ)·Ic64c, V32=λ·v32 SD +α·Ik32c+(1-α-λ)·Ic32c, V16=λ·v16 SD +α·Ik16c+(1-α-λ)·Ic16c, V8=λ·v8 SD +α·Ik8c+(1-α-λ)·Ic8c; where, v64 SD 、v32 SD 、v16 SD 、v8 SD They are SD convolution features respectively, Ik64c, Ik32c, Ik16c, and Ik8c are style features after convolution respectively, Ic64c, Ic32c, Ic16c, and Ic8c are content features after convolution respectively, and λ and α are hyperparameters.

[0058] The innovative features of the embodiments of the present invention include:

[0059] 1. In this embodiment, the characteristics of the target group are converted into prompts by converting statistical histograms into prompts, which are used as inputs of the SD model, thereby guiding the model to generate the required responses so as to better control the generated outputs. This realizes a very concise and accurate prompt automatic generation method, and solves the problem in the prior art that prompts cannot be generated adaptively. This allows large models to be used for production in an automated and rapid manner, which is one of the innovative features of the embodiments of the present invention.

[0060] 2. In this embodiment, based on the method of converting statistical histograms to prompts, a new CLIP model loss function is designed, and the CLIP model is trained according to the loss function, so that the model can not only align on a single style, but also perform computable style alignment under different weight fusions, so that any advertising style can be combined to achieve accurate and colorful advertising image generation, which is one of the innovations of the embodiment of the present invention.

[0061] 3. In this embodiment, image generation is performed by fusing SD convolution features, advertising style features, and advertising content features, thereby achieving stable training of the model and accurate generation of images, which is one of the innovative features of the embodiment of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0063] Figure 1 A flow chart of a method for producing a text map according to an embodiment of the present invention;

[0064] Figure 2 A framework structure diagram of the Wensheng diagram method provided by an embodiment of the present invention;

[0065] Figure 3 A flow chart of converting a feature histogram into a prompt provided by an embodiment of the present invention;

[0066] Figure 4 A flow chart of a loss calculation function provided by an embodiment of the present invention;

[0067] Figure 5 A flow chart of SD model fusion provided by an embodiment of the present invention;

[0068] Figure 6 A schematic diagram of the structure of a culture graph system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0069] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] It should be noted that the terms "including" and "having" and any variations thereof in the embodiments of the present invention and the accompanying drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0071] The embodiment of the present invention discloses a method and system for generating a vinculogram based on statistical data features, which are described in detail below.

[0072] Figure 1 A flow chart of a Wensheng diagram method provided by an embodiment of the present invention, Figure 2 For a framework diagram of the Wensheng diagram method provided by an embodiment of the present invention, please refer to Figure 1 and Figure 2 The embodiment of the present invention provides a method for generating a cultural graph based on statistical data features, including:

[0073] Step 1: Count the feature histogram of the target group and convert the feature histogram into prompt to obtain the first prompt;

[0074] Step 2: Collect multiple single-style design images and corresponding style descriptions to obtain multiple training samples; the multiple training samples correspond to n style features;

[0075] Step 3: Select m types of style features from n types of style features and assign weights to the m types of style features;

[0076] Step 4: Convert the weights corresponding to the m style features into prompts to obtain the second prompt;

[0077] Step 5: Calculate the loss function using the second prompt and the corresponding design image;

[0078] Step 6: Train the CLIP model according to the loss function to obtain the trained CLIP model;

[0079] Step 7: Use the trained CLIP model to encode the design images in the training samples to form a style feature library of a single style;

[0080] Step 8: Send the style feature library, content feature library and the first prompt of a single style into the SD model, fuse the SD convolutional features, content features and style features at the same scale, and obtain the image features after fusion at each scale;

[0081] Step 9: Input the image features fused at each scale into the SD model to generate the result graph.

[0082] Specifically, the text image method based on statistical data features provided by the embodiment of the present invention is divided into three parts. One part is the SD model, which is mainly responsible for generating images; one part is the neural network controlNet, which is responsible for guiding SD generation with advertising content, so that the image content generated by the SD model is consistent with or related to the target; one part is advertising style guidance, which is mainly realized based on the feature library of the CLIP (Contrastive Language-Image Pre-Training, multimodal pre-trained neural network) model.

[0083] Please refer to Figure 1 The present invention firstly calculates the characteristic histogram of the target group through step 1. The characteristic here is the characteristic label related to style that is pre-defined and quantified in the industry, such as gender, age, consumption concept, consumption level and other characteristics. The target group is generally divided into different groups by professional marketing personnel based on the statistical data of the mall operation. After obtaining the characteristic histogram of the target group, the characteristic histogram is used as input, and each data therein is converted into a part of the prompt, and finally the first prompt is obtained.

[0084] Prompt refers to text or instructions that provide input to the model to guide it to generate specific output. It is a text paragraph provided by the user when interacting with the model, which is used to describe the information, answers, text, etc. that the user wants to obtain from the model. The purpose of the prompt is to guide the model to produce the required response in order to better control the generated output. The present invention converts the characteristics of the target group into the first prompt as the input of the SD model by converting the statistical histogram to the prompt, thereby guiding the model to produce the required response in order to better control the generated output. A very concise and accurate prompt automatic generation method is implemented, which solves the problem that the prompt cannot be generated adaptively in the prior art, so that large models can be used for production in an automated and rapid manner.

[0085] In the prior art, the corresponding style can generally only be learned from an image and then transferred to other images. It is impossible to generate any image style or quantitatively control the style. Based on this, the present invention designs advertisements based on big data statistical results, so that any advertisement style can be combined to achieve accurate and colorful advertisement image generation.

[0086] CLIP is an existing text and image alignment and training model. The present invention uses the advertising style design images and style descriptions that have been accumulated in the business for fine-tuning training. First, multiple training samples are collected through step 2. In order to accurately control the fusion of multiple styles, it is necessary to accurately control the image features of a single style. Therefore, the training samples in the present invention are all design images of a single style and the corresponding style descriptions, that is, samples that fuse multiple styles do not participate in training. Here, multiple training samples correspond to n style features, where one style feature can correspond to multiple training samples.

[0087] The loss function of the CLIP model is recorded as loss(I,T), where I is the feature after image encoding and T is the feature after text encoding. The training process of the present invention is similar to that of CLIP, except that the loss function is calculated. The CLIP model calculates the loss for each pair of text and image features. When calculating the loss each time during training, the present invention first selects m types of style features from n types through step 3, where 1<=k<=n, and randomly assigns weights {r1, v2,…, r m}; where r i represents the weight of the i-th style feature. It should be noted that when assigning weights, it is necessary to ensure that the sum of the weights is 1. For example, when there is only one style feature, the weight is 1.

[0088] In step 4, after the weights of the m types of style features are sorted, they are automatically converted into prompts according to the method of converting prompts by statistical histograms to obtain a second prompt. In step 5, the second prompt and the corresponding design image are respectively input into a text encoder and an image encoder to obtain text features and image features, and the loss function can be calculated based on the text features and image features. In the present invention, the prompt encoder is a text encoder of CLIP, which is aligned with the image features during training, and is essentially similar to the mean of the image features corresponding to the text, and is a fuzzy average feature.

[0089] After obtaining the loss function, in step 6, the CLIP model is trained according to the loss function to obtain the trained CLIP model. After the above training, the model can not only align on a single style, but also perform computable style alignment under different weight fusion.

[0090] After the training is completed, the trained CLIP model can be used to create an advertising style feature library in step 7. By encoding the design images in the training samples through the trained CLIP model, a style feature library of a single style can be formed, and each style feature vector in the style feature library corresponds to a style label.

[0091] It should be noted that when making a style feature library, it is necessary to refer to the image encoding model layer. Figure 2 For each style feature, image features v64, v32, v16, v8 of 64x64, 32x32, 16x16, and 8x8 scales are generated respectively, and the image features v64, v32, v16, and v8 of each scale are stored to form a feature library of a single style. For any style feature, these different feature blocks correspond to the same style label, and the style feature library will be used to generate fine feature control of the image.

[0092] After obtaining the first prompt and the style feature library of a single style, SD forward generation of images can be performed. Through step 8, the style feature library of a single style, the content feature library and the first prompt are sent to the SD model, and the SD convolutional features, content features and style features are fused at the same scale to obtain the image features after fusion at each scale.

[0093] The content feature library here is obtained through the advertising content guidance network part, and the advertising content guidance network part adopts the existing controlNet framework and technology. In the present invention, the input of controlNet is the mask of the advertising content, that is, a part of the image is the main content of the advertisement, which is a normal RGB three-channel color image content, and this part of the image content will not be changed in the newly generated advertising image; the rest is a black image area, and the image content can be filled through the style guidance algorithm structure to generate a style consistent with the prompt.

[0094] After obtaining the image features after fusion at each scale, in step 9, the image features after fusion at each scale are input into the SD model to generate a result graph.

[0095] The present invention generates images by fusing SD convolution features, advertisement style features, and advertisement content features, thereby achieving stable training of the model and accurate generation of images.

[0096] Optionally, Figure 3 For a flowchart of converting a feature histogram into a prompt provided by an embodiment of the present invention, please refer to Figure 1 and Figure 3 In step 1, the feature histogram of the target group is counted and converted into prompt, specifically: step 11, the feature histogram of the target group is counted to obtain the style feature statistics {k1, k2, ..., k l}, where l represents the number of style features, k i Indicates the number of people with the corresponding feature; Step 12, normalize the feature histogram to obtain the probability of each feature: Step 13: Arrange the probability of each feature in descending order to obtain the sorted feature probability {q1, q2, ..., q l}, and record the corresponding feature labels at the same time; Step 14, based on the sorted feature probabilities, sequentially traverse the feature labels to generate a prompt description criterion set containing the feature label sorting index as {the proportion of the "feature label" in the i-th order to the total data is q i}.

[0097] Specifically, please refer to Figure 3 The present invention converts the characteristics of the target group into the first prompt by converting the statistical histogram into prompt. First, in step 11, the characteristic histogram of the target group is counted to obtain the statistical data of the style characteristics of the corresponding target group {k1, k2, ..., k l}, where l represents the number of style features, k i Indicates the number of people with the corresponding feature.

[0098] The statistical histogram data does not have a clear overall quantitative feature, but is a relative feature. Therefore, in step 12, the feature histogram is normalized into a probability form to obtain the probability of each feature {p1, p2, ..., p l}, where the probability of the i-th feature is expressed as

[0099] Prompt needs to enable SD to understand the requirements for generating images. It is a feature obtained by compression based on the language model. Therefore, the semantic features of the characters before and after prompt are more obvious. The form of label feature plus overall occupancy probability can only describe the proportion of the feature in the total, and cannot describe its front and back dependencies or sequential relationships. On the other hand, the SD model is more dependent on and sensitive to the index data in the process, such as the noise level index value in the denoising process. Therefore, this embodiment adds a feature label sorting index to prompt, thereby enhancing the explanation ability of prompt.

[0100] First, through step 13, the probability of each feature is arranged in descending order to obtain the sorted feature probability {q1, q2, ..., q l}, and record the corresponding feature labels. Then in step 14, each of the data is sequentially converted into a part of the prompt, and a prompt description criterion containing the feature label sorting index is generated. The description criterion here is {the proportion of the "feature label" in the i-th order to the total data is q i}. By traversing l feature tags, we can get the prompt description criteria set. The variable quantity i, "feature tag", and q in prompt iAll of them can be automatically generated by scripts based on statistical data, so the prompt in the present invention is also automatically generated by scripts.

[0101] Optionally, Figure 4 For a flow chart of calculating the loss function provided by an embodiment of the present invention, please refer to Figure 1 and Figure 4 In step 5, the loss function is calculated using the second prompt and the corresponding design image. Specifically, in step 51, the second prompt is sent to the text encoder of the CLIP model for encoding to obtain the text feature Tm; in step 52, the design image corresponding to the m style features is sent to the image encoder of the CLIP model for encoding to obtain the corresponding image features {v1, v2, ..., v m}; Step 53, the image features {v1, v2, ..., v m} to fuse and obtain image features Among them, r i represents the weight of the i-th style feature; step 54, according to the image features and text features, the loss function is calculated to be Loss=loss(Im, Tm).

[0102] Specifically, please refer to Figure 4 In the present invention, the CLIP model calculates the loss for each pair of text and image features. Each time the loss is calculated, the text features and image features need to be obtained respectively. First, in step 51, the second prompt is sent to the text encoder of the CLIP model for encoding to obtain the text feature Tm. Then in step 52, the design images corresponding to the m style features are sent to the image encoder of the CLIP model for encoding to obtain the corresponding image features {v1, v2, ..., v m}.

[0103] After obtaining the image features of each design image, it is necessary to fuse them according to the weights of various style features. In step 53, the image features {v1, v2, ..., v m} to fuse and obtain image features Among them, r i represents the weight of the i-th style feature. After the image feature Im is fused, in step 54, the loss function Loss = loss (Im, Tm) can be calculated based on the image feature Im and the text feature Tm. It should be noted here that when k = 1, the loss function degenerates into the loss function of CLIP.

[0104] Optionally, Figure 5 For a flow chart of SD model fusion provided by an embodiment of the present invention, please refer to Figure 1 and Figure 5 In step 8, the style feature library, content feature library and the first prompt of a single style are sent to the SD model, and the SD convolutional features, content features and style features are fused at the same scale. Specifically, step 81, according to the first prompt, the style features of each scale are fused respectively, and the fused style features of each scale are obtained respectively. Step 82, convolve the style features after fusion at each scale to obtain the convolved style features; Step 83, at the same scale, fuse the SD convolution features, the content features after convolution, and the style features after convolution to obtain the image features after fusion at each scale, respectively: V64 = λ·v64 SD +α·Ik64c+(1-α-λ)·Ic64c, V32=λ·v32 SD +α·Ik32c+(1-α-λ)·Ic32c, V16=λ·v16 SD +α·Ik16c+(1-α-λ)·Ic16c, V8=λ·v8 SD +α·Ik8c+(1-α-λ)·Ic8c; where, v64 SD 、v32 SD 、v16 SD 、v8 SD They are SD convolution features respectively, Ik16c, Ik32c, Ik16c, and Ik8c are style features after convolution respectively, Ic64c, Ic32c, Ic16c, and Ic8c are content features after convolution respectively, and λ and α are hyperparameters.

[0105] Specifically, please refer to Figure 5 After obtaining the first prompt and the style feature library of a single style, the SD can be used to generate the image forward. In step 81 of this embodiment, according to the first prompt, the style features of each scale are fused respectively, and the style features after fusion of each scale are obtained respectively.

[0106]

[0107] After obtaining the style features after fusion of each scale, in step 82, the style features after fusion of each scale are convolved to obtain the convolved style features. Here, the zero convolution convolution method can be used, and the style features corresponding to the SD decoding layer scale can be obtained through zero convolution convolution.

[0108] After obtaining the style features after convolution at each scale, the SD convolution features, the content features after convolution, and the style features after convolution can be fused at the same scale through step 83 to obtain the fused image features at the corresponding scales: V64 = λ·v64 SD +α·Ik64c+(1-α-λ)·Ic64c, V32=λ·v32 SD +α·Ik32c+(1-α-λ)·Ic32c, V16=λ·v16 SD +α·Ik16c+(1-α-λ)·Ic16c, V8=λ·v8 SD +α·Ik8c+(1-α-λ)·Ic8c. Among them, v64 SD 、v32 SD 、v16 SD 、v8 SD They are SD convolution features, Ik64c, Ik32c, Ik16c, and Ik8c are style features after convolution, Ic64c, Ic32c, Ic16c, and Ic8c are content features after convolution, and λ and α are hyperparameters, which are mainly used to adjust the proportion of content features and style features in the generated image.

[0109] Based on the same inventive concept, the present invention also provides a cultural graph system based on statistical data features. Figure 6 A structural diagram of the Wenshengtu system provided by the embodiment of the present invention, please refer to Figure 6 The present invention provides a cultural graph system 100 based on statistical data features, including:

[0110] A statistical module 101 is used to count the characteristic histogram of the target group and convert the characteristic histogram into a prompt to obtain a first prompt;

[0111] The collection module 102 is used to collect multiple design images of a single style and corresponding style descriptions to obtain multiple training samples; the multiple training samples correspond to n types of style features;

[0112] A selection module 103 is used to select m types of style features from n types of style features and assign weights to the m types of style features;

[0113] A conversion module 104, configured to convert the weights corresponding to the m style features into prompts to obtain a second prompt;

[0114] A calculation module 105, used to calculate a loss function using the second prompt and the corresponding design image;

[0115] A training module 106, used to train the CLIP model according to the loss function to obtain a trained CLIP model;

[0116] The encoding module 107 is used to encode the design images in the training samples using the trained CLIP model to form a style feature library of a single style;

[0117] A fusion module 108 is used to send a style feature library of a single style, a content feature library and a first prompt into the SD model, and fuse the SD convolution features, content features and style features at the same scale to obtain image features after fusion at each scale;

[0118] The generation module 109 is used to input the image features after fusion of each scale into the SD model to generate a result image.

[0119] Specifically, please refer to Figure 6 The text-based image system based on statistical data features provided by the present invention is divided into three parts. One part is the SD model, which is mainly responsible for generating images; one part is the neural network controlNet, which is responsible for guiding SD generation with advertising content, so that the image content generated by the SD model is consistent with or related to the target; one part is advertising style guidance, which is mainly realized based on the feature library of CLIP (Contrastive Language-Image Pre-Training, multimodal pre-training neural network) model.

[0120] Please refer to Figure 6 The present invention firstly uses the statistical module 101 to count the characteristic histogram of the target group. The characteristic here is a characteristic label related to style that is pre-defined and quantified in the industry, such as gender, age, consumption concept, consumption level and other characteristics. The target group is generally divided into different groups by professional marketing personnel based on the statistical data of the mall operation. After obtaining the characteristic histogram of the target group, the characteristic histogram is used as input, and each data therein is converted into a part of the prompt, and finally the first prompt is obtained.

[0121] Prompt refers to text or instructions that provide input to the model to guide it to generate specific output. It is a text paragraph provided by the user when interacting with the model, which is used to describe the information, answers, text, etc. that the user wants to obtain from the model. The purpose of the prompt is to guide the model to produce the required response in order to better control the generated output. The present invention converts the characteristics of the target group into the first prompt as the input of the SD model by converting the statistical histogram to the prompt, thereby guiding the model to produce the required response in order to better control the generated output. A very concise and accurate prompt automatic generation method is implemented, which solves the problem that the prompt cannot be generated adaptively in the prior art, so that large models can be used for production in an automated and rapid manner.

[0122] In the prior art, the corresponding style can generally only be learned from an image and then transferred to other images. It is impossible to generate any image style or quantitatively control the style. Based on this, the present invention designs advertisements based on big data statistical results, so that any advertisement style can be combined to achieve accurate and colorful advertisement image generation.

[0123] CLIP is an existing text and image alignment and training model. The present invention uses the advertising style design images and style descriptions that have been accumulated in the business for fine-tuning training. First, multiple training samples are collected through the acquisition module 102. In order to accurately control the fusion of multiple styles, it is necessary to accurately control the image features of a single style. Therefore, the training samples in the present invention are all design images of a single style and corresponding style descriptions, that is, samples that fuse multiple styles do not participate in training. Here, multiple training samples correspond to n style features, where one style feature can correspond to multiple training samples.

[0124] The loss function of the CLIP model is recorded as loss(I,T), where I is the feature after image encoding and T is the feature after text encoding. The training process of the present invention is similar to that of CLIP, except that the loss function is calculated. The CLIP model calculates the loss for each pair of text and image features. When calculating the loss each time during training, the present invention first selects m types of style features from n types through the selection module 103, where 1<=k<=n, and randomly assigns weights {r1, v2, ..., r m}; where r i represents the weight of the i-th style feature. It should be noted that when assigning weights, it is necessary to ensure that the sum of the weights is 1. For example, when there is only one style feature, the weight is 1.

[0125] After sorting the weights of the m style features, the conversion module 104 automatically converts the prompt into a prompt according to the statistical histogram conversion method to obtain a second prompt. The calculation module 105 inputs the second prompt and the corresponding design image into the text encoder and the image encoder respectively to obtain text features and image features. The loss function can be calculated based on the text features and the image features. In the present invention, the prompt encoder is a CLIP text encoder, which is aligned with the image features during the training process, which is essentially similar to the mean of the image features corresponding to the text, and is a fuzzy average feature.

[0126] After obtaining the loss function, the training module 106 trains the CLIP model according to the loss function to obtain the trained CLIP model. After the above training, the model can not only perform alignment on a single style, but also perform computable style alignment under different weight fusion.

[0127] After the training is completed, the encoding module 107 uses the trained CLIP model to create an advertisement style feature library. By encoding the design images in the training samples through the trained CLIP model, a style feature library of a single style can be formed, and each style feature vector in the style feature library corresponds to a style label.

[0128] It should be noted that when making a style feature library, it is necessary to refer to the image encoding model layer. Figure 2 For each style feature, image features v64, v32, v16, v8 of 64x64, 32x32, 16x16, and 8x8 scales are generated respectively, and the image features v64, v32, v16, and v8 of each scale are stored to form a feature library of a single style. For any style feature, these different feature blocks correspond to the same style label, and the style feature library will be used to generate fine feature control of the image.

[0129] After obtaining the first prompt and the style feature library of a single style, SD forward generation of images can be performed. The style feature library of a single style, the content feature library and the first prompt are sent to the SD model through the fusion module 108, and the SD convolution features, content features and style features are fused at the same scale to obtain image features after fusion at each scale.

[0130] The content feature library here is obtained through the advertising content guidance network part, and the advertising content guidance network part adopts the existing controlNet framework and technology. In the present invention, the input of controlNet is the mask of the advertising content, that is, a part of the image is the main content of the advertisement, which is a normal RGB three-channel color image content, and this part of the image content will not be changed in the newly generated advertising image; the rest is a black image area, and the image content can be filled through the style guidance algorithm structure to generate a style consistent with the prompt.

[0131] After obtaining the image features after fusion of each scale, the generation module 109 inputs the image features after fusion of each scale into the SD model to generate a result graph.

[0132] The present invention generates images by fusing SD convolution features, advertisement style features, and advertisement content features, thereby achieving stable training of the model and accurate generation of images.

[0133] Optionally, the statistical module 101 is specifically used to: count the characteristic histogram of the target group to obtain the style characteristic statistical data {k1, k2, ..., k l}, where l represents the number of style features, k iRepresents the number of people with the corresponding feature; normalize the feature histogram to get the probability of each feature: Arrange the probability of each feature in descending order to obtain the sorted feature probability {q1, q2, ..., q l}, and record the corresponding feature labels at the same time; based on the sorted feature probabilities, traverse the feature labels in order to generate a prompt description criterion set containing the feature label sorting index as {the proportion of the "feature label" in the i-th order to the total data is q i}.

[0134] Specifically, please refer to Figure 6 The present invention converts the characteristics of the target group into the first prompt by converting the statistical histogram to prompt. First, the characteristic histogram of the target group is counted to obtain the statistical data of the style characteristics of the corresponding target group {k1, k2, ..., k l}, where l represents the number of style features, k i Indicates the number of people with the corresponding feature.

[0135] The statistical histogram data does not have a clear overall quantitative feature, but is a relative feature. Therefore, this embodiment normalizes the feature histogram into a probability form to obtain the probability of each feature {p1, p2, ..., p l}, where the probability of the i-th feature is expressed as

[0136] Prompt needs to enable SD to understand the requirements for generating images. It is a feature obtained by compression based on the language model. Therefore, the semantic features of the characters before and after prompt are more obvious. The form of label feature plus overall occupancy probability can only describe the proportion of the feature in the total, and cannot describe its front and back dependencies or sequential relationships. On the other hand, the SD model is more dependent on and sensitive to the index data in the process, such as the noise level index value in the denoising process. Therefore, this embodiment adds a feature label sorting index to prompt, thereby enhancing the explanation ability of prompt.

[0137] First, arrange the probability of each feature in descending order to obtain the sorted feature probability {q1, q2, ..., q l}, and record the corresponding feature labels. Then, each of the data is converted into a part of the prompt in sequence, and a prompt description criterion containing the feature label sorting index is generated. The description criterion here is {the proportion of the "feature label" in the i-th order to the total data is q i}. By traversing l feature tags, we can get the prompt description criteria set. The variable quantity i, "feature tag", and q in prompt iAll of them can be automatically generated by scripts based on statistical data, so the prompt in the present invention is also automatically generated by scripts.

[0138] Optionally, the calculation module 105 is specifically used to: send the second prompt to the text encoder of the CLIP model for encoding to obtain the text feature Tm; send the design image corresponding to the m style features to the image encoder of the CLIP model for encoding to obtain the corresponding image features {v1, v2, ..., v m}; According to the weights of m style features, image features {v1, v2, …, v m} to fuse and obtain image features Among them, r i Represents the weight of the style feature in the i-th image; based on the image features and text features, the loss function is calculated as Loss=loss(Im, Tm).

[0139] Specifically, please refer to Figure 6 In the present invention, the CLIP model calculates the loss for each pair of text and image features. Each time the loss is calculated, the text features and image features need to be obtained respectively. First, the second prompt is sent to the text encoder of the CLIP model for encoding to obtain the text feature Tm. Then, the design images corresponding to the m style features are sent to the image encoder of the CLIP model for encoding to obtain the corresponding image features {v1, v2, ..., v m}.

[0140] After obtaining the image features of each design image, it is necessary to weight the image features {v1, v2, …, v m} to fuse and obtain image features Among them, r i Represents the weight of the i-th style feature. After the image feature Im is fused, the loss function can be calculated based on the image feature Im and the text feature Tm as Loss = loss (Im, Tm). It should be noted here that when k = 1, the loss function degenerates into the loss function of CLIP.

[0141] Optionally, the fusion module 108 is specifically used to: according to the first prompt, fuse the style features of each scale respectively, and obtain the style features of each scale after fusion respectively.

[0142] The style features after fusion at each scale are convolved to obtain the convolved style features; at the same scale, the SD convolution features, the content features after convolution, and the style features after convolution are fused to obtain the image features after fusion at each scale: V64 = λ·v64SD +α·Ik64c+(1-α-λ)·Ic64c, V32=λ·v32 SD +α·Ik32c+(1-α-λ)·Ic32c, V16=λ·v16 SD +α·Ik16c+(1-α-λ)·Ic16c, V8=λ·v8 SD +α·Ik8c+(1-α-λ)·Ic8c; where, v64 SD 、v32 SD 、v16 SD 、v8 SD They are SD convolution features respectively, Ik16c, Ik32c, Ik16c, and Ik8c are style features after convolution respectively, Ic64c, Ic32c, Ic16c, and Ic8c are content features after convolution respectively, and λ and α are hyperparameters.

[0143] Specifically, please refer to Figure 6 After obtaining the first prompt and the style feature library of a single style, the SD can be used to generate the image forward. In this embodiment, the style features of each scale are fused according to the first prompt, and the fused style features of each scale are obtained respectively.

[0144]

[0145] After obtaining the style features after fusion of each scale, the style features after fusion of each scale are convolved to obtain the convolved style features. Here, the zero convolution convolution method can be used, and the style features corresponding to the SD decoding layer scale can be obtained after zero convolution convolution.

[0146] After obtaining the style features after convolution at each scale, the SD convolution features, the content features after convolution, and the style features after convolution can be fused at the same scale to obtain the fused image features at the corresponding scales: V64 = λ·v64 SD +α·Ik64c+(1-α-λ)·Ic64c, V32=λ·v32 SD +α·Ik32c+(1-α-λ)·Ic32c, V16=λ·v16 SD +α·Ik16c+(1-α-λ)·Ic16c, V8=λ·v8 SD +α·Ik8c+(1-α-λ)·Ic8c. Among them, v64 SD 、v32 SD 、v16 SD 、v8 SDThey are SD convolution features, Ik64c, Ik32c, Ik16c, and Ik8c are style features after convolution, Ic64c, Ic32c, Ic16c, and Ic8c are content features after convolution, and λ and α are hyperparameters, which are mainly used to adjust the proportion of content features and style features in the generated image.

[0147] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0148] Those skilled in the art can understand that the modules in the device in the embodiment can be distributed in the device in the embodiment according to the description of the embodiment, or can be changed accordingly and located in one or more devices different from the embodiment. The modules in the above embodiment can be combined into one module, or can be further divided into multiple sub-modules.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method of using a Vincent graph based on statistical data features, characterized in that: include: Counting the characteristic histogram of the target group, and converting the characteristic histogram into prompts to obtain a first prompt; Collecting multiple single-style design images and corresponding style descriptions to obtain multiple training samples; the multiple training samples correspond to n types of style features; Select m types of style features from n types of style features and assign weights to the m types of style features; Convert the weights corresponding to the m style features into prompts to obtain the second prompt; Using the second prompt and the corresponding design image, calculate the loss function; Training the CLIP model according to the loss function to obtain a trained CLIP model; Using the trained CLIP model, encoding the design images in the training samples to form a style feature library of a single style; Send the style feature library, content feature library and the first prompt of a single style into the SD model, fuse the SD convolutional features, content features and style features at the same scale, and obtain the image features after fusion at each scale; The image features after fusion at each scale are input into the SD model to generate the result graph.

2. The method of using a Vincent graph based on statistical data features according to claim 1, characterized in that: The characteristic histogram of the target group is counted and converted into prompt, specifically: Count the characteristic histogram of the target group and get the style feature statistics {k1, k2, ..., k l }, where l represents the number of style features, k i Indicates the number of people with the corresponding characteristics; The feature histogram is normalized to obtain the probability of each feature: Arrange the probability of each feature in descending order to obtain the sorted feature probability {q1, q2, ..., q l }, and record the corresponding feature labels; Based on the sorted feature probabilities, the feature labels are traversed sequentially to generate a prompt description criterion set containing the feature label sorting index as {the proportion of the "feature label" in the i-th order to the total data is q i }.

3. The method of using a Vincent graph based on statistical data features according to claim 1, characterized in that: Using the second prompt and the corresponding design image, the loss function is calculated, specifically: Send the second prompt to the text encoder of the CLIP model for encoding to obtain text features Tm; The design images corresponding to the m style features are sent to the image encoder of the CLIP model for encoding to obtain the corresponding image features {v1, v2, ..., v m }; According to the weights of m style features, the image features {v1, v2, ..., v m } to fuse and obtain image features Among them, r i represents the weight of the i-th style feature; According to the image features and the text features, the loss function is calculated as Loss=loss(Im, Tm).

4. The method of using a Vincent graph based on statistical data features according to claim 2, characterized in that: The design images in the training samples are encoded to form a style feature library of a single style, specifically: For each design image with style features, according to the image coding model layer, image features v64, v32, v16, v8 of 64x64, 32x32, 16x16, and 8x8 scales are generated respectively, and the image features v64, v32, v16, and v8 of each scale are stored to form a feature library of a single style.

5. The method of using a Vincent graph based on statistical data features according to claim 4, characterized in that: The style feature library, content feature library and the first prompt of a single style are sent to the SD model, and the SD convolutional features, content features and style features are fused at the same scale, specifically: According to the first prompt, the style features of each scale are fused respectively, and the fused style features of each scale are obtained respectively. Convolve the fused style features at each scale to obtain the convolved style features; At the same scale, the SD convolution features, the content features after convolution, and the style features after convolution are fused to obtain the image features after fusion at each scale: V64 = λ·v64 SD +α·Ik64c+(1-α-λ)·Ic64c, V32=λ·v32 SD +α·Ik32c+(1-α-λ)·Ic32c, V16=λ·v16 SD +α·Ik16c+(1-α-λ)·Ic16c, V8=λ·v8 SD +α·Ik8c+(1-α-λ)·Ic8c; where, v64 SD 、v32 SD 、v16 SD 、v8 SD They are SD convolution features respectively, Ik64c, Ik32c, Ik16c, and Ik8c are style features after convolution respectively, Ic64c, Ic32c, Ic16c, and Ic8c are content features after convolution respectively, and λ and α are hyperparameters.

6. A cultural graph system based on statistical data features, characterized in that: include: A statistical module, used for counting a characteristic histogram of a target group and converting the characteristic histogram into a prompt to obtain a first prompt; A collection module, used to collect multiple design images of a single style and corresponding style descriptions to obtain multiple training samples; the multiple training samples correspond to n types of style features; A selection module is used to select m types of style features from n types of style features and assign weights to the m types of style features; A conversion module is used to convert the weights corresponding to the m style features into prompts to obtain a second prompt; A calculation module, used to calculate a loss function using the second prompt and a corresponding design image; A training module, used for training the CLIP model according to the loss function to obtain a trained CLIP model; An encoding module, used to encode the design images in the training samples using the trained CLIP model to form a style feature library of a single style; A fusion module is used to send a style feature library of a single style, a content feature library and the first prompt into the SD model, fuse the SD convolution features, content features and style features at the same scale, and obtain image features after fusion at each scale; The generation module is used to input the image features after fusion at each scale into the SD model to generate the result image.

7. The cultural graph system based on statistical data features according to claim 6 is characterized in that: The statistical module is specifically used for: Count the characteristic histogram of the target group and get the style feature statistics {k1, k2, ..., k l }, where l represents the number of style features, k i Indicates the number of people with the corresponding characteristics; The feature histogram is normalized to obtain the probability of each feature: Arrange the probability of each feature in descending order to obtain the sorted feature probability {q1, q2, ..., q l }, and record the corresponding feature labels; Based on the sorted feature probabilities, the feature labels are traversed sequentially to generate a prompt description criterion set containing the feature label sorting index as {the proportion of the "feature label" in the i-th order to the total data is q i }.

8. The cultural graph system based on statistical data features according to claim 6, characterized in that: The computing module is specifically used for: Send the second prompt to the text encoder of the CLIP model for encoding to obtain text features Tm; The design images corresponding to the m style features are sent to the image encoder of the CLIP model for encoding to obtain the corresponding image features {v1, v2, ..., v m }; According to the weights of m style features, the image features {v1, v2, ..., v m } to fuse and obtain image features Among them, r i represents the weight of the i-th style feature; According to the image features and the text features, the loss function is calculated as Loss=loss(Im, Tm).

9. The cultural graph system based on statistical data features according to claim 7, characterized in that: The encoding module is specifically used for: For each design image with style features, according to the image coding model layer, image features v64, v32, v16, v8 of 64x64, 32x32, 16x16, and 8x8 scales are generated respectively, and the image features v64, v32, v16, and v8 of each scale are stored to form a feature library of a single style.

10. The cultural graph system based on statistical data features according to claim 9, characterized in that: The fusion module is specifically used for: According to the first prompt, the style features of each scale are fused respectively, and the fused style features of each scale are obtained respectively. Convolve the fused style features at each scale to obtain the convolved style features; At the same scale, the SD convolution features, the content features after convolution, and the style features after convolution are fused to obtain the image features after fusion at each scale: V64 = λ·v64 SD +α·Ik64c+(1-α-λ)·Ic64c, V32=λ·v32 SD +α·Ik32c+(1-α-λ)·Ic32c, V16=λ·v16 SD +α·Ik16c+(1-α-λ)·Ic16c, V8=λ·v8 SD +α·Ik8c+(1-α-λ)·Ic8c; where, v64 SD 、v32 SD 、v16 SD 、v8 SD They are SD convolution features respectively, Ik64c, Ik32c, Ik16c, and Ik8c are style features after convolution respectively, Ic64c, Ic32c, Ic16c, and Ic8c are content features after convolution respectively, and λ and α are hyperparameters.

Images