Painting generation method and device, equipment and storage medium

Abstract

The invention discloses a picture generation method and device, equipment and a storage medium, and relates to the technical field of image generation. According to the method, the paintings are labeled, so that each painting has the corresponding feature tag, and the paintings and the corresponding tags are input into the model, so that the painting model has semantic knowledge, the relationship between the paintings and semantic information in the text can be fully mined, and accurate paintings can be generated based on the semantic text. By training the adjustment model and combining the trained optimal adjustment model and the initial painting model, the training of the initial painting model is realized, the training efficiency is improved, and the required calculation amount of the training model is reduced, so that the redundancy of the painting model is reduced, and the efficient generation of the painting of the painting model is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of image generation, and in particular to a method, device, equipment and storage medium for generating a painting. Background Art

[0002] As a carrier of art, traditional paintings have extremely high value. With the rapid development of deep learning and neural network technology, it has gradually become common to use deep learning and neural network technology to repair and generate traditional paintings.

[0003] However, in the process of generating traditional paintings using deep learning and neural network technology in the prior art, usually only how to generate paintings is considered, but the impact of user needs on painting generation is not considered, and the accuracy and performance of painting generation are also ignored, resulting in unsatisfactory traditional paintings. Therefore, a new painting generation method is urgently needed to solve the defects of the prior art. Summary of the invention

[0004] The present invention provides a method and device for generating a painting to solve the technical problem of how to generate a painting accurately and efficiently.

[0005] In order to solve the above technical problems, an embodiment of the present invention provides a method for generating a painting, comprising:

[0006] According to a preset labeling method, in combination with a preset number of initial paintings, a number of third paintings and a third label set corresponding to each third painting are obtained;

[0007] Inputting each of the third paintings and the corresponding third label set into a preset initial painting model in sequence, and training a preset adjustment model in combination with the initial painting model to obtain an optimal adjustment model;

[0008] Combining the optimal adjustment model and the initial painting model to obtain a painting generation model;

[0009] A painting generation word is obtained, and the painting generation word is input into the painting generation model to generate a painting.

[0010] It can be understood that, compared with the prior art, the present invention annotates paintings so that each painting has its own corresponding feature label, and inputs the paintings and corresponding labels into the model, so that the painting model has semantic knowledge, and can fully explore the relationship between the painting and the semantic information in the text, so that accurate paintings can be generated based on the semantic text. By training the adjustment model and merging the trained optimal adjustment model with the initial painting model, the training of the initial painting model is realized, the training efficiency is improved, and the required calculation amount of the training model is reduced, thereby reducing the redundancy of the painting model and realizing the efficient generation of paintings by the painting model.

[0011] As a preferred solution, the method of obtaining a plurality of third paintings and a third label set corresponding to each third painting according to a preset labeling method and in combination with a plurality of preset initial paintings specifically includes:

[0012] Get several preset initial paintings;

[0013] Screening a number of the initial paintings according to the clarity of each of the initial paintings to obtain a number of second paintings;

[0014] Each of the second paintings is labeled according to a preset labeling method to obtain a plurality of third paintings and a third label set corresponding to each third painting.

[0015] This preferred solution annotates the paintings so that each painting has its own corresponding feature label, and inputs the paintings and the corresponding labels into the model, so that the painting model has semantic knowledge and can fully mine the relationship between the painting and the semantic information in the text, so as to generate accurate paintings based on the semantic text.

[0016] As a preferred solution, the method of screening a plurality of the initial paintings according to the clarity of each of the initial paintings to obtain a plurality of first paintings specifically includes:

[0017] Calculating the clarity of each of the initial paintings according to a preset image clarity evaluation algorithm;

[0018] Get the preset clarity threshold;

[0019] Screening each of the initial paintings according to the clarity threshold to obtain a plurality of first paintings;

[0020] Duplicate pictures in the first pictures are eliminated to obtain a number of second pictures.

[0021] This preferred solution ensures that the paintings used for subsequent annotation and model training meet basic requirements by judging the clarity of each initial painting, so that each painting has its own corresponding feature label. The painting and the corresponding label are input into the model, so that the painting model has semantic knowledge and can fully mine the relationship between the painting and the semantic information in the text, so as to generate accurate paintings based on semantic text.

[0022] As a preferred solution, the step of labeling each of the second paintings according to a preset labeling method to obtain a plurality of third paintings and a third label set corresponding to each third painting specifically includes:

[0023] Annotate each of the second paintings according to a preset annotation method to obtain a second label set corresponding to each of the second paintings;

[0024] Cropping each of the second paintings according to a preset cropping standard to obtain a third painting corresponding to each of the second paintings;

[0025] According to the second label set corresponding to each second painting, a third label set corresponding to each third painting is obtained.

[0026] This game solution labels the paintings so that each painting has its own corresponding feature label, and crops the paintings so that each painting has a uniform size, providing a basis for subsequent model training.

[0027] As a preferred solution, the step of labeling each second painting according to a preset labeling method to obtain a second label set corresponding to each second painting specifically includes:

[0028] Get the preset labeler;

[0029] Inputting each of the second paintings into the labeler to obtain an initial label set corresponding to each of the second paintings;

[0030] According to a preset deletion standard, the tags in each initial tag set are deleted to obtain a plurality of first tag sets;

[0031] The preset keywords are added to each of the first tag sets in sequence to obtain a plurality of second tag sets corresponding to each second painting.

[0032] This preferred solution labels the paintings so that each painting has its own corresponding feature label, and crops the paintings so that each painting has a uniform size, thereby providing a basis for subsequent model training.

[0033] As a preferred solution, the step of sequentially inputting each of the third paintings and the corresponding third label set into a preset initial painting model, training a preset adjustment model in combination with the initial painting model, and obtaining an optimal adjustment model specifically includes:

[0034] Obtain a preset adjustment model and a preset initial painting model;

[0035] Adding the low-rank matrix of the adjusted model to the initial painting model;

[0036] Inputting each of the third paintings and the corresponding third label sets into the initial painting model in sequence;

[0037] According to a preset training method, the low-rank matrix of the adjustment model is iteratively trained in combination with the initial painting model;

[0038] According to a preset evaluation standard, the training result of the adjustment model is evaluated to obtain an evaluation result;

[0039] The hyperparameters of the adjustment model are adjusted according to the evaluation results until the evaluation results meet the preset requirements, and the iterative training of the adjustment model is completed to obtain the optimal adjustment model.

[0040] This preferred solution fully explores the relationship between the painting and the semantic information in the text through training and adjustment of the model, so that accurate paintings can be generated based on the semantic text, and the training of the initial painting model is realized, the training efficiency is improved, and the required computing power of the training model is reduced, thereby reducing the redundancy of the painting model and realizing the efficient generation of paintings by the painting model.

[0041] As a preferred solution, the iterative training of the low-rank matrix of the adjustment model in combination with the initial painting model according to a preset training method specifically includes:

[0042] Changing the rank of the low-rank matrix in turn, and evaluating the impact score of each rank on the low-rank matrix;

[0043] According to the impact score, an optimal rank of the low-rank matrix is ​​obtained;

[0044] The parameters of the weight matrix in the initial painting model are fixed, and the parameters of the low-rank matrix are iteratively updated in combination with the optimal rank.

[0045] This preferred solution fully explores the relationship between the painting and the semantic information in the text through training and adjustment of the model, so that accurate paintings can be generated based on the semantic text, and the training of the initial painting model is realized, the training efficiency is improved, and the required computing power of the training model is reduced, thereby reducing the redundancy of the painting model and realizing the efficient generation of paintings by the painting model.

[0046] Accordingly, an embodiment of the present invention provides a painting generation device, comprising: a labeling module, a model training module, a model merging module and a painting generation module;

[0047] The labeling module is used to obtain a plurality of third paintings and a third label set corresponding to each third painting according to a preset labeling method and in combination with a plurality of preset initial paintings;

[0048] The model training module is used to input each of the third paintings and the corresponding third label set into a preset initial painting model in sequence, and train a preset adjustment model in combination with the initial painting model to obtain an optimal adjustment model;

[0049] The model merging module is used to merge the optimal adjustment model and the initial painting model to obtain a painting generation model;

[0050] The painting generation module is used to obtain painting keywords, input the painting keywords into the painting generation model, and obtain the painting output by the painting generation model.

[0051] Accordingly, an embodiment of the present invention provides a terminal device, including:

[0052] one or more processors;

[0053] A memory, coupled to the processor, for storing one or more programs;

[0054] When the one or more programs are executed by the one or more processors, the one or more processors implement the method for generating a painting as described above.

[0055] Accordingly, an embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the method for generating a painting as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 : A flowchart of a method for generating a painting provided by an embodiment of the present invention;

[0057] Figure 2 : A flowchart of the steps of painting screening provided by an embodiment of the present invention;

[0058] Figure 3 : A processing step diagram of a painting label provided by an embodiment of the present invention;

[0059] Figure 4 : A flowchart of an implementation of a painting generation model provided by an embodiment of the present invention;

[0060] Figure 5 : A structural diagram of an SDXL model provided by an embodiment of the present invention;

[0061] Figure 6 : A schematic diagram of a LORA fine-tuning model provided by an embodiment of the present invention;

[0062] Figure 7 :A flow chart of a LORA fine-tuning model provided by an embodiment of the present invention;

[0063] Figure 8 : A painting generation graph of a painting generation model provided by an embodiment of the present invention;

[0064] Fig. 9 : A flow chart of generating a Song Dynasty painting provided by an embodiment of the present invention;

[0065] Fig.10 : A schematic diagram of the structure of a device for generating a painting provided by an embodiment of the present invention;

[0066] Among them, 201: labeling module; 202: model training module; 203: model merging module; 204: painting generation module. DETAILED DESCRIPTION

[0067] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] It should be noted that the paintings described in the embodiments of the present invention refer to Song Dynasty paintings, and the painting generation model refers to a Song Dynasty painting generation model.

[0069] Embodiment 1

[0070] Please refer to Figure 1 , is a flowchart of a method for generating a painting provided in an embodiment of the present invention, including steps S101-S104.

[0071] Step S101: according to a preset labeling method, in combination with a preset number of initial paintings, a number of third paintings and a third label set corresponding to each third painting are obtained.

[0072] In this embodiment, the method of obtaining a plurality of third paintings and a third label set corresponding to each third painting according to a preset labeling method and in combination with a plurality of preset initial paintings specifically includes:

[0073] Get several preset initial paintings;

[0074] Screening a number of the initial paintings according to the clarity of each of the initial paintings to obtain a number of second paintings;

[0075] Each of the second paintings is labeled according to a preset labeling method to obtain a plurality of third paintings and a third label set corresponding to each third painting.

[0076] It should be noted that the initial paintings described in the present invention refer to Song Dynasty paintings, wherein the required number of each category of Song Dynasty paintings ranges from 100 to 200; the initial paintings obtained should avoid painting homogeneity, and the painting homogeneity refers to paintings with similar composition, details and characters; the initial paintings should be selected from open source Song Dynasty paintings.

[0077] In this embodiment, the screening of the initial paintings according to the definition of each initial painting to obtain the second paintings specifically includes:

[0078] Calculating the clarity of each of the initial paintings according to a preset image clarity evaluation algorithm;

[0079] Get the preset clarity threshold;

[0080] Screening each of the initial paintings according to the clarity threshold to obtain a plurality of first paintings;

[0081] Duplicate pictures in the first pictures are eliminated to obtain a number of second pictures.

[0082] In an alternative embodiment, please refer to Figure 2 , which is a flow chart of the steps of painting screening provided by an embodiment of the present invention, wherein non-open source paintings are removed from the initial paintings, then unclear paintings are removed, and finally duplicate paintings are removed, so as to retain 100 to 200 representative paintings.

[0083] It should be noted that the preset image clarity evaluation algorithm described in the embodiment of the present invention includes: algorithms for image clarity evaluation such as the Tenengrad gradient function and the Laplacian gradient function; wherein, in the Laplacian gradient function adopted in the embodiment of the present invention, the Laplacian initial score is set to 10, and in the Laplacian gradient function, the higher the score of the picture, the higher the clarity.

[0084] In this embodiment, the step of labeling each of the second paintings according to a preset labeling method to obtain a plurality of third paintings and a third label set corresponding to each third painting specifically includes:

[0085] Annotate each of the second paintings according to a preset annotation method to obtain a second label set corresponding to each of the second paintings;

[0086] Cropping each of the second paintings according to a preset cropping standard to obtain a third painting corresponding to each of the second paintings;

[0087] According to the second label set corresponding to each second painting, a third label set corresponding to each third painting is obtained.

[0088] It should be noted that, obtaining the third label set corresponding to each third painting according to the second label set corresponding to each second painting means that the third label set corresponding to the third painting obtained by cropping the second painting is still consistent with the second label set corresponding to the second painting to which the third painting belongs.

[0089] In this embodiment, labeling each second painting according to a preset labeling method to obtain a second label set corresponding to each second painting specifically includes:

[0090] Get the preset labeler;

[0091] Inputting each of the second paintings into the labeler to obtain an initial label set corresponding to each of the second paintings;

[0092] According to a preset deletion standard, the tags in each initial tag set are deleted to obtain a plurality of first tag sets;

[0093] The preset keywords are added to each of the first tag sets in sequence to obtain a plurality of second tag sets corresponding to each second painting.

[0094] In an optional embodiment, the preset labeler refers to a machine learning model such as a WD1.4 labeler for labeling training data; wherein the second painting is input into the WD14-Tagger model so that the WD14-Tagger model processes the second painting to obtain a label probability distribution corresponding to the second painting.

[0095] It should be noted that the labeling of paintings described in the present invention can also be obtained by manual labeling; the WD14-Tagger model (i.e., WD1.4 labeler) used in the present invention is derived from DeepDanbooru, which is a deep learning-based image labeling and classification system specifically used for labeling animation and illustration images. The goal of this system is to automatically identify and label various elements in animation images, such as characters, scenes, actions, etc., through deep learning technology.

[0096] In an alternative embodiment, please refer to Figure 3, is a processing step diagram of a painting label provided by an embodiment of the present invention, wherein, in the initial label set of each painting, unimportant labels are deleted to obtain a first label set, and the first label set has 4 to 5 main labels; then, style keywords are added to the label set, including: figure painting, landscape painting, flower and bird painting, etc.; scene and action keywords are added to the label set, including: grassland, lake, horse riding, etc.

[0097] In an optional embodiment, the step of cropping each of the second paintings according to a preset cropping standard to obtain a third painting corresponding to each of the second paintings specifically includes:

[0098] The second painting is cropped to keep the size of the painting uniform, wherein one side of the painting is ensured to be 1024 pixels and the other side is ensured to be a multiple of 64; in the process of cropping the painting, the integrity of the painting elements is maintained, and the integrity of the painting elements refers to avoiding images of people, flowers and birds, etc. that are cropped in half.

[0099] In an optional embodiment, after the cropping and labeling of the painting is completed, the painting is matched with the tag set one by one, the painting and the corresponding tag set are placed in the same folder, and the folder is named painting name.txt.

[0100] Step S102: input each of the third paintings and the corresponding third label set into a preset initial painting model in sequence, and train a preset adjustment model in combination with the initial painting model to obtain an optimal adjustment model.

[0101] In this embodiment, labeling each second painting according to a preset labeling method to obtain a second label set corresponding to each second painting specifically includes:

[0102] Get the preset labeler;

[0103] Inputting each of the second paintings into the labeler to obtain an initial label set corresponding to each of the second paintings;

[0104] According to a preset deletion standard, the tags in each initial tag set are deleted to obtain a plurality of first tag sets;

[0105] The preset keywords are added to each of the first tag sets in sequence to obtain a plurality of second tag sets corresponding to each second painting.

[0106] In this embodiment, the step of sequentially inputting each of the third paintings and the corresponding third label set into a preset initial painting model, and training a preset adjustment model in combination with the initial painting model to obtain an optimal adjustment model specifically includes:

[0107] Obtain a preset adjustment model and a preset initial painting model;

[0108] Adding the low-rank matrix of the adjusted model to the initial painting model;

[0109] Inputting each of the third paintings and the corresponding third tag sets into the initial painting model in sequence;

[0110] According to a preset training method, the low-rank matrix of the adjustment model is iteratively trained in combination with the initial painting model;

[0111] According to a preset evaluation standard, the training result of the adjustment model is evaluated to obtain an evaluation result;

[0112] The hyperparameters of the adjustment model are adjusted according to the evaluation results until the evaluation results meet the preset requirements, and the iterative training of the adjustment model is completed to obtain the optimal adjustment model.

[0113] In this embodiment, the iterative training of the low-rank matrix of the adjustment model in combination with the initial painting model according to the preset training method specifically includes:

[0114] Changing the rank of the low-rank matrix in turn, and evaluating the impact score of each rank on the low-rank matrix;

[0115] According to the impact score, an optimal rank of the low-rank matrix is ​​obtained;

[0116] The parameters of the weight matrix in the initial painting model are fixed, and the parameters of the low-rank matrix of the adjustment model are iteratively updated in combination with the optimal rank.

[0117] Step S103: merging the optimal adjustment model and the initial painting model to obtain a painting generation model.

[0118] In an alternative embodiment, please refer to Figure 4 , which is an implementation flow chart of a painting generation model provided in an embodiment of the present invention, wherein a suitable basic model is first selected. The basic model selected in the embodiment of the present invention is the SDXL model (stablediffusionXL model), and then LORA is used to fine-tune the basic model to generate a Song Dynasty painting generation domain model (i.e., a painting generation model). Finally, the performance of the Song Dynasty painting generation domain model is evaluated.

[0119] Please note that, please refer to Figure 5, is a structural diagram of an SDXL model provided in an embodiment of the present invention, wherein the SDXL model (stable diffusion XL model) is a two-stage cascade diffusion model, including a Base model and a Refiner model; the Base model works in accordance with the Stable Diffusion model, and has functions such as text-generated graphs, image-generated graphs, and image restoration. After the Base model, the Refiner model is cascaded, and the Refiner model can refine the implicit features (latent features) of the image generated by the Base model, and the refinement refers to image-generated graphs. The SDXL model described in this embodiment is based on a latent diffusion architecture, and the latent diffusion architecture first uses an autoencoder model to compress the image into latent, and then the diffusion model is used to generate latent, and the generated latent can be reconstructed into an image through the decoder of the autoencoder.

[0120] Please note that, please refer to Figure 6 , which is a schematic diagram of a LORA fine-tuning model provided by an embodiment of the present invention; wherein LORA fine-tuning (LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS) refers to adding a side branch next to the Linear layer of the basic model, and the side branch is used to replace the parameter matrix of the basic model for training. As shown in the figure, the matrix W on the left is the pre-trained weight matrix, that is, the SDXL basic model; the matrices A and B on the right are additional matrices added, that is, the Songhua LORA model that needs to be trained.

[0121] In an alternative embodiment, if Figure 6 As shown, using the LORA fine-tuning model, it is only necessary to fix the parameters of the W matrix and only train the A matrix and the B matrix. Since the number of parameters of the A matrix and the B matrix is ​​much smaller than that of the W matrix, the requirements for hardware resources can be reduced and the training efficiency can be improved. In the process of the LORA fine-tuning model, the rank r of the A matrix and the B matrix has a greater impact on the training effect. The larger the rank r, the higher the number of parameters that need to be trained. However, when the rank exceeds a certain number, there will be no significant increase in the effect of training. In the fine-tuning model process described in this embodiment, the size of the initial selection rank r of the A matrix and the B matrix is ​​[2,4,6,8,10,12]. After verification, it can be obtained that the optimal rank of the present invention on the Song Dynasty painting data set is 8.

[0122] In an optional implementation of this embodiment, please refer to Figure 7, is a flow chart of a LORA fine-tuning model provided by an embodiment of the present invention, wherein, Song paintings and corresponding label sets for fine-tuning are obtained; then, a fine-tuning task is defined, and the fine-tuning task of this embodiment refers to training a LORA model; then, a pre-trained model is loaded, and the pre-trained model described in this embodiment refers to selecting an SDXL model as a basis and loading its weights; then, the obtained Song paintings and corresponding label sets are subjected to data batch processing, which refers to dividing the Song paintings and corresponding label sets into small batch samples to improve the model training efficiency and performance; then, a loss function is selected according to the characteristics of the fine-tuning task; then, the Song paintings and corresponding label sets are input into the SDXL model, and the weights of the model are adjusted by the back propagation algorithm to minimize the loss function. Among them, the speed and direction of the gradient update can be controlled by using AdamW; then, multiple iterations of training are performed through Song paintings and corresponding label sets, each iteration is an epoch, and this embodiment trains two epochs, and the performance of the model is regularly evaluated during the training process, and the hyperparameters of the model are adjusted according to the model evaluation results until the evaluation results of the model meet the preset requirements; after the fine-tuning model is completed, the generation effect of the model is tested.

[0123] It should be noted that the epoch refers to the process of one forward propagation and one backward propagation of the entire training data set through the neural network model. When a complete traversal of all samples in the training data set is completed and the weights of the model are updated to minimize the loss function, an epoch is completed. The performance of the evaluation model described in this embodiment refers to the use of a validation set or cross-validation method to evaluate the precision, recall rate, and F1 score of the model; the hyperparameter tuning described in this embodiment refers to adjusting the hyperparameters of the model, and the hyperparameters include: learning rate, batch size, number of layers, number of nodes, etc.

[0124] In an optional embodiment, the merging of the optimal adjustment model and the initial painting model to obtain a painting generation model specifically includes:

[0125] Add the parameters of the corresponding layers of the optimal adjustment model (i.e. the trained LORA model) and the initial painting model (i.e. the basic model SDXL model):

[0126] h=W0x+ΔW x =W0x+BAx,

[0127] Among them, W0x represents the SDXL model, and BAx represents the trained LORA model.

[0128] Step S104: obtaining painting generation words, and inputting the painting generation words into the painting generation model to generate a painting.

[0129] Please refer to Figure 8 , which is a painting generation graph of a painting generation model provided by an embodiment of the present invention; wherein, the painting generation words “charming landscape featuring the style of the Song Dynasty, including misty mountains and tranquil lakes” are input into the painting generation model, and the following is obtained: Figure 8 The Song Dynasty painting shown.

[0130] Please refer to Fig. 9 , which is a flow chart of generating a Song Dynasty painting provided by an embodiment of the present invention, wherein Song Dynasty painting data is first obtained, then the Song Dynasty painting data is screened, and the Song Dynasty painting data is marked and cropped, and finally integrated to obtain training data; the LORA model is obtained by training the training data, and the SDXL model and the LORA model are merged to obtain a Song Dynasty painting generation model, and the user inputs keywords into the Song Dynasty painting generation model to obtain paintings in the Song Dynasty style.

[0131] This embodiment annotates paintings so that each painting has its own corresponding feature label, and inputs paintings and corresponding labels into the model, so that the painting model has semantic knowledge, and can fully explore the relationship between paintings and semantic information in text, so that accurate paintings can be generated based on semantic text. By training the adjustment model and merging the trained optimal adjustment model with the initial painting model, the training of the initial painting model is realized, the training efficiency is improved, and the required calculation amount of the training model is reduced, thereby reducing the redundancy of the painting model and realizing the efficient generation of paintings by the painting model.

[0132] Embodiment 2

[0133] Please refer to Fig.10 , is a structural schematic diagram of a painting generation device provided in an embodiment of the present invention, comprising: a labeling module 201, a model training module 202, a model merging module 203 and a painting generation module 204.

[0134] The labeling module 201 is used to obtain a plurality of third paintings and a third label set corresponding to each third painting according to a preset labeling method and in combination with a plurality of preset initial paintings.

[0135] In this embodiment, the labeling module 201 includes: a labeling submodule;

[0136] The marking submodule is used to obtain a plurality of preset initial paintings;

[0137] Screening a number of the initial paintings according to the clarity of each of the initial paintings to obtain a number of second paintings;

[0138] Each of the second paintings is labeled according to a preset labeling method to obtain a plurality of third paintings and a third label set corresponding to each third painting.

[0139] In this embodiment, the annotation submodule includes: a painting screening unit;

[0140] The painting screening unit calculates the clarity of each of the initial paintings according to a preset image clarity evaluation algorithm;

[0141] Get the preset clarity threshold;

[0142] Screening each of the initial paintings according to the clarity threshold to obtain a plurality of first paintings;

[0143] Duplicate pictures in the first pictures are eliminated to obtain a number of second pictures.

[0144] In this embodiment, the labeling submodule includes: a label acquisition unit;

[0145] The label acquisition subunit is used to label each of the second paintings according to a preset labeling method to obtain a second label set corresponding to each of the second paintings;

[0146] Cropping each of the second paintings according to a preset cropping standard to obtain a third painting corresponding to each of the second paintings;

[0147] According to the second label set corresponding to each second painting, a third label set corresponding to each third painting is obtained.

[0148] In this embodiment, the label acquisition unit includes: a label acquisition subunit;

[0149] The label acquisition subunit is used to acquire a preset labeler;

[0150] Inputting each of the second paintings into the labeler to obtain an initial label set corresponding to each of the second paintings;

[0151] According to a preset deletion standard, the tags in each initial tag set are deleted to obtain a plurality of first tag sets;

[0152] The preset keywords are added to each of the first tag sets in sequence to obtain a plurality of second tag sets corresponding to each second painting.

[0153] The model training module 202 is used to input each of the third paintings and the corresponding third label sets into a preset initial painting model in sequence, and train a preset adjustment model in combination with the initial painting model to obtain an optimal adjustment model.

[0154] In this embodiment, the model training module 202 includes: a model training submodule;

[0155] The model training submodule is used to obtain a preset adjustment model and a preset initial painting model;

[0156] Adding the low-rank matrix of the adjusted model to the initial painting model;

[0157] Inputting each of the third paintings and the corresponding third tag sets into the initial painting model in sequence;

[0158] According to a preset training method, the low-rank matrix of the adjustment model is iteratively trained in combination with the initial painting model;

[0159] According to a preset evaluation standard, the training result of the adjustment model is evaluated to obtain an evaluation result;

[0160] The hyperparameters of the adjustment model are adjusted according to the evaluation results until the evaluation results meet the preset requirements, and the iterative training of the adjustment model is completed to obtain the optimal adjustment model.

[0161] In this embodiment, the model training submodule includes: a model training unit;

[0162] The model training unit is used to sequentially change the rank of the low-rank matrix and evaluate the impact score of each rank on the low-rank matrix;

[0163] According to the impact score, an optimal rank of the low-rank matrix is ​​obtained;

[0164] The parameters of the weight matrix in the initial painting model are fixed, and the parameters of the low-rank matrix of the adjustment model are iteratively updated in combination with the optimal rank.

[0165] The model merging module 203 is used to merge the optimal adjustment model and the initial painting model to obtain a painting generation model.

[0166] The painting generation module 204 is used to obtain painting generation words, input the painting generation words into the painting generation model, and obtain the painting output by the painting generation model.

[0167] This embodiment annotates paintings so that each painting has its own corresponding feature label, and inputs paintings and corresponding labels into the model, so that the painting model has semantic knowledge, and can fully explore the relationship between the painting and the semantic information in the text, so that accurate paintings can be generated based on the semantic text. By training the adjustment model and merging the trained optimal adjustment model with the initial painting model, the training of the initial painting model is realized, the training efficiency is improved, and the required calculation amount of the training model is reduced, thereby reducing the redundancy of the painting model and realizing the efficient generation of paintings by the painting model.

[0168] Embodiment 3

[0169] The embodiments of the present invention provide a terminal device and a computer-readable storage medium.

[0170] The terminal device includes: one or more processors; a memory coupled to the processor and used to store one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement a method for generating a painting as described in the above embodiment.

[0171] The computer-readable storage medium stores a computer program thereon, wherein the computer program is executed by a processor to implement a method for generating a painting as described in the above embodiment.

[0172] In summary, this embodiment annotates paintings so that each painting has its own corresponding feature label, and inputs paintings and corresponding labels into the model, so that the painting model has semantic knowledge, and can fully explore the relationship between the painting and the semantic information in the text, so that accurate paintings can be generated based on the semantic text. By training the adjustment model and merging the trained optimal adjustment model with the initial painting model, the training of the initial painting model is realized, the training efficiency is improved, and the required calculation amount of the training model is reduced, thereby reducing the redundancy of the painting model and realizing the efficient generation of paintings by the painting model.

[0173] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for generating a painting, characterized in that: include: According to a preset labeling method, in combination with a preset number of initial paintings, a number of third paintings and a third label set corresponding to each third painting are obtained; Inputting each of the third paintings and the corresponding third label set into a preset initial painting model in sequence, and training a preset adjustment model in combination with the initial painting model to obtain an optimal adjustment model; Combining the optimal adjustment model and the initial painting model to obtain a painting generation model; A painting generation word is obtained, and the painting generation word is input into the painting generation model to generate a painting.

2. A method for generating a painting as claimed in claim 1, characterized in that: The method of obtaining a plurality of third paintings and a third label set corresponding to each third painting according to a preset labeling method and in combination with a plurality of preset initial paintings specifically includes: Get several preset initial paintings; Screening a number of the initial paintings according to the clarity of each of the initial paintings to obtain a number of second paintings; Each of the second paintings is labeled according to a preset labeling method to obtain a plurality of third paintings and a third label set corresponding to each third painting.

3. A method for generating a painting as claimed in claim 2, characterized in that: The step of screening a plurality of the initial paintings according to the clarity of each of the initial paintings to obtain a plurality of second paintings specifically includes: Calculating the clarity of each of the initial paintings according to a preset image clarity evaluation algorithm; Get the preset clarity threshold; Screening each of the initial paintings according to the clarity threshold to obtain a plurality of first paintings; Duplicate pictures in the first pictures are eliminated to obtain a number of second pictures.

4. A method for generating a painting as claimed in claim 2, characterized in that: The step of labeling each of the second paintings according to a preset labeling method to obtain a plurality of third paintings and a third label set corresponding to each third painting specifically includes: Annotate each of the second paintings according to a preset annotation method to obtain a second label set corresponding to each of the second paintings; Cropping each of the second paintings according to a preset cropping standard to obtain a third painting corresponding to each of the second paintings; According to the second label set corresponding to each second painting, a third label set corresponding to each third painting is obtained.

5. A method for generating a painting as claimed in claim 4, characterized in that: The step of labeling each of the second paintings according to a preset labeling method to obtain a second label set corresponding to each of the second paintings specifically includes: Get the preset labeler; Inputting each of the second paintings into the labeler to obtain an initial label set corresponding to each of the second paintings; According to a preset deletion standard, the tags in each initial tag set are deleted to obtain a plurality of first tag sets; The preset keywords are added to each of the first tag sets in sequence to obtain a plurality of second tag sets corresponding to each second painting.

6. A method for generating a painting according to any one of claims 1 to 5, characterized in that: The step of sequentially inputting each of the third paintings and the corresponding third label set into a preset initial painting model, and training a preset adjustment model in combination with the initial painting model to obtain an optimal adjustment model specifically includes: Obtain a preset adjustment model and a preset initial painting model; Adding the low-rank matrix of the adjusted model to the initial painting model; Inputting each of the third paintings and the corresponding third tag sets into the initial painting model in sequence; According to a preset training method, the low-rank matrix of the adjustment model is iteratively trained in combination with the initial painting model; According to a preset evaluation standard, the training result of the adjustment model is evaluated to obtain an evaluation result; The hyperparameters of the adjustment model are adjusted according to the evaluation results until the evaluation results meet the preset requirements, and the iterative training of the adjustment model is completed to obtain the optimal adjustment model.

7. A method for generating a painting as claimed in claim 6, characterized in that: The iterative training of the low-rank matrix of the adjustment model in combination with the initial painting model according to the preset training method specifically includes: Changing the rank of the low-rank matrix in turn, and evaluating the impact score of each rank on the low-rank matrix; According to the impact score, an optimal rank of the low-rank matrix is ​​obtained; The parameters of the weight matrix in the initial painting model are fixed, and the parameters of the low-rank matrix of the adjustment model are iteratively updated in combination with the optimal rank.

8. A device for generating a painting, characterized in that: include: Annotation module, model training module, model merging module and painting generation module; The labeling module is used to obtain a plurality of third paintings and a third label set corresponding to each third painting according to a preset labeling method and in combination with a plurality of preset initial paintings; The model training module is used to input each of the third paintings and the corresponding third label set into a preset initial painting model in sequence, and train a preset adjustment model in combination with the initial painting model to obtain an optimal adjustment model; The model merging module is used to merge the optimal adjustment model and the initial painting model to obtain a painting generation model; The painting generation module is used to obtain painting generation words, input the painting generation words into the painting generation model, and obtain the painting output by the painting generation model.

9. A terminal device, characterized in that: include: one or more processors; A memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method for generating a painting as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement a method for generating a painting as described in any one of claims 1 to 7.

Images