Style determination model training method, style image generation method and related equipment

Abstract

The invention discloses a style determination model training method, a style image generation method and related equipment. The training method comprises the steps that a style training sample is acquired, the style training sample comprises a sample image with a target style and a text instruction corresponding to the sample image, and the style training sample is used for training a target style model for generating an image with the target style based on the text instruction; determining an initial training sample based on the style training sample; performing initial training on the style determination model based on the initial training sample; an optimization training sample is obtained, the optimization training sample is determined based on an interaction behavior of a target style image, and the target style image is generated by utilizing a target style model based on a target text instruction input by a user; and performing optimization training on a style determination model at least based on the optimization training sample to obtain a trained style determination model, the style determination model being used for determining whether the text instruction corresponds to the target style.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method for a style determination model, a style image generation method, and related equipment. Background Art

[0002] Text-generated images in generative artificial intelligence (AIGC) are a research hotspot that can generate corresponding images based on text.

[0003] In the stylized text image application, images of different styles can also be generated based on text. Images of different styles can be generated by different style models, such as low-rank adaptation of large language models (LoRa) of different styles. However, the style accuracy of the style images generated by the existing method is low, and the matching degree with the style expected by the user is low, which reduces the user experience. Summary of the invention

[0004] The present application provides a style determination model training method, a style image generation method and related equipment to improve the style accuracy of the style image, improve the matching degree with the user's expected style, and thus improve the user experience.

[0005] In a first aspect, the present application provides a method for training a style determination model, the method comprising:

[0006] Acquire a style training sample, the style training sample including a sample image with a target style and a text instruction corresponding to the sample image, the style training sample is used to train a target style model, and the target style model is used to generate an image with a target style based on the text instruction;

[0007] Based on the style training samples, determining an initial training sample, where the initial training sample includes a target style and a plurality of first text instructions corresponding to the target style;

[0008] Based on the initial training samples, the style determination model is initially trained;

[0009] Obtaining an optimized training sample, the optimized training sample including a target style and a target text instruction input by a user corresponding to the target style, the optimized training sample being determined based on an interactive behavior of a target style image, and the target style image being generated using a target style model based on the target text instruction input by the user;

[0010] At least based on the optimized training samples, the style determination model is optimized and trained to obtain a trained style determination model, and the style determination model is used to determine whether the text instruction corresponds to the target style.

[0011] In a second aspect, the present application provides a style image generation method, the method comprising:

[0012] Get the text to be processed;

[0013] Determine the target style of the text to be processed based on the trained style determination model obtained by the method of the first aspect;

[0014] A target image is generated based on the target style and the text to be processed, wherein the style of the target image is the target style.

[0015] In a third aspect, the present application provides a training device for a style determination model, the device comprising: a first acquisition unit and a first processing unit;

[0016] A first acquisition unit is used to acquire a style training sample, the style training sample includes a sample image with a target style and a text instruction corresponding to the sample image, the style training sample is used to train a target style model, and the target style model is used to generate an image with a target style based on the text instruction;

[0017] A first processing unit, configured to determine an initial training sample based on the style training sample, wherein the initial training sample includes a target style and a plurality of first text instructions corresponding to the target style;

[0018] The first processing unit is further used to perform initial training on the style determination model based on the initial training sample;

[0019] The first acquisition unit is further used to acquire an optimized training sample, the optimized training sample includes a target style and a target text instruction input by a user corresponding to the target style, the optimized training sample is determined based on an interactive behavior of a target style image, and the target style image is generated based on the target text instruction input by the user using a target style model;

[0020] The first processing unit is further configured to perform optimization training on the style determination model based at least on the optimized training samples to obtain a trained style determination model, wherein the style determination model is used to determine whether the text instruction corresponds to a target style.

[0021] In a fourth aspect, the present application provides a style image generation device, the device comprising: a second acquisition unit and a second processing unit;

[0022] A second acquisition unit, used to acquire a unit to be processed;

[0023] A second processing unit, configured to determine a target style corresponding to the text to be processed based on the trained style determination model obtained by the method of the first aspect;

[0024] The second processing unit is further used to generate a target image based on the target style and the text to be processed, wherein the style of the target image is the target style.

[0025] In a fifth aspect, the present application provides an electronic device, comprising: a processor and a memory, the processor being connected to the memory, the memory being used to store a computer program, and the processor being used to execute the computer program stored in the memory, so that the electronic device executes the method of the first aspect or the second aspect.

[0026] In a sixth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program enables a computer to execute the method of the first aspect or the second aspect.

[0027] In a seventh aspect, the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and is computer-operable to cause the computer to execute the method of the first aspect or the second aspect.

[0028] The implementation of this application has the following beneficial effects:

[0029] First, a style training sample is obtained, the style training sample includes a sample image with a target style and a text instruction corresponding to the sample image, the style training sample is used to train a target style model, and the target style model is used to generate an image with a target style based on the text instruction; then, based on the style training sample, an initial training sample is determined, the initial training sample includes a target style and a plurality of first text instructions corresponding to the target style; then, based on the initial training sample, the style determination model is initially trained, at this time, when the user does not specify a specific style for generating an image, the style determination model can be used to recommend a corresponding style for the user's input text instruction; further, in order to improve the accuracy of style determination, an optimized training sample can be obtained, the optimized training sample includes a target style and a target text instruction input by the user corresponding to the target style, the optimized training sample is determined based on the interactive behavior of the target style image, and the target style image is generated based on the target text instruction input by the user using the target style model, that is, the style determination model after the initial training determines the corresponding style for the text instruction input by the user, and then generates an image based on the determined style The style image corresponding to the style image is then determined based on the user's interactive behavior on the style image, and the optimized training samples are determined, that is, the style determination model after the initial training is applied, and the optimized training samples are determined based on the application results, which can improve the effect of model training and thus improve the accuracy of style determination; then, at least based on the optimized training samples, the style determination model is optimized and trained to obtain a trained style determination model, that is, the style determination model is first determined through the initial training, and then the style determination model after the initial training is applied in combination with the text instructions input by the user's text image to obtain the optimized training samples, and then the style determination model is optimized and trained at least based on the optimized training samples to obtain a trained style determination model, which can improve the accuracy of style determination, so as to improve the match with the user's expected style, and thus satisfy the user's experience. Of course, when generating a style image based on text, especially when the user does not specify a specific style for generating an image, the style determination model can be used to recommend a matching style for the text, so that the style of the style image obtained by the text image is the matching style, which can increase the user's recognition of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0031] Figure 1 A schematic diagram of a flow chart of a method for training a style determination model provided in an embodiment of the present application;

[0032] Figure 2 A schematic diagram of training a style determination model provided in an embodiment of the present application;

[0033] Figure 3 A schematic diagram of a flow chart of a style image generation method according to an embodiment of the present application;

[0034] Figure 4 A schematic diagram of a style image generation system provided in an embodiment of the present application;

[0035] Figure 5 A schematic diagram of a scene for generating a style image provided in an embodiment of the present application;

[0036] Figure 6 A block diagram of the functional units of a training device for a style determination model provided in an embodiment of the present application;

[0037] Figure 7 A block diagram of the functional units of a style image generation device provided in an embodiment of the present application;

[0038] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0040] The terms "first", "second", "third" and "fourth" etc. in the specification, claims and drawings of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0041] Reference to "embodiments" herein means that a particular feature, result, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0042] From the above background technology, it can be seen that in the application of stylized text images, users are generally required to specify the generated style, and then generate a style image corresponding to the style. However, if the user does not specify the generated style, the final generated image will not have a certain style, and the image cannot be guaranteed to be more beautiful, thereby reducing the image's attractiveness to users. For example, in an Internet community, it will reduce the user's image posting rate, browsing rate, like rate, download rate, sharing rate, etc.

[0043] Therefore, the present application obtains style training samples, the style training samples include sample images with target style and text instructions corresponding to the sample images, the style training samples are used to train the target style model, and the target style model is used to generate images with target style based on the text instructions; then, based on the style training samples, an initial training sample is determined, the initial training sample includes the target style and multiple first text instructions corresponding to the target style; then, based on the initial training samples, the style determination model is initially trained, and at this time, when the user does not specify a specific style for the generated image, the corresponding style can be recommended for the user's input text instructions based on the style determination model; further, in order to improve the accuracy of style determination, an optimized training sample can also be obtained, the optimized training sample includes the target style and the target text instructions input by the user corresponding to the target style, the optimized training sample is determined based on the interactive behavior of the target style image, and the target style image is generated based on the target text instructions input by the user using the target style model, that is, the corresponding style is determined for the text instructions input by the user based on the style determination model after the initial training, and then the corresponding style image is generated based on the determined style, and then based on The user's interactive behavior on the style image determines the optimized training sample, that is, the style determination model after the initial training is applied, and the optimized training sample is determined based on the application result, which can improve the effect of model training and thus improve the accuracy of style determination; then, at least based on the optimized training sample, the style determination model is optimized and trained to obtain a trained style determination model, that is, the style determination model is first determined by the initial training, and then the style determination model after the initial training is applied in combination with the text instruction input by the user's text image to obtain the optimized training sample, and then the style determination model is optimized and trained at least based on the optimized training sample to obtain a trained style determination model, which can improve the accuracy of style determination, so as to improve the matching degree with the user's expected style, and thus satisfy the user's experience. Of course, when generating a style image based on text, especially when the user does not specify the specific style of the generated image, the style determination model can be used to recommend a matching style for the text, so that the style of the style image obtained by the text image is the matching style, which can increase the user's recognition of the image, such as increasing the user's publishing, liking, browsing, downloading, sharing, etc. of the image.

[0044] See also Figure 1 , Figure 1 A flow chart of a method for training a style determination model provided in an embodiment of the present application. The method is applied to the training device of the style determination model, and the method includes but is not limited to steps S101-S105:

[0045] S101: Obtain style training samples.

[0046] In an embodiment of the present application, the style training sample includes a sample image having a target style and a text instruction corresponding to the sample image (which can also be understood as an image description corresponding to the sample image). The style training sample is used to train a target style model, and the target style model is used to generate an image having a target style based on the text instruction.

[0047] It should be noted that in the application of stylized text images, a style model can be used to generate a corresponding style image for the text. The style model can be the above-mentioned LoRa model, and images of corresponding styles can be generated by LoRa models of different styles, that is, one style corresponding to one LoRa model. When training LoRa models of different styles, the training data used is the corresponding style training samples, that is, stylized image-text pairs, wherein the image in the stylized image-text pair of each style of LoRa model is an image with a style corresponding to the LoRa model, and the text in the stylized image-text pair of each style of LoRa model is a text instruction or image description corresponding to the image.

[0048] Therefore, in an embodiment of the present application, in order to obtain the potential connection between text instructions and style, the training data required for the embodiment of the present application when training the style determination model can be determined based on the training data used to train one or more LoRa models of different styles, that is, the style training samples corresponding to one or more LoRa models of different styles. That is, at this time, the style training samples obtained in step S101 of the embodiment of the present application include style training samples for training one or more LoRa models of different styles; if the acquired style training samples only include style training samples of one style of LoRa model, the number of target styles is one, and the trained style determination model is single-classification; if the acquired style training samples include style training samples of multiple styles of LoRa models, the target style includes multiple styles to be trained, that is, styles corresponding to multiple styles of LoRa models, and the trained style determination model is multi-classification.

[0049] S102: Determine an initial training sample based on the style training sample.

[0050] In an embodiment of the present application, the initial training sample includes a target style and a plurality of first text instructions corresponding to the target style. Exemplarily, since the style training sample includes a sample image with a target style and a text instruction corresponding to the sample image, an image description corresponding to the sample image (also referred to as the fifth text instruction) can be generated based on the sample image in the style training sample, i.e., image-to-text conversion; then, based on the image description corresponding to the sample image and the target style to which the sample image belongs, the initial training sample is obtained. It should be noted that since the text instructions corresponding to the sample image included in the style training sample are created when training the target style model (such as artificial generation, generation using a pre-trained model, etc.), the creation method is not necessarily the same as the principle of converting the sample image into the corresponding image description at this time, the text instructions and image description corresponding to the same sample image are not necessarily the same.

[0051] Therefore, if the text instructions and image descriptions corresponding to the same sample image are the same, then only the image description or text instructions corresponding to the sample image can be used as multiple first text instructions in the initial training sample. Optionally, the text instructions or image descriptions corresponding to the sample image can also be deformed to obtain deformed text instructions corresponding to the text instructions or image descriptions corresponding to the sample image, for example: the text instructions or image descriptions of the sample image are randomly segmented to obtain multiple sub-text instructions corresponding to the text instructions or image descriptions of the sample image, wherein the number of random segments is not specifically limited in this application; then the multiple sub-text instructions corresponding to the text instructions or image descriptions of the sample image are randomly combined with the text instructions or image descriptions of other sample images to obtain deformed text instructions corresponding to the sample image; then the image description or text instructions corresponding to the sample image and the deformed text instructions corresponding to the sample image are used as multiple first text instructions in the initial training sample. If the text instructions and image descriptions corresponding to the same sample image are different, then both the text instructions and image descriptions corresponding to the sample image can be used as multiple first text instructions. Similarly, optionally, the text instructions and image descriptions corresponding to the sample image can be deformed respectively to obtain the text instructions corresponding to the sample image and the deformed text instructions corresponding to the image description. The deformation principle is not repeated here. Then, the text instructions, image descriptions, and deformed text instructions corresponding to the sample image are all used as multiple first text instructions to expand and enhance the training data.

[0052] S103: Perform initial training on the style determination model based on the initial training samples.

[0053] In an embodiment of the present application, feature extraction is performed on each first text instruction in the initial training sample to obtain the first text feature corresponding to each first text instruction; then, style classification is performed on the first text feature corresponding to each first text instruction to obtain the classification probability of each first text instruction under the target style; then, based on the classification probability of each first text instruction under the target style and the style label corresponding to each first text instruction, the loss corresponding to each first text instruction is determined, for example, the loss can be a cross entropy loss, a mean square error loss, etc., and the present application does not limit the specific calculation principle of the loss; it should be noted that if the target style includes multiple styles to be trained, style classification is performed on the first text feature corresponding to each first text instruction to obtain the classification probability of each first text instruction under each style to be trained among the multiple styles to be trained; then, based on the classification probability of each first text instruction under each style to be trained, the recommended style corresponding to each first text instruction is determined; then, based on the recommended style corresponding to each first text instruction and the style label corresponding to each first text instruction, the loss corresponding to each first text instruction is determined, for example, the loss can be a cross entropy loss, a mean square error loss, etc., and the present application does not limit the specific calculation principle of the loss. Then, based on the loss corresponding to each first text instruction, a first loss is determined, such as a weighted average of the losses corresponding to multiple first text instructions, which is not specifically limited in this application; then, based on the first loss, the style determination model is initially trained.

[0054] Exemplarily, in an embodiment of the present application, the style determination model may include a feature extractor and a classifier, wherein the feature extractor may be a text feature extractor in a pre-trained contrastive learning-based multimodal model (Contrastive Language-Image Pre-train, CLIP), and the classifier may be a multilayer perceptron (MLP). The multimodal CLIP model may also include an image feature extractor, and the multimodal model may be trained based on an image-text pair, wherein the image-text pair includes an image and a text matching the image. For example, the training method of the CLIP model may include: obtaining N image-text pairs to be trained, wherein each image-text pair to be trained includes a first image and a first text, the content of the first image includes the content in the first text, and N is an integer greater than 1; then inputting each first text into the text feature extractor in the CLIP model to obtain a text feature vector corresponding to each first text, and inputting each first image into the image feature extractor in the CLIP model to obtain an image feature vector corresponding to each first image; then inputting each first text into the image feature extractor in the CLIP model to obtain an image feature vector corresponding to each first image; then The text feature vector corresponding to the present invention and the image feature vector corresponding to each first image are mapped to the same dimension respectively, and the first text feature vector corresponding to each first text and the first image feature vector corresponding to each first image are obtained respectively, that is, at this time, the first text feature vector corresponding to each first text and the first image feature vector corresponding to each first image have the same dimension; then the similarity between the second text feature vector and the first image feature vector corresponding to each first image in the N first images is determined, and N similarities corresponding to the second text feature vector are obtained, wherein the second text feature vector is any one of the N first text feature vectors, and then the N similarities corresponding to each first text feature vector are obtained in the same way, and the total is N at this time 2 similarities, where the similarity can be cosine similarity, etc., which is not specifically limited in the present application; then, based on the N similarities corresponding to each first text feature vector, a second loss is determined, such as contrast loss (Contrastive Loss), etc., which is not limited in the present application; then, based on the second loss, the CLIP model is trained, that is, the text feature extractor and the image feature extractor in the CLIP model are trained, until the similarity between the first image and the first text in each image-text pair to be trained is maximized and the similarity between the first image in each image-text pair to be trained and the first text in other image-text pairs to be trained is minimized, so as to obtain a trained CLIP model.

[0055] Then, feature extraction can be performed on each first text instruction based on the text feature extractor in the trained CLIP model to obtain the first text feature corresponding to each first text instruction; it should be noted that from the training method of the CLIP model, it can be learned that the text feature extractor is trained by the similarity between the image and the text, that is, the text feature extractor contains multimodal semantic information of the image and the text, so that the text feature extractor itself can learn what kind of image content is suitable for presenting a certain text, and then use the text feature extractor to extract the features of each first text instruction, which can maximize the richness of the extracted features and improve the expressiveness of the features, and then in the subsequent classifier training, the text feature can be classified into the most matching style, thereby improving the effect of model training.

[0056] Then, based on the multi-layer perceptron, the first text feature corresponding to each first text instruction is style classified to obtain the recommended style corresponding to each first text instruction; then, based on the recommended style corresponding to each first text instruction and the style label corresponding to each first text instruction, the first loss is determined, and the determination principle of the first loss is not repeated here; then, based on the first loss, the multi-layer perceptron in the style determination model is iteratively trained to obtain the style determination model, that is, because the text feature extractor adopted in the embodiment of the present application is in the pre-trained CLIP model, then in the process of initial training of the style determination model, the parameters of the text feature extractor in the style determination model can be fixed, and only the multi-layer perceptron is trained, that is, only the parameters of the classifier are adjusted, and the parameters of the feature extractor are not adjusted to avoid overfitting.

[0057] S104: Obtain optimized training samples.

[0058] In an embodiment of the present application, the optimized training samples include a target style and a target text instruction input by a user corresponding to the target style. The optimized training samples are determined based on the interactive behavior with respect to the target style image. The target style image is generated using a target style model based on the target text instruction input by the user.

[0059] Exemplarily, if a user wants to generate a corresponding style image based on text, the user can input at least one candidate text instruction through the client, and the present application takes multiple instructions as an example for explanation; then the client sends the candidate text instruction to the server; then the server determines the style corresponding to the candidate text instruction, for example, the server inputs the candidate text instruction into a style determination model after initial training, and outputs the style corresponding to the candidate text instruction; then the server generates a style image corresponding to the candidate text instruction based on the candidate text instruction and the style corresponding to the candidate text instruction, for example: first, a style word corresponding to the style corresponding to the candidate text instruction is added to the candidate text instruction to obtain a fourth text instruction; then, feature extraction is performed on the candidate text instruction and the fourth text instruction respectively to obtain a third text feature vector and a fourth text feature vector respectively; then a noise image is obtained , and extract features from the noise image to obtain a second image feature vector; then, based on the third text feature vector, denoise the second image feature vector multiple times to obtain a third image feature vector, wherein the number of denoising times is not specifically limited in this application; then, decode the third image feature vector to obtain a second image; then, add noise to the second image multiple times to obtain a third image, wherein the number of denoising times is not specifically limited in this application; then, extract features from the third image to obtain a fourth image feature vector; then, based on the fourth text feature vector, denoise the fourth image feature vector multiple times to obtain a fifth image feature vector, wherein the number of denoising times is not specifically limited; then, decode the fifth image feature vector to obtain a fourth image, at which point the fourth image is the style image corresponding to the candidate text instruction, and the content of the fourth image also includes the content corresponding to the candidate text instruction.

[0060] The server then sends the style image corresponding to the candidate text instruction to the client. If the user approves of the style image, the user can download the style image or publish the style image in the Internet community, and then other users can browse, like, and share the style image through the Internet community. Then, the heat value of the style image corresponding to each candidate text instruction can be obtained. At this time, the heat value of the style image corresponding to each candidate text instruction may include one or more of the number of downloads, views, likes, shares, and comments on the style image by the user. This application does not make specific limitations. Then, based on the heat value of the style image corresponding to each candidate text instruction, multiple candidate text instructions are screened to obtain multiple fifth text instructions. For example, if the heat value of the style image corresponding to each candidate text instruction may include any one of the number of downloads, views, likes, shares, and comments on the style image by the user, then the heat value can be obtained. The candidate text instruction corresponding to the style image whose heat value is greater than or equal to the fourth threshold is determined as the fifth text instruction. If the heat value of the style image corresponding to each candidate text instruction can include multiple of the number of downloads, views, likes, shares, comments, etc. of the style image by the user, that is, the heat value of the style image corresponding to each candidate text instruction includes multiple heat values, then the multiple heat values ​​of the style image corresponding to each candidate text instruction can be averaged to obtain the average heat value of the style image corresponding to each candidate text instruction, and then the candidate text instruction corresponding to the style image whose average heat value is greater than or equal to the third threshold is determined as the fifth text instruction. Then, based on multiple fifth text instructions, the optimized training sample is determined, such as using multiple fifth text instructions as text instructions in the optimized training sample.

[0061] S105 . Optimize and train the style determination model at least based on the optimized training samples to obtain a trained style determination model.

[0062] In an embodiment of the present application, the style determination model is used to determine whether the text instruction corresponds to the target style. After determining the optimized training sample, the style determination model can be directly optimized and trained based on the optimized training sample to obtain a trained style determination model, wherein the principle of the optimization training is similar to the principle of the initial training described above and will not be repeated here; during the optimization training process, the parameters of the classifier can also be frozen and only the feature extractor can be trained, that is, only the parameters of the classifier can be adjusted without adjusting the parameters of the feature extractor.

[0063] In an optional embodiment, the expanded training samples can also be determined based on the style training samples. For example, since the style training samples in the above embodiment include styles corresponding to LoRa models of different styles, and the styles corresponding to LoRa models of all styles are the styles that need to be recommended in the style determination model in the embodiment of the present application, that is, the style training samples include multiple styles to be trained, and the multiple styles to be trained correspond one-to-one to the multiple style models. Therefore, based on the style of the style image corresponding to each fifth text instruction (i.e., the style determined by the style determination model), the number of fifth text instructions corresponding to each style to be trained in multiple fifth text instructions can be determined; then, multiple text instructions corresponding to the first style are obtained from the style training samples, wherein the number of fifth text instructions corresponding to the first style in the multiple fifth text instructions can be less than or equal to the fourth threshold, and the first style is one or more of the multiple styles to be trained. That is to say, the fifth text instruction can be understood as the amount of data feedback on the user side. The number of fifth text instructions corresponding to the first style is less than or equal to the fourth threshold, indicating that the amount of data feedback on the user side corresponding to the first style is small, and the following situations exist: (1) The first style is recommended to the candidate text instruction, but the user's recognition is low (which can be reflected from the heat value of the corresponding style image), which indirectly reflects that The style determination model needs to be improved in style recommendation; (2) the first style is a newly added style, that is, the styles to be trained in the initial training stage of the style determination model do not include the first style, but are the styles added during the subsequent optimization training; for these two situations, in order to ensure that the training data corresponding to the first style is sufficient during the subsequent optimization training, multiple text instructions corresponding to the first style are obtained from the style training samples to enhance the training data corresponding to the first style; then the text instructions corresponding to the first style are deformed to obtain multiple second text instructions, which are different from the first text instructions. The deformation principle will not be repeated here; then based on the target style and the multiple second text instructions, an expanded training sample is obtained, that is, the expanded training sample includes the target style and the multiple second text instructions corresponding to the target style, so as to ensure that the training data for the style with less data feedback on the user side or the new style is sufficient.

[0064] Then, based on the optimized training samples and the expanded training samples, the style determination model is optimized and trained to obtain a trained style determination model. The optimization principle is similar to the principle of the initial training mentioned above and will not be repeated here. In addition, since the optimized training samples and the expanded training samples can be understood as the total training data in the optimization training process, the style determination model needs to be trained in multiple batches during the optimization training process, and the training samples of each batch are obtained from the total training data.

[0065] Therefore, in an optional embodiment, the preset number of training samples in each batch can also be obtained; and the first ratio between the optimized training samples and the expanded training samples in the preset training samples in each batch can be obtained; then based on the first ratio and the preset number of training samples in each batch, the first target training samples of each batch are determined from the optimized training samples and the expanded training samples, at which time the number of text instructions in the first target training samples is the number of training samples in each batch, and the ratio between the text instructions extracted from the optimized training samples and the text instructions extracted from the expanded training samples in the first target training samples is the first ratio; then based on the first target training samples of each batch, the style determination model is optimized and trained to obtain a trained style determination model, and the principle of optimization training is not repeated here.

[0066] In an optional embodiment, when the style determination model performs multi-classification, the target style includes multiple styles to be trained, and the preset number of training samples in each batch can also be obtained; and the second ratio between the text instructions corresponding to the multiple styles to be trained in each batch of training samples is obtained, exemplarily: the number of candidate text instructions corresponding to each style to be trained in multiple candidate text instructions is obtained; then, based on the ratio of the number of candidate text instructions corresponding to each style to be trained to the number of multiple candidate text instructions, the first ratio corresponding to each style to be trained is determined; then, based on the interactive behavior of the target style image corresponding to each candidate text instruction, the heat value corresponding to each style to be trained is determined, and the generation principle of the heat value is not repeated here; then, based on the heat value corresponding to each style to be trained, the recommended ratio corresponding to each style to be trained is determined. Recommendation accuracy, for example, based on the mapping relationship between the preset heat value and the recommendation accuracy, and the heat value of the style image corresponding to the candidate text instruction corresponding to each style to be trained, determine the first recommendation accuracy corresponding to the candidate text instruction corresponding to each style to be trained, and then determine the recommendation accuracy corresponding to each style to be trained based on the first recommendation accuracy corresponding to the candidate text instruction corresponding to each style to be trained, such as averaging the first recommendation accuracy corresponding to the candidate text instruction corresponding to each style to be trained, which is not specifically limited in this application, wherein the value range of the recommendation accuracy can be 0 to 1; then based on the first ratio corresponding to each style to be trained and the recommendation accuracy corresponding to each style to be trained, determine the second ratio between the text instructions corresponding to the multiple styles to be trained in each batch of training samples, exemplarily:

[0067] First, a third ratio between text instructions corresponding to multiple styles to be trained in each batch of training samples is obtained; then, based on the first ratio corresponding to each style to be trained and the recommendation accuracy corresponding to each style to be trained, a training weight corresponding to each style to be trained is determined, wherein the training weight corresponding to each style to be trained indicates the importance of training the style to be trained in the training process. For example, a first mapping relationship between the ratio corresponding to the style to be trained and the training weight is preset, and a second mapping relationship between the recommendation accuracy and the training weight is preset. Then, based on the first ratio corresponding to each style to be trained and the first mapping relationship, a first training weight corresponding to each style to be trained can be determined, and based on the recommendation accuracy corresponding to each style to be trained and the second mapping relationship, a second training weight corresponding to each style to be trained can be determined. Then, based on the first training weight and the second training weight corresponding to each style to be trained, a training weight corresponding to each style to be trained can be determined, such as averaging the first training weight and the second training weight corresponding to each style to be trained, etc., which is not specifically limited in this application. Of course, optionally, the training weight corresponding to any one of the multiple styles to be trained, i.e., the i-th style to be trained, can also be obtained by formula (1):

[0068]

[0069] Among them, Q i is the training weight corresponding to the i-th style to be trained, is the first ratio corresponding to the i-th style to be trained, H i is the recommendation accuracy corresponding to the i-th style to be trained, N i is the number of candidate text instructions corresponding to the ith style to be trained, and M is the number of multiple candidate text instructions; it can be seen that the more candidate text instructions corresponding to the ith style to be trained, that is, the larger the first ratio, it means that the style determination model recommends the ith style to be trained, which can be more recognized by users. It also indirectly reflects that the accuracy of the style determination model in recommending the ith style to be trained is also higher, and then the training weight corresponding to the ith style to be trained in this case in the second stage of training is smaller. Similarly, the higher the recommendation accuracy corresponding to the ith style to be trained, the smaller the training weight corresponding to the ith style to be trained in this case in the second stage of training. That is, the second stage training process can reduce the training data corresponding to the ith style to be trained, and increase the training data for the styles to be trained with larger training weights, so as to focus on training in the second stage of training and balance the training results.

[0070] Therefore, based on the training weight corresponding to each style to be trained, the third ratio between the text instructions corresponding to the multiple styles to be trained in each batch of training samples can be adjusted to obtain the second ratio between the text instructions corresponding to the multiple styles to be trained in each batch of training samples, exemplarily:

[0071] First, based on the training weights corresponding to each style to be trained, the first style to be trained and the second style to be trained are determined, wherein the training weight corresponding to the first style to be trained is less than or equal to the first threshold, the training weight corresponding to the second style to be trained is greater than or equal to the second threshold, and the first threshold is less than the second threshold; then, based on the training weight corresponding to the first style to be trained, the reduction ratio of training samples corresponding to the first style to be trained is determined, and based on the training weight corresponding to the second style to be trained, the increase ratio of training samples corresponding to the second style to be trained is determined, for example, the reduction ratio of training text corresponding to the first style to be trained may be the first to third ratio, and the increase ratio of training text corresponding to the second style to be trained may be the second to third ratio; of course, optionally: the first threshold and the first style to be trained may also be determined. The method comprises the steps of: obtaining a first difference between the training weights corresponding to the second style to be trained and determining a second difference between the training weight corresponding to the second style to be trained and the second threshold; then obtaining a third mapping relationship between the difference between the preset first threshold and the training weight corresponding to the style to be trained whose training weight is less than or equal to the first threshold, and the training sample increase ratio; and obtaining a fourth mapping relationship between the difference between the training weights corresponding to the style to be trained whose training weight is greater than or equal to the second threshold, and the second threshold; then determining the training sample reduction ratio corresponding to the first style to be trained based on the third mapping relationship and the first difference, and determining the training sample increase ratio corresponding to the second style to be trained based on the fourth mapping relationship and the second difference. The present application does not specifically limit the training sample reduction ratio and the training sample increase ratio.

[0072] Then, based on the reduction ratio of training samples corresponding to the first style to be trained and the increase ratio of training samples corresponding to the second style to be trained, the ratio of the first style to be trained and the second style to be trained in the third ratio is adjusted to obtain the second ratio between the text instructions corresponding to the multiple styles to be trained in each batch of training samples. It should be noted that if the ratio of training samples corresponding to any two styles to be trained in the third ratio corresponding to each batch of training samples does not exceed the fifth threshold value (not specifically limited in this application, such as not exceeding X:1, where X is an integer greater than 1), then the ratio corresponding to the first style to be trained in the third ratio can be reduced by the reduction ratio of training samples corresponding to the first style to be trained, and the ratio corresponding to the second style to be trained in the third ratio can be increased by the increase ratio of training samples corresponding to the second style to be trained, and the ratios corresponding to the styles to be trained other than the first style to be trained and the second style to be trained in the third ratio remain unchanged, so as to obtain text instructions corresponding to the multiple styles to be trained in each batch of training samples. ; further, it can also be determined whether the ratio between any two styles to be trained in the ratio between the text instructions corresponding to the multiple styles to be trained in each batch of training samples exceeds the fifth threshold. If not, the second ratio between the text instructions corresponding to the multiple styles to be trained in each batch of training samples will not be adjusted. If exceeded, the ratio between the two styles to be trained that exceed the fifth threshold is adjusted to the fifth threshold, so as to obtain the final second ratio between the text instructions corresponding to the multiple styles to be trained in each batch of training samples; that is, for the first style to be trained whose training weight is too low, the training data with the style label of the first style to be trained is reduced in each batch of training samples; for the second style to be trained whose training weight is too high, the training data with the style label of the second style to be trained is increased in each batch of training samples, so as to avoid the style determination model's recommendation bias towards the second style to be trained, so as to balance the recommendation ability of the style determination model for each style to be trained and ensure the effect of model training.

[0073] Then, based on the second ratio between the text instructions corresponding to the multiple styles to be trained in each batch of training texts and the number of training samples in each batch, the second target training samples of each batch are obtained from the optimized training samples and the expanded training samples. For example, the first ratio between the optimized training samples and the expanded training samples in each batch of training samples is obtained, and then based on the first ratio, the second ratio, and the number of training samples in each batch, the second target training samples of each batch are obtained from the optimized training samples and the expanded training samples, wherein the number of the second target training samples of each batch is the number of training samples in each batch, the ratio between the text instructions in the optimized training samples and the text instructions in the expanded training samples included in the second target training samples of each batch is the first ratio, and the ratio between the text instructions corresponding to the multiple styles to be trained included is the second ratio. Finally, based on the second target training samples of each batch, the style determination model is optimized and trained to obtain a trained style determination model, and the principle of optimization training is not repeated here.

[0074] For easier understanding, see Figure 2 , Figure 2 A training diagram of a style determination model provided in an embodiment of the present application. Figure 2 As shown, the style determination model includes a feature extractor and a classifier. First, an initial training sample is determined based on the style training sample (the principle will not be repeated here); then the initial training sample is input into the style determination model, and the style determination model is initially trained. For example, the first text instruction in the initial training sample is input into the feature extractor of the style determination model to obtain the first text feature corresponding to each first text instruction; then the first text feature corresponding to each first text instruction is input into the classifier of the style determination model to obtain the classification result corresponding to each first text instruction, that is, the recommended style corresponding to each first text instruction; then based on the classification result corresponding to each first text instruction, a first loss is determined (the principle will not be repeated here); then the parameters of the feature extractor are frozen, and the classifier in the style determination model is initially trained based on the first loss.

[0075] Then, multiple candidate text instructions in the above embodiment are obtained; and then the multiple candidate text instructions are input into the style determination model (i.e., corresponding to Figure 2 The input branch (1) shown in the figure obtains the output result of each candidate text instruction (i.e., the corresponding Figure 2The output branch (1) shown in the figure, for example, multiple candidate text instructions are input into the feature extractor in the style determination model, and the candidate text instruction features corresponding to each candidate text instruction are output. Then, the candidate text instruction features corresponding to each candidate text instruction are input into the classifier in the style determination model, and the output results corresponding to each candidate text instruction are output, that is, the style corresponding to each candidate text instruction; then, based on the output results of each candidate text instruction and multiple candidate text instructions, the optimized training samples are determined, and the expanded training samples are determined based on the style training samples, which will not be repeated here; then, based on the optimized training samples and the expanded training samples, the style determination model is optimized and trained; then, the optimized training samples and the expanded training samples are input into the style determination model (that is, the corresponding Figure 2 The input branch (2) shown in FIG. 1 is used to calculate the output results of the optimized training samples and the expanded training samples (i.e., the corresponding Figure 2 The output branch (2) shown in the figure trains the classifier in the style determination model to obtain a trained style determination model, which will not be described in detail here.

[0076] Furthermore, the style determination model trained based on the above embodiment can be used to recommend a style for the text, and then generate a corresponding style image, see Figure 3 , Figure 3 Schematic diagram of a flow chart of a style image generation method according to an embodiment of the present application. The method is applied to a style image generation device, and the method includes but is not limited to steps S301-S303:

[0077] S301, obtaining text to be processed.

[0078] In an embodiment of the present application, the text to be processed may be input by a user through a client, and then acquired by the style image generating device from the client.

[0079] S302: Determine a target style corresponding to the text to be processed based on a style determination model.

[0080] In the embodiment of the present application, the training method of the style determination model in step S302 can refer to the relevant description of the above embodiment, which will not be repeated here.

[0081] S303: Generate a target image based on the target style corresponding to the to-be-processed text and the to-be-processed text, wherein the style of the target image is the target style.

[0082] In an embodiment of the present application, after determining the target style corresponding to the text to be processed based on the style determination model, a target image including the target style can be generated based on the target style and the text to be processed, and the content of the target image also includes the content of the text to be processed. It should be noted that the principle of step S303 is similar to the principle of generating a style image corresponding to each candidate text instruction based on the style corresponding to each candidate text instruction and each candidate text instruction in the above embodiment, and will not be repeated here. Furthermore, the image generation device can also send the target image to the client so that the user can publish, download, save, and other operations on the target image through the client to meet the needs of the user, and then other users can browse, like, share, comment, and other operations on the target image through the corresponding client to enhance the direct interactivity of users and increase the activity of the Internet community.

[0083] See also Figure 4 , Figure 4 A schematic diagram of a style image generation system provided in an embodiment of the present application.

[0084] Figure 4 The system shown includes an image generating device and a client; the image generating device can be a server, and the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, as well as basic cloud computing services such as big data and artificial intelligence platforms, which are not specifically limited in this application; the client can be a smart phone, tablet computer, laptop computer, desktop computer, smart TV, desktop computer, smart watch, smart car and other smart terminals, but is not limited to these.

[0085] A target software may be installed on the client, and the user may generate, share, and publish media content through the target software, and may also realize the function of a text image through the target software. Therefore, when the user completes the input of the text to be processed through the target software on the client, the client obtains the text to be processed accordingly, and then the image generating device obtains the text to be processed from the client; then the image generating device inputs the text to be processed into a style determination model, and outputs a target style corresponding to the text to be processed; then the image generating device generates a target image with a target style based on the target style and the text to be processed, which will not be repeated here; further, the image generating device may send a target image corresponding to the text to be processed to the client, and accordingly, after receiving the target image corresponding to the text to be processed, the client may display the target image, and then the user may publish the target image to an Internet community, or save it to a local draft box, or save it to a local photo album, etc., which is not specifically limited in this application.

[0086] It should be noted that the main scenarios applicable to the style image generation method of this application may include media content publishing scenarios, media content creation scenarios, and other scenarios involving text generation of images, such as generating advertising images from advertising texts in advertising services and providing illustrations for texts. This application does not list them one by one. Figure 5 , Figure 5 A schematic diagram of a scene for generating a style image provided in an embodiment of the present application, such as Figure 5 As shown, the user enters the text to be processed through the client ( Figure 5 What is shown is input through a virtual keyboard, of course there are other ways, which are only examples and not limited here), then the image generating device obtains the text to be processed from the client, and then the image generating device generates a target image corresponding to the text to be processed based on the style image generating method of the embodiment of the present application, and the style of the target image is the target style at this time; then the image generating device sends the target image to the client, and accordingly, the client displays the target image to the user; further, the user can publish the target image by touching the "Publish" function button.

[0087] It can be seen that in the embodiment of the present application, the initial training sample is first determined by the style training sample, and then the style determination model is initially trained based on the initial training sample, and then a plurality of candidate text instructions for the text map are obtained from the client, and the style determination model is used to determine the corresponding style for each candidate text instruction, and then based on the determined style and the corresponding candidate text instruction, a style image with the style is generated, and then the corresponding style image is sent to the client, and then the user can choose whether to publish the style image, and then other users can perform interactive operations such as likes, comments, and sharing, and these interactive operations can reflect the heat value corresponding to the style image, and then each candidate text instruction can be obtained. The heat value of the corresponding style image is calculated, and then the optimized training sample is determined based on the heat value of the style image corresponding to each candidate text instruction. That is, the second training text at this time takes into account the feedback effect of the user on the style recommendation. In addition, the expanded training sample can be determined based on the style training sample, and then the style determination model is optimized and trained based on the optimized training sample and the expanded training sample, so that the style recommendation ability of the style determination model becomes more accurate with the accumulation of feedback data from the user side, which can improve the training quality of the style determination model, and then improve the accuracy of style determination, and then recommend a more matching style to the user based on the style determination model, improve the matching degree with the user's expected style, and improve user satisfaction.

[0088] See also Figure 6 , Figure 6This is a functional unit block diagram of a style determination model training device provided in an embodiment of the present application. The style determination model training device 600 includes: a first acquisition unit 601 and a first processing unit 602;

[0089] A first acquisition unit 601 is used to acquire a style training sample, where the style training sample includes a sample image with a target style and a text instruction corresponding to the sample image, and the style training sample is used to train a target style model, and the target style model is used to generate an image with a target style based on the text instruction;

[0090] A first processing unit 602 is configured to determine an initial training sample based on the style training sample, where the initial training sample includes a target style and a plurality of first text instructions corresponding to the target style;

[0091] The first processing unit 602 is further configured to perform initial training on the style determination model based on the initial training sample;

[0092] The first acquisition unit 601 is further used to acquire an optimized training sample, the optimized training sample includes a target style and a target text instruction input by a user corresponding to the target style, the optimized training sample is determined based on an interactive behavior of a target style image, and the target style image is generated based on the target text instruction input by the user using a target style model;

[0093] The first processing unit 602 is further configured to perform optimization training on the style determination model based at least on the optimized training samples to obtain a trained style determination model, where the style determination model is used to determine whether the text instruction corresponds to a target style.

[0094] In one embodiment of the present application, in terms of optimizing and training the style determination model based at least on the optimized training sample to obtain the trained style determination model, the first processing unit 602 is specifically configured to:

[0095] Determine an expanded training sample based on the style training sample, where the expanded training sample includes a target style and a plurality of second text instructions corresponding to the target style, where the second text instructions are different from the first text instructions;

[0096] Based on the optimized training samples and the expanded training samples, the style determination model is optimized and trained to obtain a trained style determination model.

[0097] In one embodiment of the present application, in terms of optimizing and training the style determination model based on the optimized training samples and the expanded training samples to obtain the trained style determination model, the first processing unit 602 is specifically configured to:

[0098] A first ratio between optimized training samples and expanded training samples in each batch of training samples is determined based on a preset number;

[0099] Based on the first ratio and the preset number of training samples in each batch, determining a first target training sample in each batch from the optimized training samples and the expanded training samples;

[0100] Based on the first target training sample of each batch, the style determination model is optimized and trained to obtain a trained style determination model.

[0101] In one embodiment of the present application, when the style determination model performs multi-classification, the target style includes multiple styles to be trained. In terms of optimizing and training the style determination model based on the optimized training samples and the expanded training samples to obtain the trained style determination model, the first processing unit 602 is specifically used to:

[0102] Get the preset number of training samples per batch;

[0103] Obtaining a second ratio between text instructions corresponding to a plurality of to-be-trained styles in each batch of training samples;

[0104] Based on the second ratio and the number of training samples in each batch, obtaining second target training samples in each batch from the optimized training samples and the expanded training samples;

[0105] Based on the second target training samples of each batch, the style determination model is optimized and trained to obtain a trained style determination model.

[0106] In one embodiment of the present application, the target text instruction input by the user in the optimized training sample is part or all of the multiple candidate text instructions input by the user. In terms of obtaining the second ratio between the text instructions corresponding to the multiple styles to be trained in each batch of training samples, the first processing unit 602 is specifically used to:

[0107] Obtaining the number of candidate text instructions corresponding to each style to be trained from among the multiple candidate text instructions;

[0108] Determine a first ratio corresponding to each to-be-trained style based on a ratio of the number of candidate text instructions corresponding to each to-be-trained style to the number of multiple candidate text instructions;

[0109] Determine a heat value corresponding to each to-be-trained style based on an interactive behavior with respect to a target style image corresponding to each candidate text instruction;

[0110] Based on the heat value corresponding to each style to be trained, determine the recommendation accuracy corresponding to each style to be trained;

[0111] Based on the first ratio corresponding to each style to be trained and the recommendation accuracy corresponding to each style to be trained, a second ratio between text instructions corresponding to multiple styles to be trained in each batch of training samples is determined.

[0112] In one embodiment of the present application, in determining the second ratio between texts corresponding to multiple styles to be trained in each batch of training samples based on the first ratio corresponding to each style to be trained and the recommendation accuracy corresponding to each style to be trained, the first processing unit 602 is specifically configured to:

[0113] Obtaining a third ratio between text instructions corresponding to a plurality of to-be-trained styles in each batch of preset training samples;

[0114] Determine a training weight corresponding to each to-be-trained style based on the first ratio corresponding to each to-be-trained style and the recommendation accuracy corresponding to each to-be-trained style;

[0115] Based on the training weight corresponding to each to-be-trained style, the third ratio is adjusted to obtain a second ratio between texts corresponding to a plurality of to-be-trained styles in each batch of training samples.

[0116] In one embodiment of the present application, in terms of adjusting the third ratio based on the training weight corresponding to each style to be trained to obtain the second ratio between texts corresponding to multiple styles to be trained in each batch of training samples, the first processing unit 602 is specifically configured to:

[0117] Determine a first style to be trained and a second style to be trained based on the training weights corresponding to each style to be trained, wherein the training weight corresponding to the first style to be trained is less than a first threshold, the training weight corresponding to the second style to be trained is greater than a second threshold, and the first threshold is less than the second threshold;

[0118] Based on the training weight corresponding to the first style to be trained, determining a reduction ratio of training samples corresponding to the first style to be trained, and based on the training weight corresponding to the second style to be trained, determining an increase ratio of training samples corresponding to the second style to be trained;

[0119] Based on the reduction ratio of training samples corresponding to the first style to be trained and the increase ratio of training samples corresponding to the second style to be trained, the ratio of the first style to be trained and the second style to be trained in the third ratio is adjusted to obtain a second ratio between text instructions corresponding to multiple styles to be trained in each batch of training samples.

[0120] In one embodiment of the present application, in terms of obtaining the second target training samples of each batch from the optimized training samples and the expanded training samples based on the second ratio and the number of training samples of each batch, the first processing unit 602 is specifically configured to:

[0121] Obtaining a first ratio between optimized training samples and expanded training samples in each batch of preset training samples;

[0122] Based on the first ratio, the second ratio, and the number of training samples in each batch, second target training samples in each batch are obtained from the optimized training samples and the expanded training samples.

[0123] In one embodiment of the present application, the style determination model includes a feature extractor and a classifier. During the initial training and / or optimization training, only the parameters of the classifier are adjusted, and the parameters of the feature extractor are not adjusted.

[0124] In one embodiment of the present application, the feature extractor is a text feature extractor in a multimodal model, the multimodal model also includes an image feature extractor, and the multimodal model is trained based on image-text pairs, and the image-text pairs include an image and text matching the image.

[0125] In one embodiment of the present application, in determining the initial training sample based on the style training sample, the first processing unit 602 is specifically configured to:

[0126] Based on the sample images in the style training samples, an image description corresponding to the sample images is generated;

[0127] Based on the image description and the target style to which the sample image belongs, the initial training samples are obtained.

[0128] In a specific implementation, the first acquisition unit 601 and the first processing unit 602 described in the embodiment of the present invention may also execute other implementations described in the embodiment of the training method for the style determination model provided in the embodiment of the present invention, which will not be described in detail here.

[0129] See also Figure 7 , Figure 7 A functional unit composition block diagram of a style image generation device provided in an embodiment of the present application. The style image generation device 700 includes: a second acquisition unit 701 and a second processing unit 702;

[0130] The second acquisition unit 701 is used to acquire the text to be processed;

[0131] The second processing unit 702 is used to determine the target style corresponding to the text to be processed based on the style determination model trained in the above embodiment;

[0132] The second processing unit 702 is further configured to generate a target image based on the target style corresponding to the to-be-processed text and the to-be-processed text, wherein the style of the target image is the target style.

[0133] In a specific implementation, the second acquisition unit 701 and the second processing unit 702 described in the embodiment of the present invention may also execute other implementations described in the embodiment of the style image generation method provided in the embodiment of the present invention, which will not be described in detail here.

[0134] See also Figure 8 , Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 8 As shown, the electronic device 800 includes a transceiver 801, a processor 802 and a memory 803. They are connected via a bus 804. The memory 803 is used to store computer programs and data, and can transmit the data stored in the memory 803 to the processor 802.

[0135] The electronic device 800 may be an image generating device or a training device for a style determination model;

[0136] When the electronic device 800 is a training device for a style determination model, the processor 802 is used to read the computer program in the memory 803 and perform the following operations:

[0137] Controlling the transceiver 801 to obtain a style training sample, the style training sample including a sample image with a target style and a text instruction corresponding to the sample image, the style training sample being used to train a target style model, and the target style model being used to generate an image with a target style based on the text instruction;

[0138] Based on the style training samples, determining an initial training sample, where the initial training sample includes a target style and a plurality of first text instructions corresponding to the target style;

[0139] Based on the initial training samples, the style determination model is initially trained;

[0140] Controlling the transceiver 801 to obtain an optimized training sample, the optimized training sample includes a target style and a target text instruction input by a user corresponding to the target style, the optimized training sample is determined based on an interactive behavior of a target style image, and the target style image is generated based on the target text instruction input by the user using a target style model;

[0141] At least based on the optimized training samples, the style determination model is optimized and trained to obtain a trained style determination model, and the style determination model is used to determine whether the text instruction corresponds to the target style.

[0142] In a specific implementation, the transceiver 801 and the processor 802 described in the embodiment of the present invention may also execute other implementations described in the embodiment of the training method of the style determination model provided in the embodiment of the present invention, which will not be described in detail here.

[0143] When the electronic device 800 is an image generating device, the processor 802 is used to read the computer program in the memory 803 and perform the following operations:

[0144] Control transceiver 801 to obtain the text to be processed;

[0145] Based on the style determination model trained in the above embodiment, determining the target style corresponding to the text to be processed;

[0146] A target image is generated based on a target style corresponding to the text to be processed and the text to be processed, wherein the style of the target image is the target style.

[0147] In a specific implementation, the transceiver 801 and the processor 802 described in the embodiment of the present invention may also execute other implementations described in the embodiment of the style image generation method provided in the embodiment of the present invention, which will not be described in detail here.

[0148] Specifically, the transceiver 801 may be Figure 6 The first acquisition unit 601 of the training device 600 of the style determination model of the embodiment or Figure 7 The second acquisition unit 701 of the style image generation device 700 of the embodiment, the processor 802 may be Figure 6 The first processing unit 602 of the training device 600 of the style determination model of the embodiment or Figure 7 The second processing unit 702 of the style image generating apparatus 700 of the embodiment.

[0149] It should be understood that the electronic device in this application can be a training device for a style determination model or a style image generation device. Both the training device for the style determination model and the style image generation device can be servers, such as independent physical servers, or server clusters or distributed systems composed of multiple physical servers. It can also be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and basic cloud computing services such as big data and artificial intelligence platforms. This application does not make specific restrictions. The above electronic devices are only examples, not exhaustive, and include but are not limited to the above electronic devices.

[0150] It should be understood that an embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement part or all of the steps of any style determination model training method or style image generation method recorded in the above method embodiments.

[0151] An embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute part or all of the steps of any style determination model training method or style image generation method recorded in the above method embodiments.

[0152] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0153] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0154] In the several embodiments provided in the present application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0155] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0156] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of a software program module.

[0157] If the integrated unit is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of each embodiment method of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0158] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, and the memory can include: a flash drive, a read-only memory (English: Read-Only Memory, abbreviated as: ROM), a random access memory (English: Random Access Memory, abbreviated as: RAM), a magnetic disk or an optical disk, etc.

[0159] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for training a style determination model, characterized in that: The method comprises: Acquire a style training sample, wherein the style training sample includes a sample image with a target style and a text instruction corresponding to the sample image, wherein the style training sample is used to train a target style model, and the target style model is used to generate an image with the target style based on the text instruction; Based on the style training samples, determining an initial training sample, wherein the initial training sample includes a target style and a plurality of first text instructions corresponding to the target style; Based on the initial training samples, initially training the style determination model; Acquire an optimized training sample, wherein the optimized training sample includes a target style and a target text instruction input by a user corresponding to the target style, wherein the optimized training sample is determined based on an interactive behavior with respect to a target style image, and the target style image is generated based on the target text instruction input by the user using the target style model; At least based on the optimized training samples, the style determination model is optimized and trained to obtain a trained style determination model, and the style determination model is used to determine whether the text instruction corresponds to the target style.

2. The method according to claim 1, characterized in that The step of performing optimization training on the style determination model at least based on the optimized training sample to obtain a trained style determination model includes: Determining an expanded training sample based on the style training sample, wherein the expanded training sample includes a target style and a plurality of second text instructions corresponding to the target style, and the second text instructions are different from the first text instructions; Based on the optimized training samples and the expanded training samples, the style determination model is optimized and trained to obtain a trained style determination model.

3. The method according to claim 2, characterized in that The step of performing optimization training on the style determination model based on the optimized training samples and the expanded training samples to obtain a trained style determination model includes: A first ratio between optimized training samples and expanded training samples in each batch of training samples is determined based on a preset number; Based on the first ratio and the preset number of training samples in each batch, determining a first target training sample in each batch from the optimized training samples and the expanded training samples; Based on the first target training sample of each batch, the style determination model is optimized and trained to obtain a trained style determination model.

4. The method according to claim 2 or 3, characterized in that: When the style determination model performs multi-classification, the target style includes multiple styles to be trained, and the style determination model is optimized and trained based on the optimized training samples and the expanded training samples to obtain a trained style determination model, including: Get the preset number of training samples per batch; Obtaining a second ratio between the text instructions corresponding to the multiple styles to be trained in each batch of training samples; Based on the second ratio and the number of training samples in each batch, obtaining second target training samples in each batch from the optimized training samples and the expanded training samples; Based on each batch of second target training samples, the style determination model is optimized and trained to obtain a trained style determination model.

5. The method according to claim 4, characterized in that The target text instruction input by the user in the optimized training sample is part or all of the multiple candidate text instructions input by the user, and the second ratio between the text instructions corresponding to the multiple styles to be trained in each batch of training samples is obtained, including: Obtaining the number of candidate text instructions corresponding to each style to be trained from the plurality of candidate text instructions; Determine a first ratio corresponding to each to-be-trained style based on a ratio of the number of candidate text instructions corresponding to each to-be-trained style to the number of the plurality of candidate text instructions; Determine a heat value corresponding to each to-be-trained style based on an interactive behavior with respect to a target style image corresponding to each candidate text instruction; Based on the heat value corresponding to each style to be trained, determine the recommendation accuracy corresponding to each style to be trained; Based on the first ratio corresponding to each style to be trained and the recommendation accuracy corresponding to each style to be trained, a second ratio between the text instructions corresponding to the multiple styles to be trained in each batch of training samples is determined.

6. The method according to claim 5, characterized in that The determining, based on the first ratio corresponding to each style to be trained and the recommendation accuracy corresponding to each style to be trained, a second ratio between texts corresponding to the multiple styles to be trained in each batch of training samples includes: Obtaining a third ratio between the text instructions corresponding to the multiple styles to be trained in each batch of preset training samples; Determine a training weight corresponding to each style to be trained based on the first ratio corresponding to each style to be trained and the recommendation accuracy corresponding to each style to be trained; The third ratio is adjusted based on the training weight corresponding to each style to be trained, so as to obtain a second ratio between the texts corresponding to the multiple styles to be trained in each batch of training samples.

7. The method according to claim 6, characterized in that The third ratio is adjusted based on the training weight corresponding to each style to be trained to obtain a second ratio between the texts corresponding to the multiple styles to be trained in each batch of training samples, including: Determine a first style to be trained and a second style to be trained based on the training weights corresponding to each style to be trained, wherein the training weight corresponding to the first style to be trained is less than a first threshold, the training weight corresponding to the second style to be trained is greater than a second threshold, and the first threshold is less than the second threshold; Based on the training weight corresponding to the first style to be trained, determining a reduction ratio of training samples corresponding to the first style to be trained, and based on the training weight corresponding to the second style to be trained, determining an increase ratio of training samples corresponding to the second style to be trained; Based on the reduction ratio of training samples corresponding to the first style to be trained and the increase ratio of training samples corresponding to the second style to be trained, the ratio of the first style to be trained and the second style to be trained in the third ratio is adjusted to obtain a second ratio between text instructions corresponding to the multiple styles to be trained in each batch of training samples.

8. The method according to any one of claims 4 to 7, characterized in that: The step of obtaining second target training samples of each batch from the optimized training samples and the expanded training samples based on the second ratio and the number of training samples of each batch includes: Obtaining a first ratio between optimized training samples and expanded training samples in each batch of preset training samples; Based on the first ratio, the second ratio, and the number of training samples in each batch, second target training samples in each batch are obtained from the optimized training samples and the expanded training samples.

9. The method according to any one of claims 1 to 8, characterized in that: The style determination model includes a feature extractor and a classifier. During the initial training and / or the optimization training, only the parameters of the classifier are adjusted, and the parameters of the feature extractor are not adjusted.

10. The method according to claim 9, characterized in that The feature extractor is a text feature extractor in a multimodal model, and the multimodal model also includes an image feature extractor. The multimodal model is trained based on image-text pairs, and the image-text pairs include an image and text matching the image.

11. The method according to any one of claims 1 to 10, characterized in that: The determining of the initial training sample based on the style training sample comprises: Based on the sample image in the style training sample, generating an image description corresponding to the sample image; The initial training sample is obtained based on the image description and the target style to which the sample image belongs.

12. A style image generation method, characterized in that: The method comprises: Get the text to be processed; Determining a target style corresponding to the text to be processed based on the trained style determination model described in any one of claims 1 to 11; A target image is generated based on a target style corresponding to the to-be-processed text and the to-be-processed text, wherein the style of the target image is the target style.

13. A training device for a style determination model, characterized in that: The training device of the style determination model comprises: a first acquisition unit and a first processing unit; The first acquisition unit is used to acquire a style training sample, wherein the style training sample includes a sample image with a target style and a text instruction corresponding to the sample image, and the style training sample is used to train a target style model, and the target style model is used to generate an image with the target style based on the text instruction; The first processing unit is used to determine an initial training sample based on the style training sample, wherein the initial training sample includes a target style and a plurality of first text instructions corresponding to the target style; The first processing unit is further configured to perform initial training on the style determination model based on the initial training sample; The first acquisition unit is further used to acquire an optimized training sample, the optimized training sample includes a target style and a target text instruction input by a user corresponding to the target style, the optimized training sample is determined based on an interactive behavior with a target style image, and the target style image is generated based on the target text instruction input by the user using the target style model; The first processing unit is further configured to perform optimization training on the style determination model based at least on the optimized training samples to obtain a trained style determination model, wherein the style determination model is used to determine whether the text instruction corresponds to a target style.

14. A style image generating device, characterized in that: The style image generation device comprises: a second acquisition unit and a second processing unit; The second acquisition unit is used to acquire the unit to be processed; The second processing unit is used to determine the target style corresponding to the text to be processed based on the trained style determination model according to any one of claims 1 to 11; The second processing unit is further configured to generate a target image based on a target style corresponding to the text to be processed and the text to be processed, wherein the style of the target image is the target style.

15. An electronic device, characterized in that: include: A processor and a memory, the processor is connected to the memory, the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the method as described in any one of claims 1-12.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 12.

Images