Visual data generation method and device, multi-modal model training method and device and medium

Abstract

The invention provides a visual data generation method, a multi-modal model training method, a multi-modal model training device and a medium. The method comprises the steps of obtaining description information of to-be-generated visual data; the description information corresponds to a style trigger word and image area content information corresponding to the style trigger word; and obtaining output visual data based on the description information and a pre-trained multi-modal model, so that the image styles of at least two areas in the output visual data are different, the accuracy and artistry of image generation are effectively improved, a multi-style fused image can be generated according to the demand of a user, and the user experience is improved. And the requirements of the user on the generation of various styles of fused images are met.

Description

Technical Field

[0001] The present application relates to the field of image generation technology, and in particular to a visual data generation method, a multimodal model training method, a device and a medium. Background Art

[0002] In AIGC applications, the text-generated graph model plays a key role. Users can generate images with the style and content corresponding to the text by inputting text. Furthermore, the text-generated video model generates a video corresponding to a single style of the text based on the text input by the user. For example, in related technologies, in order to achieve diversity in image styles, specific style trigger words in the training data can be improved. However, this method can only generate images with a single style and cannot meet the user's generation needs for integrating multiple styles in one image.

[0003] Therefore, large models in the prior art are unable to generate visual content that integrates multiple image styles. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present application provides a visual data generation method, a multimodal model training method, a device and a medium, which are based on fine-tuning the existing text image model to achieve the generation of multiple style fusion images to meet the user's expectations for the generation of multiple style images.

[0005] The first aspect of the present application provides a method for generating visual data, comprising:

[0006] Obtaining description information of the visual data to be generated; wherein the description information corresponds to a style trigger word and image area content information corresponding to the style trigger word;

[0007] Based on the description information and the pre-trained multimodal model, obtaining output visual data;

[0008] There is a difference in image style between at least two regions in the output visual data.

[0009] Optionally, obtaining description information of the visual data to be generated includes:

[0010] Obtaining prompt information input by a user; wherein the prompt information at least includes text information;

[0011] Performing natural language processing on the text information in the prompt information to obtain a processing result;

[0012] According to the processing result, obtaining a style trigger word indicating the visual data to be generated and corresponding area content information;

[0013] Structured description information is generated according to the style trigger words and the corresponding area content information.

[0014] Optionally, generating structured description information according to the style trigger word and corresponding area content information includes:

[0015] Based on the processing result, matching corresponding style trigger words from a preset word library;

[0016] Determining image region content information corresponding to each of the matched style trigger words;

[0017] Structured description information is generated according to the matched style trigger words and corresponding area content information.

[0018] Optionally, matching corresponding style trigger words from a preset word library includes:

[0019] The similarity between the natural language processing result and the preset word library is calculated to obtain the style trigger words in the preset word library that meet the preset similarity rules.

[0020] A second aspect of the present application provides a method for training a multimodal model as described in any one of the above items, comprising:

[0021] Obtaining an initial model for image generation;

[0022] Obtain an image dataset, including image data of various styles;

[0023] Performing structured annotation on the image dataset, including annotating image region content information of the image and style trigger words of the image style;

[0024] The initial model is trained based on the structured annotated data to obtain a trained multimodal model.

[0025] Optionally, the image data includes only one image region, and the structurally annotating the image data set includes:

[0026] The image data is annotated to obtain content information of an image region in the image data and a style trigger word corresponding to the image region.

[0027] Optionally, the image data includes more than two image areas, and the structurally annotating the image data set includes:

[0028] Performing image segmentation on the image data to obtain the at least two image regions;

[0029] At least two image regions obtained by image segmentation are respectively labeled, including image region content information of the image region and style trigger words of the image style.

[0030] After marking the image data with an image style, the method further includes: generating a set of style trigger words for the same image style, wherein the set of style trigger words includes synonyms or near-synonyms.

[0031] A third aspect of the present application provides a visual data generating device, comprising:

[0032] An acquisition module, used to obtain description information of the visual data to be generated; wherein the description information corresponds to a style trigger word and image area content information corresponding to the style trigger word;

[0033] A processing module is used to obtain output visual data based on the description information and a pre-trained multimodal model; there are differences in image styles of at least two regions in the output visual data.

[0034] Optionally, get the module to:

[0035] Obtaining prompt information input by a user; wherein the prompt information at least includes text information;

[0036] Performing natural language processing on the text information in the prompt information to obtain a processing result;

[0037] According to the processing result, obtaining a style trigger word indicating the visual data to be generated and corresponding area content information;

[0038] Structured description information is generated according to the style trigger words and the corresponding area content information.

[0039] Optionally, the acquisition module is further used to:

[0040] Based on the processing result, matching corresponding style trigger words from a preset word library;

[0041] Determining image region content information corresponding to each of the matched style trigger words;

[0042] Structured description information is generated according to the matched style trigger words and corresponding area content information.

[0043] Optionally, the acquisition module is further used to:

[0044] The similarity between the processing result and the preset word library is calculated to obtain the style trigger words in the preset word library that meet the preset similarity rules.

[0045] A fourth aspect of the present application provides a device for training a visual data generation model, comprising:

[0046] An initial model acquisition model is used to obtain an initial model for image generation;

[0047] An image data set acquisition module is used to obtain image data sets, including image data of various styles;

[0048] An annotation module, used to perform structured annotation on the image data set, including annotating image region content information of the image and style trigger words of the image style;

[0049] The training module is used to train the initial model based on the structured annotated data to obtain a trained multimodal model.

[0050] Optionally, the annotation module is further used to:

[0051] The image data is annotated to obtain content information of an image region in the image data and a style trigger word corresponding to the image region.

[0052] Optionally, the annotation module is further used to:

[0053] Performing image segmentation on the image data to obtain the at least two image regions;

[0054] At least two image regions obtained by image segmentation are respectively labeled, including image region content information of the image region and style trigger words of the image style.

[0055] Optionally, the labeling module is further used to: generate a set of style trigger words for the same image style, wherein the set of style trigger words includes synonyms or near-synonyms.

[0056] A fifth aspect of the present application provides a computer-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor is caused to execute any of the methods described above.

[0057] The sixth aspect of the present application provides an electronic device, which includes: a processor; a memory for storing processor executable instructions; a processor, configured to read executable instructions from the memory and execute the instructions to implement the above-mentioned visual data generation method or multimodal model generation method.

[0058] It can be seen that in the first aspect, the present application provides a method for generating visual data, which obtains descriptive information of visual data to be generated; obtains style trigger words and image area content information corresponding to the style trigger words according to the descriptive information; obtains output visual data based on the descriptive information and a pre-trained multimodal model, so that there are differences in the image styles of at least two areas in the output visual data, effectively improving the accuracy and artistry of image generation, and can generate multi-style fusion images according to user needs, meeting user needs for generating multiple style fusion images.

[0059] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The above and other objects, features and advantages of the present application will become more apparent through a more detailed description of exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.

[0061] Figure 1 It is a flow chart of a method for generating visual data shown in an embodiment of the present application.

[0062] Figure 2 It is a schematic diagram of a target image generated according to description information shown in an embodiment of the present application.

[0063] Figure 3 It is a flow chart of the multimodal model training method shown in an embodiment of the present application.

[0064] Figure 4 It is a schematic diagram of an initial image input by a user shown in an embodiment of the present application.

[0065] Figure 5 It is another schematic diagram of a target image generated according to description information shown in an embodiment of the present application.

[0066] Figure 6 It is a structural schematic diagram of a visual data generating device shown in an embodiment of the present application.

[0067] Figure 7 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0068] The preferred embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0069] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0070] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0071] In the embodiments of the present application, it can be applied to AIGC (artificial intelligence generated content) applications for image generation or video generation, wherein the image corresponding to the text can be generated by the Wensheng graph model, and the video corresponding to the text can be generated by the Wensheng video model. Wensheng graph technology refers to generating images corresponding to text by inputting text descriptions and using Wensheng graph models. The Wensheng graph model is trained on graphic data to understand the content and intent in the text description and generate images that conform to the text description. For example, the application of the Wensheng graph model includes generating landscape paintings, task portraits, cartoon images, etc., and can generate corresponding image content according to specific user needs, such as specific scenes, task postures, color matching, etc.

[0072] In the related art, the Wensheng graph model can generate high-quality single-style images, but it has obvious deficiencies in multi-style fusion. The existing Wensheng graph model intelligently generates images of a specific single style by fine-tuning the trained basic model to generate images that conform to a certain style. Therefore, it is difficult to meet users' needs for diversified image styles.

[0073] See also Figure 1 , the visual data generation method of the present application comprises:

[0074] S100, obtaining description information of visual data to be generated; wherein the description information corresponds to a style trigger word and image region content information corresponding to the style trigger word.

[0075] The visual data may include static image data and dynamic video data. The video data generally refers to a continuous sequence of images, which is composed of a group of continuous images. The image data described below refers to static images or image data that constitutes video data.

[0076] The description information is used to indicate the content of the visual data to be generated. The description information can be information directly input by the user, or it can be information automatically generated based on the user's selection of an operation instruction (for example, an instruction to automatically generate a text description of the visual data to be generated, an instruction to change the style of the visual data to be generated, an instruction to select an image that can be referenced by the visual data to be generated, etc.).

[0077] The style trigger word is used to indicate the image style of the visual data to be generated. The image area content information is used to indicate the information of the area or element of the visual data to be generated, such as the area location description, element name, element color, etc. Through the image area content information, one or more area ranges or element objects of the visual data to be generated, as well as the image content (i.e., image screen) within the area range or the image content of the element object can be located.

[0078] Optionally, obtaining description information of the visual data to be generated includes: obtaining prompt information input by a user; wherein the prompt information includes at least text information; performing natural language processing on the text information in the prompt information to obtain a processing result; obtaining a style trigger word indicating the image to be generated and a corresponding regional content style based on the processing result; generating structured description information based on the style trigger word and the corresponding regional content style. The structured description information obtained through this embodiment can make it easier for a multimodal model that generates visual data to understand the description information, and help overcome the technical difficulty of fusing multiple styles into one visual data, thereby enabling the multimodal model to generate higher quality multi-style fused visual data.

[0079] The structured description information has a predetermined format to facilitate the understanding of the multimodal model, and the predetermined format includes the correspondence and position order between the style trigger words and the image area content information, and in the case of multiple style trigger words, the position order between multiple style trigger words, etc. The predetermined format can be set based on specific business needs. For example, for the prompt information input by the user "There is a cartoon-style blue cat on the grassland, and the mountains in the distance are in the style of ink painting.", the structured description information obtained is:

[0080] 1. Cartoon style: blue cat

[0081] Additional information: Blue cat on the grassland

[0082] 2. Ink painting style: Mountains in the distance

[0083] Additional information: None

[0084] In the embodiment of the present application, the prompt information input by the user can be text information input by the user, such as a natural language description; the prompt information can also be a combination of text information and image information, that is, the user can also send image information while inputting text information. When applied to the scenario of multimedia content input, the present application obtains the prompt information input by the user, thereby drawing multi-style fusion image data according to the user's intention, or adjusting the style of the image data.

[0085] Specifically, the prompt information input by the user usually contains two aspects of information:

[0086] One type of information is called image area content information, which refers to the name of a certain area or object or element in the image / video that the user wants to adjust or generate, which can be the subject, foreground, background, accompanying object and / or blank space in the image, such as various elements contained in the image such as people, trees, sky, rivers, etc.

[0087] Another type of information is image style information, that is, the desired style for a certain area or object in the above image, such as realistic style, comic style, flat painting style, nostalgic style, etc. This information referring to the image style is called the image style trigger word. Style trigger words may include but are not limited to artistic style, color matching, light and shadow effects, texture details, etc.

[0088] It is understandable that the prompt information input by the user may contain standard style trigger words that have been preset in the vocabulary; it may also not contain style trigger words that have been preset in the vocabulary, but use other words to refer to a certain image style. In this case, the technical solution of the present application will use the words to match or calculate similarity in the style trigger vocabulary, so as to clarify the image style that the user wants to obtain.

[0089] It should be noted that this application does not intend to define different image styles. On the contrary, in practical applications, by adding or defining new image styles, the selection of image styles becomes richer and meets business needs; similarly, when an object that has never been defined before appears in image or video data, a new definition of the object can be added. In addition, using the multimodal model training method disclosed in this application, the image processing system can generate richer objects / elements and render images with richer styles.

[0090] It is understandable that the prompt information is the content input by the user. Due to the different operating habits of different users, the content input by the user may be in various forms. As a preferred implementation, after obtaining the prompt information input by the user, the present application further processes the content (such as natural language processing, voice information extraction processing, image information extraction processing, etc.), and then extracts the above-mentioned image area content information and image style information required by the technical solution of the present application. In addition, the description information and prompt information described in the present application can be in Chinese or natural languages ​​such as English.

[0091] For example, in one case, the user wants to obtain an image in which different regions have different styles. In this case, at least two image style information or names can be obtained from the prompt information input by the user, and each image style information or name corresponds to an area in the image to be generated. For example, the information input by the user can be, "There is a cartoon-style blue cat on the grassland, and the mountains in the distance are in the style of ink painting."

[0092] In another case, the user wants to change the style of a part of an existing image. In this case, the prompt word entered by the user indicates the area where the image style is desired to be adjusted, which can be one area or multiple areas. When the style of an image area is desired to be changed, the image style of the area to be adjusted is indicated; when the styles of multiple image areas are desired to be changed, the image style of each area to be adjusted is indicated separately. For example, if the user already has an image of "there is a cartoon-style blue cat on the grassland, and the mountains in the distance are in the style of ink painting.", the user may enter "change the distant mountains to cartoon style" or similar information to achieve the purpose of modifying the style of a part of the object / element in an image.

[0093] In an embodiment of the present application, natural language word segmentation is performed on the text content input by the user, and the input user text content is decomposed into basic units that can be subsequently analyzed, for example, the user text content is decomposed into words, part-of-speech tagging, syntactic analysis, etc. For example, the user input is "There is a cartoon-style blue cat on a realistic-style grassland", and the above text is recognized, which at least includes the words "realistic style", "grassland", "on", "one", "blue cat", "cartoon style", etc. "Grassland" and "blue cat" correspond to the corresponding area content information in the image data to be generated, "on" is the spatial position relationship between the elements, and "realistic style" and "cartoon style" are the style information used to refer to the style of the object in the user input information. When the image style trigger word library preset in this application contains "realistic" and "cartoon", the style trigger word is matched in the information input by the user.

[0094] In order to generate visual data to be generated corresponding to the style and elements, in one embodiment, a plurality of style trigger words and image region content information corresponding to each style trigger word are obtained according to the description information.

[0095] In practical applications, after word segmentation processing of the user's text content, at least the first image area content information, the second image area content information, the first style trigger word and the second style trigger word are obtained. The first image area content information and the second image area content information are used to refer to any two combinations of the subject, foreground, background, accompanying body and / or blank space in the image; the first image style information indicates the image style of the first image area, and the second image style information indicates the image style of the second image area.

[0096] In an embodiment of the present application, the generating of structured description information based on the style trigger words and the corresponding area content information includes: matching the corresponding style trigger words from a preset vocabulary based on the processing results; determining the image area content information corresponding to each of the matched style trigger words; and generating structured description information based on the matched style trigger words and the corresponding area content information.

[0097] In the embodiment of the present application, the preset word library may be a predefined word library, which includes natural language words or descriptions related to style trigger words, so that the system can quickly and accurately identify key information in the user's text content. By calculating the similarity, the style trigger words that best match the text content input by the user can be screened out, thereby ensuring that the generated image can meet the user's expectations as much as possible.

[0098] In the embodiment of the present application, the similarity between the user text content and the style trigger words in the preset vocabulary library can be evaluated by similarity calculation. The similarity calculation can be performed using a variety of methods, such as cosine similarity, Jaccard similarity, edit distance, etc. By setting a threshold, only when the calculated similarity is higher than the threshold, the corresponding style trigger word will be considered a successful match.

[0099] In the application, the similarity between the processing result and the style trigger words in the style trigger word set can be calculated; the calculated similarity values ​​are sorted, and the style trigger words used to indicate the partial area in the initial image are determined based on the sorting results of the style trigger word similarity values.

[0100] In an embodiment of the present application, when calculating the similarity of style trigger words, if there are multiple words in the vocabulary whose similarity with the vocabulary input by the user is higher than a preset threshold, the words in the vocabulary that are higher than the threshold can be sorted according to the similarity from large to small, and the candidate image area content information or candidate style trigger word with the highest similarity value is used as the image area content information or style trigger word of the image to be generated.

[0101] For example, the processing results include "one", "anime style", "white", and "cat". The preset image area content information set is "cat" and "dog", and the preset style trigger word set is "realistic style", "cartoon style", and "abstract style". The similarity between each word segmentation result and each name in the style trigger word set is calculated. The similarity value between "realistic style" in the current word segmentation result and "realistic style" in the preset style trigger word set is 1.0, and the similarity values ​​between "realistic style" in the current word segmentation result and other names in the style trigger word set are all less than the preset threshold value. It can be determined that the image style corresponding to "realistic style" in the preset style trigger word set is the "realistic style" corresponding to the user text content.

[0102] The similarity between each word segmentation result and each style trigger word in the style trigger word set is calculated. The similarity between "animation style" in the current word segmentation result and "cartoon style" in the style trigger word set is 0.9, and the similarity between "realistic style" and "abstract style" is 0.7, both of which are higher than the preset threshold. According to the ranking of the candidate style trigger words, "cartoon style" is determined as the style trigger word for the current image to be generated.

[0103] S101, obtaining output visual data based on the description information and a pre-trained multimodal model, wherein image styles of at least two regions in the output visual data are different.

[0104] The multimodal model can generate images based on text information, and can also generate videos based on text information. In addition, it can also generate images based on input text information and image information, and can generate videos based on input text information and image information, and so on. In an embodiment of the present application, the multimodal model can be based on a diffusion model and trained through a pre-training data set. The pre-trained multimodal model can generate multi-style fused output visual data based on multiple style trigger words and the image area content information corresponding to each style trigger word. Among them, the pre-training data set is a data set containing a large number of images and corresponding text descriptions. During the training process, reconstruction loss is used to ensure the similarity between the generated image and the real image at the pixel level, and each image in the pre-training data set corresponds to text description information to ensure that the text-generated graph model can accurately understand the intention in the text description and generate images that conform to the text description.

[0105] There are differences in image styles of at least two regions in the output visual data, which means that an output visual data output by a pre-trained multimodal model includes two or more image styles, and each image style and each image style region correspond to each style trigger word and the image region content information corresponding to the style trigger word.

[0106] In an embodiment of the present application, descriptive information of visual data to be generated is obtained; the descriptive information corresponds to a style trigger word, and image area content information corresponding to the style trigger word; based on the descriptive information and a pre-trained multimodal model, multi-style fused output visual data is obtained, that is, there are differences in image styles of at least two areas in the output visual data, which not only effectively improves the accuracy and artistry of image generation, but also adopts an end-to-end approach, based on the descriptive information through a pre-trained multimodal model to directly output the multi-style fused visual data expected by the user, thereby meeting the user's demand for generating multiple style fusion images and lowering the threshold for users to use AI technology.

[0107] In practical applications, such as Figure 2 As shown, the prompt information of the image data to be generated input by the user is obtained, and the prompt information is "There is a cartoon-style kitten on a realistic-style grassland. In front of the kitten, there is an oil painting-style stream running through the left and right of the image. There is a cartoon goldfish in the stream."

[0108] The prompt information is processed by natural language, and the processing results obtained include the correspondence between the image area content information and the style trigger words. That is, the "realistic grassland" in the prompt information can be used as a set of correspondence between the image area content information and the style trigger words, and the remaining word segmentation processing results are "cartoon style kitten", "oil painting style stream" and "cartoon goldfish". Each set of image area content information and style trigger words corresponds to at least one image area in the image to be generated.

[0109] Then, the similarity between the obtained processing result and the preset image area content information set and the preset style trigger word set is calculated, among which, the words with the highest similarity values ​​with the style trigger words in the style trigger word set may be "realistic style", "cartoon style", "oil painting style" and "cartoon", and for "cartoon" in "cartoon little goldfish", the corresponding one according to the similarity calculation is "cartoon style".

[0110] In addition, it is recognized that "creek" is a word that is not easy for the multimodal model to understand, and the word is replaced with "river".

[0111] The style trigger words for the to-be-generated images in the word segmentation processing results are “realistic style”, “cartoon style”, “oil painting style”, and “cartoon style”, and the corresponding image area content information are “grassland”, “kitten”, “river”, and “little goldfish”.

[0112] There is a cartoon-style kitten on a realistic grassland, and in front of the kitten there is an oil painting-style stream running through the left and right sides of the image, with a cartoon goldfish in the stream.

[0113] Based on the above natural language processing, structured description information is obtained:

[0114] 1. Realistic style: grassland

[0115] Additional information: There is a kitten on the grassland

[0116] 2. Cartoon style: kitten, goldfish

[0117] Additional information: There is a river in front of the kitten, and there is a little goldfish in the river

[0118] 3. Oil painting style: River

[0119] Additional information: The river runs through the left and right sides of the image

[0120] Finally, output visual data is obtained based on the structured description information.

[0121] On the basis of the above embodiment, description information of the visual data to be generated is obtained, and based on the description information and the pre-trained multimodal model, output visual data is obtained to generate an image with multiple styles fused together, wherein the training method of the multimodal model is as follows: Figure 3 As shown, including:

[0122] S300: Obtain an initial model for image generation.

[0123] S301. Obtain an image dataset, including image data of various styles.

[0124] S302: Perform structured annotation on the image data set, including annotating image region content information of the image and style trigger words of the image style.

[0125] S303: Train the initial model based on the structured annotated data to obtain a trained multimodal model.

[0126] In an embodiment of the present application, the image data in the image data set can be images generated historically by a multimodal model, or can be image resources from various sources such as artworks and photographs. The image data is structuredly annotated. The image style annotation can be performed manually or automatically through a deep learning model. For example, a pre-trained image style recognition model can be used to classify the image data into styles such as realism, cartoon, abstract, oil painting, etc. The annotated image data will contain image area content information and corresponding style trigger words.

[0127] In an embodiment of the present application, structured annotation can be to attach annotation information with a specific hierarchy or association relationship to image data. All image area content information contained in each image data can be determined first, and the image area content information can be classified by element category. For example, the element body and background contained in the image data can be identified, where the element body can be annotated in the form of human role, animal species, still life category, and subject attribute, and the background can include but is not limited to scene lighting, lens, composition, etc.

[0128] For example, first, describe the main body of the element in the following order: character, gender, body shape, facial features, demeanor / expression, and body movements. Next, describe the background, including the characteristics of the light (such as soft or bright), lens selection (such as close-up or telephoto), shooting angle (such as horizontal or upward shooting), and the overall composition of the background. In addition, a style trigger word needs to be added before the description of each main body and background of the element to mark the specific attributes of the style so as to serve as a style guide in the subsequent image generation. It should be emphasized that style trigger words should avoid being similar to common words in natural language to ensure their uniqueness.

[0129] It can be understood that the image data in the image data set only includes one image area, and the image data only contains a single element under a single style. The image area in the image data is structuredly annotated to obtain the content information of the image area and the style trigger words corresponding to the image area.

[0130] In addition, the image data set also includes more than two image regions in the image data, and the structured annotation of the image data set includes:

[0131] Perform image segmentation on the image data to obtain the at least two image regions.

[0132] At least two image regions obtained by image segmentation are respectively labeled, including image region content information of the image region and style trigger words of the image style.

[0133] It can be understood that during the annotation process, style definitions can be performed based on the color, texture, shape and artistic style presented by each image data. Style definitions can also be performed through custom visual expression features of the image data. Each image data can be annotated with a specific style and trained and stored in the model in the form of a set of style trigger words, so that sufficient learning information can be provided to the text image model during the training process.

[0134] In the embodiment of the present application, after marking the image style of the image data, the method further includes:

[0135] For the same image style, a set of style trigger words is generated, and the set of style trigger words includes synonyms or near-synonyms.

[0136] In the embodiment of the present application, in order to facilitate the subsequent model application, the style trigger word set includes a set of synonyms or near-synonyms. For example, the image to be trained is "a cartoon-style cat", and the image area content information contained in the image is identified as "cat" and the style trigger word is "cartoon style". Therefore, in the process of annotating the image, synonyms or near-synonyms other than "cartoon style" can be used as style trigger words of the current image to be trained, such as "comic style", "animation style", "fairy tale style", etc., so as to provide the model with diversified image style expressions.

[0137] In another embodiment, comprising:

[0138] Obtain prompt information input by a user, wherein the prompt information includes text information and initial visual data; perform natural language processing on the text information in the prompt information to obtain a processing result; based on the processing result, obtain a style trigger word indicating the visual data to be generated and corresponding area content information; generate structured description information based on the style trigger word and the corresponding area content information; the description information includes a correspondence between image area content information for indicating a partial area in the initial visual data and the style trigger word.

[0139] Based on the description information and a pre-trained multimodal model, output visual data is obtained; compared with the initial visual data, the image style of the output visual data changes according to the style trigger word in the image area indicated by the image area content information.

[0140] In the embodiment of the present application, the initial visual data may be image data and video data generated by the model, or may be new visual data input by the user terminal simultaneously with the prompt information.

[0141] In the embodiment of the present application, the text content included in the prompt information can be a change or addition to the style trigger word or image area content information in the initial visual data. For example, the initial visual data input by the user is a waterfall landscape picture taken by the user with a camera, and the prompt information is "add a cartoon-style boy to the picture". The prompt information does not change the content in the initial visual data. Therefore, the initial visual data can be used as the base background, and the description information can be processed by natural language to obtain the prompt information with the style trigger word "cartoon style" and the image area content information "boy" added as the foreground area of ​​the image to the initial visual data.

[0142] In another embodiment, the initial visual data is a waterfall landscape picture taken by a user with a camera, and the prompt information may be "painting the sky in cartoon style" in the initial visual data, and the prompt information is to change the style trigger words of some areas in the initial visual data. First, the initial visual data is identified, and the image area corresponding to the "sky" in the initial image is determined by an image segmentation algorithm for segmentation, and the image that meets the image area content information of "sky" and the style trigger word of "cartoon style" is added to the initial image visual data to generate output visual data.

[0143] In the embodiment of the present application, in practical application, Figure 4 As shown, the initial image can be an image input by the user, or an initial image generated in advance by the multimodal model according to the relevant text. Assume that when the initial image is generated by the user, the prompt information input by the user at the same time as the initial image is "a realistic forest with a fairy-tale-style deer and a cartoon-style castle in the forest".

[0144] The prompt information is subjected to natural language processing to obtain image region content information and style trigger words indicating a portion of the area in the initial image. For example, the portion of the area may refer to an area to be changed determined according to the text content of the prompt information, that is, the area to be changed refers to the image regions of "deer" and "castle" in the initial image. The obtained processing result includes the correspondence between the image region content information and the style trigger words, that is, "a fairy tale style deer" and "a comic style castle" in the prompt information are respectively regarded as a set of correspondence between image region content information and style trigger words.

[0145] According to the image area content information in the prompt information, the area to be changed in the initial image is determined, and the style trigger words corresponding to the image area content information in each area to be changed are extracted, and the similarity between the obtained processing structure and the preset style trigger word set is calculated, wherein "fairy tale style" and "comic style" are calculated to have the highest similarity with the style trigger words in the style trigger word set. The style trigger words with the highest similarity value may be "fairy tale style" and "comic style".

[0146] It is understandable that, in this embodiment, the content of the prompt information does not include words such as image area change, so the area corresponding to the image area content information in the initial image can be used as the image area to be changed corresponding to the current style trigger word.

[0147] Finally, if Figure 5 As shown, the output image in the pre-trained Wensheng graph model is sent to the user end, wherein compared with the initial image, in the image areas indicated by "deer" and "castle", the image style of "deer" is changed to "fairy tale style", and the image style of "castle" is changed to "comic style".

[0148] like Figure 6 As shown, the present application provides a visual data generating device, comprising:

[0149] The acquisition module 61 is used to obtain description information of the visual data to be generated; wherein the description information corresponds to a style trigger word and image area content information corresponding to the style trigger word;

[0150] The processing module 62 is used to obtain output visual data based on the description information and a pre-trained multimodal model; there are differences in image styles of at least two regions in the output visual data.

[0151] In the embodiment of the present application, the acquisition module 61 is used to:

[0152] Obtaining prompt information input by a user; wherein the prompt information at least includes text information;

[0153] Performing natural language processing on the text information in the prompt information to obtain a processing result;

[0154] According to the processing result, obtaining a style trigger word indicating the visual data to be generated and corresponding area content information;

[0155] Structured description information is generated according to the style trigger words and the corresponding area content information.

[0156] In the embodiment of the present application, the acquisition module 61 is further used for:

[0157] Based on the processing result, matching corresponding style trigger words from a preset word library;

[0158] Determining image region content information corresponding to each of the matched style trigger words;

[0159] Structured description information is generated according to the matched style trigger words and corresponding area content information.

[0160] Optionally, the acquisition module 61 is further used for:

[0161] The similarity between the processing result and the preset word library is calculated to obtain the style trigger words in the preset word library that meet the preset similarity rules.

[0162] The training method of the above multimodal model includes:

[0163] Obtaining an initial model for image generation;

[0164] Obtain an image dataset, including image data of various styles;

[0165] Performing structured annotation on the image dataset, including annotating image region content information of the image and style trigger words of the image style;

[0166] The initial model is trained based on the structured annotated data to obtain a trained multimodal model.

[0167] Optionally, the image data includes only one image region, and the structural annotation of the image data set includes:

[0168] The image data is annotated to obtain content information of an image region in the image data and a style trigger word corresponding to the image region.

[0169] Optionally, the image data includes more than two image regions, and the structurally annotating the image data set includes:

[0170] Performing image segmentation on the image data to obtain the at least two image regions;

[0171] At least two image regions obtained by image segmentation are respectively labeled, including image region content information of the image region and style trigger words of the image style.

[0172] Optionally, after labeling the image data with an image style, the method further includes: generating a set of style trigger words for the same image style, wherein the set of style trigger words includes synonyms or near-synonyms.

[0173] See also Figure 7 , Figure 7 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application. The electronic device 700 includes a memory 710 and a processor 720.

[0174] The present application can be implemented as a non-temporary machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or electronic device, server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present application.

[0175] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the application herein may be implemented as electronic hardware, computer software, or combinations of both.

[0176] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system and method according to multiple embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a part of a module, a program segment or a code, and the part of the module, the program segment or the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or the flow chart, and the combination of the square boxes in the block diagram and / or the flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0177] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for generating visual data, characterized in that: include: Obtaining description information of the visual data to be generated; wherein the description information corresponds to a style trigger word and image area content information corresponding to the style trigger word; Based on the description information and the pre-trained multimodal model, obtaining output visual data; There is a difference in image style between at least two regions in the output visual data.

2. The method according to claim 1, characterized in that The obtaining description information of the visual data to be generated includes: Obtaining prompt information input by a user; wherein the prompt information at least includes text information; Performing natural language processing on the text information in the prompt information to obtain a processing result; According to the processing result, obtaining a style trigger word indicating the visual data to be generated and corresponding area content information; Structured description information is generated according to the style trigger words and the corresponding area content information.

3. The method according to claim 2, characterized in that The generating structured description information according to the style trigger word and the corresponding area content information includes: Based on the processing result, matching corresponding style trigger words from a preset word library; Determining image region content information corresponding to each of the matched style trigger words; Structured description information is generated according to the matched style trigger words and corresponding area content information.

4. The method according to claim 3, characterized in that The matching of the corresponding style trigger words from the preset word library includes: The similarity between the processing result and the preset word library is calculated to obtain the style trigger words in the preset word library that meet the preset similarity rules.

5. A method for training a multimodal model as claimed in claims 1 to 4, characterized in that: include: Obtaining an initial model for image generation; Obtain image datasets, including image data of various styles; Performing structured annotation on the image dataset, including annotating image region content information of the image and style trigger words of the image style; The initial model is trained based on the structured annotated data to obtain a trained multimodal model.

6. The method according to claim 5, characterized in that The image data only includes one image region, and the structural annotation of the image data set includes: The image data is annotated to obtain content information of an image region in the image data and a style trigger word corresponding to the image region.

7. The method according to claim 5, characterized in that The image data includes more than two image regions, and the structural annotation of the image data set includes: Performing image segmentation on the image data to obtain the at least two image regions; At least two image regions obtained by image segmentation are respectively labeled, including image region content information of the image region and style trigger words of the image style.

8. The method according to any one of claims 5 to 7, characterized in that: After marking the image data with an image style, the method further includes: generating a set of style trigger words for the same image style, wherein the set of style trigger words includes synonyms or near-synonyms.

9. A visual data generating device, characterized in that: include: An acquisition module, used to obtain description information of the visual data to be generated; wherein the description information corresponds to a style trigger word and image area content information corresponding to the style trigger word; A processing module is used to obtain output visual data based on the description information and a pre-trained multimodal model; there are differences in image styles of at least two regions in the output visual data.

10. A computer-readable storage medium, characterized in that: An executable code is stored thereon, and when the executable code is executed by a processor of an electronic device, the processor is caused to execute the method as claimed in any one of claims 1 to 8.

Images