Method, device and storage medium for generating visual content for multiple objects

Abstract

The present invention discloses a method, device and storage medium for generating visual content for multiple objects, the method comprising: parsing an original text description containing multiple objects to obtain a parsing result, the parsing result at least comprising: a category description and an attribute description of each object; based on the parsing result, obtaining a revised text description corresponding to the original text description, the revised text description not including the attribute description of each object; based on the parsing result, fusing the category description and the attribute description of each object respectively to obtain a fusion feature of each object; replacing the category description of each object in the revised text description with the corresponding fusion feature in turn to obtain a final text description; inputting the final text description into a preset multimodal model to generate visual content; the present invention can effectively associate objects in the text description with their corresponding feature descriptions, thereby improving the quality and stability of the content generation task.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device and storage medium for generating visual content for multiple objects. Background Art

[0002] AIGC (Artificial Intelligence Generated Content) is a method that uses artificial intelligence technology to automatically generate various types of content, including text, images, audio, and video. Among them, the development of AIGC technology in application fields such as image generation and video generation continues to push the boundaries of content creation, bringing innovation and efficiency improvements to multiple industries such as art creation, game development, and advertising production.

[0003] AIGC technology can generate various types of images based on text descriptions, and can also generate videos with coherent plots through text descriptions. For tasks such as generating data content such as images or videos based on text, there are often situations involving multiple objects (for example, people, animals, buildings). In this case, there is a problem of feature crosstalk between multiple objects. Feature crosstalk refers to the inability of the model to effectively distinguish feature descriptions of different objects, especially the inability to effectively distinguish feature descriptions of different objects belonging to the same category, which leads to feature confusion of objects in the generated content, making the generated content inconsistent with the input text description. The existence of this feature crosstalk seriously affects the performance of the generation system and user experience.

[0004] Therefore, how to solve the problem of feature crosstalk in generation tasks involving multiple objects has become a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] The purpose of the present invention is to provide a method, device, equipment and storage medium for generating visual content for multiple objects, which can effectively associate objects in text descriptions with their corresponding feature descriptions, thereby improving the quality and stability of content generation tasks.

[0006] According to one aspect of the present invention, a method for generating visual content for multiple objects is provided, the method comprising: Parsing the original text description containing multiple objects to obtain a parsing result, wherein the parsing result at least includes: a category description and an attribute description of each of the objects; Based on the parsing result, obtaining a revised text description corresponding to the original text description, wherein the revised text description does not include an attribute description of each of the objects; Based on the analysis result, the category description and the attribute description of each object are fused to obtain a fusion feature of each object; replacing the category description of each object in the revised text description with the corresponding fusion feature in turn to obtain a final text description; The final text description is input into a preset multimodal model to generate visual content for representing the original text description.

[0007] Optionally, the parsing of the original text description containing multiple objects to obtain a parsing result specifically includes: The original text description is structuredly annotated to obtain a structured annotated text; the structured annotated text includes: a background description, an event description, and an object description, wherein the object description includes a category description and an attribute description of each object in the original text description, the background description is used to characterize the event scene, and the event description is used to characterize the association relationship between the various objects.

[0008] Optionally, the performing structured annotation on the original text description to obtain structured annotated text specifically includes: Performing semantic recognition on the original text description to determine each object in the original text description, a category description and an attribute description of each object, and a background description; Based on the category description and the attribute description of each of the objects, a unique identifier is set for each of the objects, and the identifier can be associated with the category description and the attribute description of each of the objects; Based on the identifier of each of the objects, determining an event description of the original text description; Based on the background description, the event description and the identifier of each of the objects, a structured annotated text corresponding to the original text description is obtained.

[0009] Optionally, obtaining a revised text description corresponding to the original text description based on the parsing result specifically includes: Acquire the background description and the event description from the structured annotation text; Identify the identifiers of each object from the event description, and bind the category description of the corresponding object to the identifier of each object to form a new event description; The background description is added to the new event description to form the revised text description.

[0010] Optionally, based on the parsing result, the category description and the attribute description of each object are fused to obtain a fusion feature of each object, specifically including: The category description of the i-th object in the revised text description is respectively divided into the number of the attribute description of the i-th object Make a copy to get the new revised text description; The category descriptions of each object in the new revised text description are sequentially traversed, and the j-th category description and the j-th attribute description of the currently traversed i-th object are sequentially input into the preset fusion model to obtain the currently traversed i-th object fusion features; where j∈[1, ].

[0011] Optionally, the step of sequentially replacing the category description of each object in the revised text description with a corresponding fusion feature to obtain a final text description specifically includes: The new revised text description of the i-th object The category descriptions are replaced by the corresponding fused features to obtain the final text description.

[0012] Optionally, based on the parsing result, the category description and the attribute description of each object are fused to obtain a fusion feature of each object, specifically including: Tokenizing the revised text description to obtain a text token sequence, and marking the category tokens corresponding to the category description of each object from the text token sequence; Tokenizing the attribute description of each object respectively to obtain an attribute token sequence of each object; The category tokens of the i-th object in the text token sequence are respectively divided into the number of attribute tokens in the attribute token sequence of the i-th object. Make a copy to get a new text token sequence; The new text token sequence is input into a preset text encoder to obtain a text embedding vector group, and the text embedding vector group corresponding to the i-th object is marked respectively. Category embedding vectors; The attribute token sequence of each object is input into the text encoder to obtain the attribute embedding vector group of each object; wherein the attribute embedding vector group of the i-th object includes attribute embedding vectors; The category embedding vectors of each object in the text embedding vector group are traversed in sequence, and the j-th category embedding vector and the j-th attribute embedding vector of the currently traversed i-th object are input into the preset fusion model in order to obtain the i-th object of the current traversal. fused feature embedding vector; where j∈[1, ].

[0013] Optionally, the step of sequentially replacing the category description of each object in the revised text description with the corresponding feature description to obtain a final text description specifically includes: The text is embedded in the i-th object of the vector group The category embedding vectors are replaced by the corresponding The fused feature embedding vectors are used to obtain a new text embedding vector group; The step of inputting the final text description into a preset multimodal model to generate visual content for representing the original text description specifically includes: The new text embedding vector group is input into a preset multimodal model to generate an image or video for representing the original text description.

[0014] In order to achieve the above object, the present invention further provides a visual content generation device for multiple objects, the device comprising: A parsing module, used to parse the original text description containing multiple objects to obtain a parsing result, wherein the parsing result at least includes: a category description and an attribute description of each of the objects; A revision module, configured to obtain a revised text description corresponding to the original text description based on the parsing result, wherein the revised text description does not include an attribute description of each of the objects; A fusion module, based on the parsing result, respectively fuses the category description and the attribute description of each object to obtain a fusion feature of each object; A replacement module, used to replace the category description of each object in the revised text description with the corresponding fusion feature in turn to obtain a final text description; A processing module is used to input the final text description into a preset multimodal model to generate visual content for representing the original text description.

[0015] In order to achieve the above-mentioned purpose, the present invention also provides a computer device, which specifically includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for generating visual content for multiple objects when executing the computer program.

[0016] In order to achieve the above object, the present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method for generating visual content for multiple objects introduced above are implemented.

[0017] The visual content generation method, device, equipment and storage medium for multiple objects provided by the present invention propose a simple, efficient and universal association mechanism, which effectively associates the category description of each object with its corresponding attribute description, thereby largely avoiding the information crosstalk problem in the content generation task involving multiple objects, and effectively improving the quality and stability of the content generation task. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings: Figure 1 An optional flowchart of the method for generating visual content for multiple objects provided in Embodiment 1; Figure 2 A schematic diagram of another optional flow chart of the method for generating visual content for multiple objects provided in the first embodiment; Figure 3 A schematic diagram of an optional composition structure of a visual content generation device for multiple objects provided in Embodiment 2; Figure 4 This is a schematic diagram of an optional hardware structure of a computer device provided in Example 3. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0020] Embodiment 1 The embodiment of the present invention provides a method for generating visual content for multiple objects, such as Figure 1 As shown, the method specifically comprises the following steps: Step S101: parsing an original text description containing multiple objects to obtain a parsing result, wherein the parsing result at least includes: a category description and an attribute description of each of the objects.

[0021] Among them, the category description is the content in the original text description used to distinguish objects. Optionally, the category description is used to classify objects. The specific categories can be divided according to business. Its function is to distinguish the multiple objects included in the original text description from the category, such as the species and gender of each object. In some embodiments, different objects can be distinguished by the species of the object combined with the gender. For example, the category description is: man, woman, cat, dog. Different objects can be distinguished to a certain extent through the category description. The attribute description is the information about the appearance, clothing, posture, etc. of an object in the original text description that can be used for object identification. Through the attribute description, the object corresponding to the attribute description in the original text description can be uniquely identified, so different objects can be further completely distinguished through the attribute description. Preferably, an object has a category description and one or more attribute descriptions.

[0022] Category description and attribute description are key information for understanding and analyzing the original text description. Category description provides the basic classification of objects, while attribute description provides detailed characteristics or status of each object. The purpose of extracting category description and attribute description from the original text description is to mark the category description and attribute description, so that they can be extracted from the entire original text description for subsequent feature fusion.

[0023] Step S102: Based on the parsing result, a revised text description corresponding to the original text description is obtained, where the revised text description does not include an attribute description of each of the objects.

[0024] In this embodiment, since the category description and attribute description of an object will be fused and the category description of each object will be replaced with the corresponding fused feature in subsequent steps, a revised text description that does not include the attribute description of each object can be directly obtained through step S102, which not only facilitates subsequent processing but also ensures that the revised text description does not include redundant attribute description information.

[0025] Step S103: Based on the analysis result, the category description and the attribute description of each object are fused respectively to obtain a fusion feature of each object.

[0026] The fused feature represents the identification information of each object, that is, the corresponding object in the original text description can be directly and uniquely identified through the fused feature, that is, the relevant information corresponding to each object can be associated through step S103.

[0027] In this embodiment, the category description and attribute description of an object are input into a pre-trained fusion model, wherein the category description is used to distinguish objects to a certain extent, while the attribute description can completely distinguish different objects. In this embodiment, by fusing the category description and attribute description of each object, the relevant information of each object can be bound and associated, so that the multimodal can facilitate the understanding of the relevant information of each object during the visual content generation process, thereby avoiding the problem of information crosstalk between multiple attribute descriptions of multiple objects.

[0028] Step S104: sequentially replacing the category description of each object in the revised text description with the corresponding fusion feature to obtain a final text description.

[0029] In this embodiment, since the fused feature represents the identification information of each object, by replacing the category description of each object in the revised text description with the corresponding fused feature, not only the identification information of each object is added to the final text description, but also the revised text description does not include redundant category descriptions of each object that can only distinguish the object to a certain extent (the category descriptions of different objects may be the same). Therefore, the revised text description does not directly include the attribute description and category description of each object separately, so that the multimodal model can better understand the relevant identification information of each object during the generation process, avoiding information crosstalk between different objects.

[0030] Step S105: inputting the final text description into a preset multimodal model to generate visual content for representing the original text description.

[0031] Among them, the multimodal model can adopt an existing model, such as a diffusion model of a UNet structure; the visual content is a static image or a dynamic video. In this embodiment, the multimodal model is a pre-trained AI content generation model. When the final text description is input into a preset multimodal model to generate the visual content, the final text description can be used as a conditional guide for the multimodal model, and the random noise is denoised by the multimodal model to finally generate the visual content used to characterize the original text description. In the embodiment of the present invention, by replacing the category description of each object in the revised text description with the corresponding fusion feature, not only the final text description includes the fusion feature of the category description and attribute description of each object, but also the category description of each object is strongly associated with its corresponding attribute description, thereby avoiding information crosstalk between different objects, and thus achieving the effect of high consistency of objects in the visual content generation character.

[0032] In this embodiment, by obtaining the category description and attribute description of each object, different objects can be preliminarily identified and judged, and further by fusing the category description and attribute description of each object, and replacing the category description of each object in the revised text description that does not include the attribute description of each object with the corresponding fusion feature, it is achieved based on a simple, efficient and versatile association mechanism to effectively associate the category description of each object with its corresponding attribute description, and to a large extent avoid the problem of information crosstalk in the visual content generation task involving multiple objects, effectively improving the quality and stability of the content generation task.

[0033] Specifically, the step S101 includes the following steps: The original text description is structuredly annotated to obtain a structured annotated text; the structured annotated text includes: a background description, an event description, and an object description, wherein the object description includes a category description and an attribute description of each object in the original text description, the background description is used to characterize the event scene, and the event description is used to characterize the association relationship between the various objects.

[0034] Furthermore, the step of performing structured annotation on the original text description to obtain structured annotated text specifically includes: Step A1: performing semantic recognition on the original text description to determine each object in the original text description, a category description and an attribute description of each object, and a background description; In a specific embodiment, a large language model may be used to perform semantic recognition on the original text description to quickly and accurately identify the category description and the attribute description; Step A2: Based on the category description and attribute description of each object, a unique identifier is set for each object, and the identifier can be associated with the category description and attribute description of each object; Step A3: determining the event description of the original text description based on the identifier of each of the objects; Step A4: Based on the background description, the event description and the identifier of each object, a structured annotation text corresponding to the original text description is obtained.

[0035] In this embodiment, a preset category vocabulary can be used to identify category words belonging to the category vocabulary from the original text description as the category description; for example, the category vocabulary includes: male, female, cat, dog, etc. The attribute description is the content about the appearance, clothing, and posture of an object in the original text description. In addition, since the original text description may include multiple objects belonging to the same category description, in order to distinguish each object, a unique identifier is set for each object. The background description is the content about the scene, environment, and location in the original text description. The event description is the content about the interactive behavior between each object in the original text description, which is usually the verb in the original text description. It should also be noted that in order to make the event description clearer, the identifier of each object can also be added to the event description, that is, the event description consists of the identifier of each object and the verb used to characterize the interactive behavior between each object.

[0036] Preferably, a structured data framework in a standard format can be created in advance, and a structured annotation text can be generated based on the structured data framework; wherein the structured data framework includes: a background primary key, an event primary key, and an object primary key; in the process of generating the structured annotation text, the background description is set to the value of the background primary key, the event description is set to the value of the event primary key, and the object description is set to the value of the object primary key; in addition, the identifier of each object is set to each subkey in the object description, and the category description and attribute description of each object are respectively formed into an array as the value of the corresponding subkey. In this embodiment, the use of a fixed format structured annotation text is conducive to the information recognition of the model and the rapid acquisition of corresponding information during text processing.

[0037] In this embodiment, the structured annotation text adopts a nested data structure. For example, the structured annotation text is a JSON object. As an example, the original text is described as: male wearning xx hit female wearning yy in playground, and the structured annotation text is as follows: { Background: "in playground", Action: "character_a hit character_b", Characters: character_a : ["male", "wearing xx"], character_b : ["female", "wearing yy"] } } Among them, the value of the background primary key Background (i.e., the background description) is "in playground", the value of the event primary key Action (i.e., the event description) is "character_a hit character_b", where "character_a" and "character_b" are the identifiers of two objects respectively; the value of the object primary key Characters (i.e., the object description) is "character_a : ["male", "wearing xx"], character_b : ["female", "wearingyy"]", where the category description of the object "character_a" is "male" and the attribute description is "wearing xx", and the category description of the object "character_b" is "female" and the attribute description is "wearing yy".

[0038] In this embodiment, a structured annotation method in the field of natural language processing is used to extract key information from the unstructured original text description, and organize and represent it in a structured format to form a structured data format.

[0039] Specifically, step S102 includes the following steps: Step B1: acquiring the background description and the event description from the structured annotation text; Step B2: identifying identifiers of various objects from the event description, and binding the class description of the corresponding object to the identifier of each object to form a new event description; Step B3: Add the background description to the new event description to form the revised text description.

[0040] In this embodiment, since there may be a situation where multiple objects have the same category description, for example, the category descriptions of object character_a and object character_c are both male, if only the category descriptions of each object are retained in the revised text description, it will result in the inability to distinguish the objects based solely on the category description during subsequent processing. Therefore, it is also necessary to retain the identifiers of each object in the revised text description. In particular, if each category description corresponds to only one object in the original text description, each identified identifier can be replaced with the corresponding category description in the above step B2 to form a new event description. At this time, it is not necessary to retain the identifiers of each object in the revised text description. For example, based on the above-mentioned example, a new event description "male hit female" can be formed based on the event description "character_a hit character_b", and then the revised text description "male hit female in playground" is obtained.

[0041] Specifically, step S103 includes the following steps: Step C1: respectively divide the category description of the i-th object in the revised text description into the number of attribute descriptions of the i-th object Copy to obtain a new revised text description; where i∈M, M is the number of objects in the original text description, and is a positive integer; For example, based on the example mentioned above, if the object "character_a" has three attribute descriptions and the object "character_b" has two attribute descriptions, the new revised text description is: "male male male hit female female in playground".

[0042] Step C2: sequentially traverse the category descriptions of each object in the new revised text description, and sequentially input the j-th category description and the j-th attribute description of the currently traversed i-th object into the preset fusion model to obtain the j-th category description and the j-th attribute description of the currently traversed i-th object. fusion features; where j∈[1, ].

[0043] It should be noted that, in practical applications, a category description of an object can be fused with multiple attribute descriptions to obtain a fused feature of the object; however, when there are many attribute descriptions of an object, the fusion effect of the fusion model will deteriorate. In this embodiment, in order to reduce the training complexity of the fusion model and improve the fusion accuracy of the fusion process, so as to better fuse the category description and attribute description of an object, the fusion model used can only use two fully connected networks (MLPs) to fuse the two input contents; at this time, only one category description and one attribute description of an object can be input into the fusion model. If an object has N attribute descriptions (N is a positive integer), each attribute description of the object needs to be fused with the category description of the object in turn to obtain N fused features.

[0044] At this time, the step S104 specifically includes: The new revised text description of the i-th object The category descriptions are replaced by the corresponding fused features to obtain the final text description.

[0045] Furthermore, the step C1 specifically includes the following steps: Step C11: tokenizing the revised text description to obtain a text token sequence, and marking the category tokens corresponding to the category description of each object from the text token sequence; Step C12: tokenizing the attribute description of each object to obtain an attribute token sequence of each object; Step C13: The category tokens of the i-th object in the text token sequence are respectively divided into the number of attribute tokens in the attribute token sequence of the i-th object. Make a copy to get a new sequence of text tokens.

[0046] Among them, tokenization is a processing method in natural language processing. Tokenization is to convert text into a series of tokens to form a token sequence; these tokens can be words, subwords, characters or other text fragments. Through tokenization, the revised text description is broken down into smaller units to facilitate processing and understanding by the subsequent text encoder.

[0047] Furthermore, the step C2 specifically includes the following steps: Step C21: Input the new text token sequence into a preset text encoder to obtain a text embedding vector group, and mark the text tokens related to the i-th object from the text embedding vector group. The corresponding class tokens Category embedding vectors; Step C22: Input the attribute token sequence of each object into the text encoder to obtain the attribute embedding vector group of each object; wherein the attribute embedding vector group of the i-th object includes the attribute embedding vector group of the i-th object. The attribute tokens correspond to attribute embedding vectors; Step C23: traverse the category embedding vectors of each object in the text embedding vector group in sequence, and input the j-th category embedding vector and the j-th attribute embedding vector of the currently traversed i-th object into the preset fusion model in order to obtain the i-th object of the currently traversed fused feature embedding vector; where j∈[1, ].

[0048] Among them, the text encoder is a pre-trained language model or a custom encoder network, such as a CLIP network. The text encoder encodes the input token sequence and converts it into a vector representation of the corresponding space.

[0049] At this time, the step S104 specifically includes: The text is embedded in the i-th object of the vector group The category embedding vectors are replaced by the corresponding The feature embedding vectors are fused to obtain a new text embedding vector group.

[0050] The step S105 specifically includes: The new text embedding vector group is input into a preset multimodal model to generate an image or video for representing the original text description.

[0051] like Figure 2 As shown, in this embodiment, the text description is first structured labeled to extract the category description and attribute description of each object from the text description; then, based on the structured labeling result, the text description is tokenized to accurately determine the category token and attribute token of each object; then, the token sequence including multiple tokens obtained after the tokenization is input into the text encoder for encoding, thereby obtaining a text embedding vector; then, the category embedding vector and attribute embedding vector of each object in the text embedding vector are fused and the fused embedding vector replaces the original category embedding vector; finally, the replaced text embedding vector is input into the multimodal model to obtain an image corresponding to the text description. In the above manner, all descriptions belonging to an object can be effectively clustered and fused, avoiding the crosstalk problem between different descriptions of different objects, and improving the stability and accuracy of the visual content generation task.

[0052] Embodiment 2 The embodiment of the present invention provides a visual content generation device for multiple objects, such as Figure 3 As shown, the device specifically includes the following components: The parsing module 301 is used to parse the original text description containing multiple objects to obtain a parsing result, wherein the parsing result at least includes: a category description and an attribute description of each of the objects; A revision module 302, configured to obtain a revised text description corresponding to the original text description based on the parsing result, wherein the revised text description does not include an attribute description of each of the objects; A fusion module 303 is used to fuse the category description and the attribute description of each object based on the parsing result to obtain a fusion feature of each object; A replacement module 304 is used to replace the category description of each object in the revised text description with the corresponding fusion feature in turn to obtain a final text description; The processing module 305 is used to input the final text description into a preset multimodal model to generate visual content for representing the original text description.

[0053] Specifically, the parsing module 301 is used to: The original text description is structuredly annotated to obtain a structured annotated text; the structured annotated text includes: a background description, an event description, and an object description, wherein the object description includes a category description and an attribute description of each object in the original text description, the background description is used to characterize the event scene, and the event description is used to characterize the association relationship between the various objects.

[0054] Furthermore, when the parsing module 301 implements the function of structurally annotating the original text description to obtain a structured annotated text, it is specifically used to: Performing semantic recognition on the original text description to determine each object in the original text description, a category description and an attribute description of each object, and a background description; Based on the category description and the attribute description of each of the objects, a unique identifier is set for each of the objects, and the identifier can be associated with the category description and the attribute description of each of the objects; Based on the identifier of each of the objects, determining an event description of the original text description; Based on the background description, the event description and the identifier of each of the objects, a structured annotated text corresponding to the original text description is obtained.

[0055] Furthermore, the revision module 302 is specifically used for: Acquire the background description and the event description from the structured annotation text; Identify the identifiers of each object from the event description, and bind the category description of the corresponding object to the identifier of each object to form a new event description; The background description is added to the new event description to form the revised text description.

[0056] Furthermore, the fusion module 303 is specifically used for: The category description of the i-th object in the revised text description is respectively divided into the number of the attribute description of the i-th object Make a copy to get the new revised text description; The category descriptions of each object in the new revised text description are sequentially traversed, and the j-th category description and the j-th attribute description of the currently traversed i-th object are sequentially input into the preset fusion model to obtain the currently traversed i-th object fusion features; where j∈[1, ].

[0057] At this time, the replacement module 304 is specifically used to: The new revised text description of the i-th object The category descriptions are replaced by the corresponding fused features to obtain the final text description.

[0058] Furthermore, the fusion module 303 is also used for: Tokenizing the revised text description to obtain a text token sequence, and marking the category tokens corresponding to the category description of each object from the text token sequence; Tokenizing the attribute description of each object respectively to obtain an attribute token sequence of each object; The category tokens of the i-th object in the text token sequence are respectively divided into the number of attribute tokens in the attribute token sequence of the i-th object. Make a copy to get a new text token sequence; The new text token sequence is input into a preset text encoder to obtain a text embedding vector group, and the text embedding vector group corresponding to the i-th object is marked respectively. Category embedding vectors; The attribute token sequence of each object is input into the text encoder to obtain the attribute embedding vector group of each object; wherein the attribute embedding vector group of the i-th object includes attribute embedding vectors; The category embedding vectors of each object in the text embedding vector group are traversed in sequence, and the j-th category embedding vector and the j-th attribute embedding vector of the currently traversed i-th object are input into the preset fusion model in order to obtain the i-th object of the current traversal. fused feature embedding vector; where j∈[1, ].

[0059] At this time, the replacement module 304 is specifically used to: The text is embedded in the i-th object of the vector group The category embedding vectors are replaced by the corresponding The feature embedding vectors are fused to obtain a new text embedding vector group.

[0060] The processing module 305 is specifically used for: The new text embedding vector group is input into a preset multimodal model to generate an image or video for representing the original text description.

[0061] Embodiment 3 This embodiment also provides a computer device, such as a smart phone, tablet computer, laptop computer, desktop computer, rack server, blade server, tower server or cabinet server (including an independent server or a server cluster composed of multiple servers) that can execute programs. Figure 4 As shown, the computer device 40 of this embodiment includes at least but not limited to: a memory 401 and a processor 402 that can communicate with each other via a system bus. It should be noted that Figure 4 Only computer device 40 is shown with components 401 - 402 , but it should be understood that implementing all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0062] In this embodiment, the memory 401 (i.e., readable storage medium) includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 401 can be an internal storage unit of the computer device 40, such as a hard disk or memory of the computer device 40. In other embodiments, the memory 401 can also be an external storage device of the computer device 40, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 40. Of course, the memory 401 can also include both the internal storage unit of the computer device 40 and its external storage device. In this embodiment, the memory 401 is generally used to store the operating system and various application software installed on the computer device 40. In addition, the memory 401 can also be used to temporarily store various types of data that have been output or are to be output.

[0063] In some embodiments, the processor 402 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 402 is generally used to control the overall operation of the computer device 40 .

[0064] Specifically, in this embodiment, the processor 402 is used to execute the program of the visual content generation method for multiple objects stored in the memory 401, and the program of the visual content generation method for multiple objects implements the following steps when being executed: Parsing the original text description containing multiple objects to obtain a parsing result, wherein the parsing result at least includes: a category description and an attribute description of each of the objects; Based on the parsing result, obtaining a revised text description corresponding to the original text description, wherein the revised text description does not include an attribute description of each of the objects; Based on the analysis result, the category description and the attribute description of each object are fused to obtain a fusion feature of each object; replacing the category description of each object in the revised text description with the corresponding fusion feature in turn to obtain a final text description; The final text description is input into a preset multimodal model to generate visual content for representing the original text description.

[0065] The specific implementation process of the above method steps can be found in Example 1, and this embodiment will not be repeated here.

[0066] Embodiment 4 This embodiment also provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, a server, an App application store, etc., on which a computer program is stored. When the computer program is executed by a processor, the following method steps are implemented: Parsing the original text description containing multiple objects to obtain a parsing result, wherein the parsing result at least includes: a category description and an attribute description of each of the objects; Based on the parsing result, obtaining a revised text description corresponding to the original text description, wherein the revised text description does not include an attribute description of each of the objects; Based on the analysis result, the category description and the attribute description of each object are fused to obtain a fusion feature of each object; replacing the category description of each object in the revised text description with the corresponding fusion feature in turn to obtain a final text description; The final text description is input into a preset multimodal model to generate visual content for representing the original text description.

[0067] The specific implementation process of the above method steps can be found in the first embodiment, and this embodiment will not be repeated here.

[0068] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0069] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0070] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0071] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for generating visual content for multiple objects, characterized in that: The method comprises: Parsing the original text description containing multiple objects to obtain a parsing result, wherein the parsing result at least includes: a category description and an attribute description of each of the objects; Based on the parsing result, obtaining a revised text description corresponding to the original text description, wherein the revised text description does not include an attribute description of each of the objects; Based on the analysis result, the category description and the attribute description of each object are fused to obtain a fusion feature of each object; replacing the category description of each object in the revised text description with the corresponding fusion feature in turn to obtain a final text description; The final text description is input into a preset multimodal model to generate visual content for representing the original text description.

2. The method for generating visual content for multiple objects according to claim 1, characterized in that: The original text description containing multiple objects is parsed to obtain parsing results, which specifically include: The original text description is structuredly annotated to obtain a structured annotated text; the structured annotated text includes: a background description, an event description, and an object description, wherein the object description includes a category description and an attribute description of each object in the original text description, the background description is used to characterize the event scene, and the event description is used to characterize the association relationship between the various objects.

3. The method for generating visual content for multiple objects according to claim 2, characterized in that: The structural annotation of the original text description to obtain the structured annotation text specifically includes: Performing semantic recognition on the original text description to determine each object in the original text description, a category description and an attribute description of each object, and a background description; Based on the category description and the attribute description of each of the objects, a unique identifier is set for each of the objects, and the identifier can be associated with the category description and the attribute description of each of the objects; Based on the identifier of each of the objects, determining an event description of the original text description; Based on the background description, the event description and the identifier of each of the objects, a structured annotated text corresponding to the original text description is obtained.

4. The method for generating visual content for multiple objects according to claim 3, characterized in that: The obtaining, based on the parsing result, a revised text description corresponding to the original text description specifically includes: Acquire the background description and the event description from the structured annotation text; Identify the identifiers of each object from the event description, and bind the category description of the corresponding object to the identifier of each object to form a new event description; The background description is added to the new event description to form the revised text description.

5. The method for generating visual content for multiple objects according to any one of claims 1 to 4, characterized in that: Based on the analysis result, the category description and the attribute description of each object are respectively fused to obtain the fusion feature of each object, specifically including: The category description of the i-th object in the revised text description is respectively divided into the number of the attribute description of the i-th object Make a copy to get the new revised text description; The category descriptions of each object in the new revised text description are sequentially traversed, and the j-th category description and the j-th attribute description of the currently traversed i-th object are sequentially input into the preset fusion model to obtain the i-th object currently traversed. fusion features; where j∈[1, ].

6. The method for generating visual content for multiple objects according to claim 5, characterized in that: The step of sequentially replacing the category description of each object in the revised text description with the corresponding fusion feature to obtain a final text description specifically includes: The new revised text description of the i-th object The category descriptions are replaced by the corresponding fused features to obtain the final text description.

7. The method for generating visual content for multiple objects according to any one of claims 1 to 4, characterized in that: Based on the analysis result, the category description and the attribute description of each object are respectively fused to obtain the fusion feature of each object, specifically including: Tokenizing the revised text description to obtain a text token sequence, and marking the category tokens corresponding to the category description of each object from the text token sequence; Tokenizing the attribute description of each object respectively to obtain an attribute token sequence of each object; The category tokens of the i-th object in the text token sequence are respectively divided into the number of attribute tokens in the attribute token sequence of the i-th object. Make a copy to get a new text token sequence; The new text token sequence is input into a preset text encoder to obtain a text embedding vector group, and the text embedding vector group corresponding to the i-th object is marked respectively. Class embedding vectors; The attribute token sequence of each object is input into the text encoder to obtain the attribute embedding vector group of each object; wherein the attribute embedding vector group of the i-th object includes attribute embedding vectors; The category embedding vectors of each object in the text embedding vector group are traversed in sequence, and the j-th category embedding vector and the j-th attribute embedding vector of the currently traversed i-th object are input into the preset fusion model in order to obtain the i-th object of the current traversal. fused feature embedding vector; where j∈[1, ].

8. The method for generating visual content for multiple objects according to claim 7, characterized in that: The step of sequentially replacing the category description of each object in the revised text description with the corresponding feature description to obtain a final text description specifically includes: The text is embedded in the i-th object of the vector group The category embedding vectors are replaced by the corresponding The fused feature embedding vectors are used to obtain a new text embedding vector group; The step of inputting the final text description into a preset multimodal model to generate visual content for representing the original text description specifically includes: The new text embedding vector group is input into a preset multimodal model to generate an image or video for representing the original text description.

9. A visual content generation device for multiple objects, characterized in that: The device comprises: A parsing module, used to parse the original text description containing multiple objects to obtain a parsing result, wherein the parsing result at least includes: a category description and an attribute description of each of the objects; A revision module, configured to obtain a revised text description corresponding to the original text description based on the parsing result, wherein the revised text description does not include an attribute description of each of the objects; A fusion module, based on the parsing result, respectively fuses the category description and the attribute description of each object to obtain a fusion feature of each object; A replacement module, used to replace the category description of each object in the revised text description with the corresponding fusion feature in turn to obtain a final text description; A processing module is used to input the final text description into a preset multimodal model to generate visual content for representing the original text description.

10. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 8 when executing the computer program.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images