Method, system, device, storage medium and product for constructing image and text data sets

Abstract

The present application discloses a method, system, device, storage medium and product for constructing a graphic-text dataset, which relates to the field of text processing technology, including: extracting entities, entity attributes and relationships between entities based on the original text corresponding to each original image, and constructing a foreground feature knowledge base based on the entities, entity attributes and relationships between entities; extracting background description samples from each original text, and constructing a background feature knowledge base based on the background description samples; constructing text based on the foreground feature knowledge base and the background feature knowledge base to obtain target prompt text; generating a corresponding target image using the target prompt text, and forming a graphic-text pair based on the target image and the corresponding target prompt text for constructing a graphic-text dataset. The technical solution of the present application aims to solve the technical problem of how to construct a high-quality text-generated graphic model prompt text so as to construct a high-quality graphic-text dataset.

Description

Technical Field

[0001] The present application relates to the field of text processing technology, and in particular to a method, system, device, storage medium and product for constructing a graphic and text data set. Background Art

[0002] A text-based graph model is a generative AI model that can automatically generate semantically associated images based on user-provided text descriptions. These models typically employ diffusion models, generative adversarial networks, or autoregressive architectures, and are trained on massive amounts of text-based image data to learn the mapping relationship between text and visual features. For example, the MidJourney model excels at generating images with a strong artistic style, making it suitable for illustrations and creative design. The diffusion model, leveraging latent space diffusion technology, supports user-defined training and generates high-resolution images. The DALL-E 3 model significantly improves its ability to restore details in complex text by integrating CLIP semantic alignment with the diffusion model. However, despite significant technological advancements, the core performance of text-based graph models remains highly dependent on the quality of the input text. Specifically, if the prompt words lack precision, logic, or detailed descriptions, the text-based graph model may generate images that deviate from expectations due to semantic ambiguity or insufficient information, making it difficult to construct high-quality text-based image datasets.

[0003] Therefore, how to construct a high-quality text-to-text model prompt text so as to construct a high-quality text-to-text dataset is a technical problem that has yet to be solved by those skilled in the art. Summary of the Invention

[0004] The main purpose of this application is to provide a method, system, device, storage medium and product for constructing a graphic and text dataset, aiming to solve the technical problem of how to construct a high-quality text-based graphic model prompt text so as to construct a high-quality graphic and text dataset.

[0005] To achieve the above objectives, this application proposes a method for constructing a graphic and text dataset, the method comprising:

[0006] Extracting entities, entity attributes, and inter-entity relationships based on the original text corresponding to each original image, and building a foreground feature knowledge base based on the entities, the entity attributes, and the inter-entity relationships;

[0007] Extracting background description samples from each of the original texts, and building a background feature knowledge base based on the background description samples;

[0008] Constructing text based on the foreground feature knowledge base and the background feature knowledge base to obtain target prompt text;

[0009] A corresponding target image is generated using the target prompt text, and a picture-text pair is formed based on the target image and the corresponding target prompt text to construct the picture-text dataset.

[0010] In one embodiment, before the step of extracting entities, entity attributes, and inter-entity relationships based on the original text corresponding to each original image, the method for constructing the image-text dataset further includes:

[0011] Perform image segmentation and classification labeling on each original image to obtain various visual features;

[0012] Based on a preset semantic description knowledge base and each of the visual features, original text corresponding to each of the original images is generated.

[0013] In one embodiment, the method further comprises:

[0014] Perform data cleaning and spelling correction on the original text corresponding to each original image to obtain preprocessed text;

[0015] The step of extracting entities, entity attributes, and relationships between entities based on the original text corresponding to each original image includes:

[0016] Based on the preprocessed texts, entities, entity attributes and relationships between entities are extracted.

[0017] In one embodiment, the step of extracting entities, entity attributes, and relationships between entities based on the original text corresponding to each original image includes:

[0018] Performing semantic segmentation on the original texts corresponding to the original images to obtain the words corresponding to the original texts;

[0019] Entities in each of the words are identified through a large language model, and based on the large language model and the original texts, the relationship between entities and the entity attributes corresponding to each of the entities are obtained, wherein the entity attributes include: functional attributes, shape attributes, size attributes, color attributes, pattern attributes, texture attributes, material attributes, situation attributes, opacity attributes, direction attributes, action attributes, and text attributes.

[0020] In one embodiment, after the step of obtaining the relationship between entities and the entity attributes corresponding to each entity based on the large language model and each of the original texts, the method further includes:

[0021] Classifying each of the entities to obtain an entity category corresponding to each of the entities;

[0022] The entities, the relationships between the entities, the entity attributes corresponding to the entities, and the entity categories corresponding to the entities are stored.

[0023] In one embodiment, the step of constructing a background feature knowledge base based on the background description samples includes:

[0024] Extracting various background attributes from the background description sample, the background attributes including: scene attributes, background subject attributes, picture style attributes, composition attributes, lighting attributes, color palette attributes, texture attributes, depth of field attributes, theme attributes, emotion attributes, count attributes, and perspective attributes;

[0025] A background feature knowledge base is constructed according to each of the background attributes.

[0026] In one embodiment, the step of constructing text based on the foreground feature knowledge base and the background feature knowledge base to obtain target prompt text includes:

[0027] When it is detected that the original prompt text does not exist, determining a first random feature in the foreground feature knowledge base and determining a second random feature in the background feature knowledge base, the original prompt text being the previously received prompt text;

[0028] A text is constructed based on the first random feature and the second random feature through a large language model to obtain a target prompt text.

[0029] In one embodiment, the step of constructing text based on the foreground feature knowledge base and the background feature knowledge base to obtain target prompt text includes:

[0030] When the original prompt text is detected, the foreground feature knowledge base is filtered using a large language model to select a first target feature corresponding to the original prompt text, and a foreground description text is constructed based on the original prompt text and the first target feature using the large language model;

[0031] screening a second target feature corresponding to the original prompt text in the background feature knowledge base by using the large language model, and constructing a background description text based on the original prompt text and the second target feature by using the large language model;

[0032] A target prompt text is obtained according to the foreground description text and the background description text.

[0033] In one embodiment, the step of obtaining the target prompt text according to the foreground description text and the background description text includes:

[0034] splicing the foreground description text and the background description text by using the large language model to obtain an intermediate prompt text;

[0035] According to preset word replacement rules, the words to be replaced in the intermediate prompt text are determined, and target words are generated based on the words to be replaced through the large language model, and the words to be replaced are replaced based on the target words to obtain a target prompt text. The word replacement rules include at least: adjective replacement, position relationship replacement and quantity relationship replacement.

[0036] In one embodiment, the step of generating a corresponding target image using the target prompt text, and forming an image-text pair based on the target image and the corresponding target prompt text, includes:

[0037] Inputting the target prompt text into a preset text-based image model to obtain a target image;

[0038] Inputting the target image and the target prompt text into a comparative language image pre-training model to obtain the similarity between the target image and the target prompt text;

[0039] When it is detected that the similarity is greater than or equal to a preset similarity threshold, the target image and the target prompt text are constructed into an image-text pair, and the image-text pair is stored.

[0040] In one embodiment, after the step of constructing the target image and the target prompt text into an image-text pair, the method further includes:

[0041] Among various preset image types, a target image type corresponding to the target image is determined, and the image-text pairs are divided into an image-text pair data set corresponding to the target image type.

[0042] In addition, to achieve the above-mentioned purpose, the present application also proposes a system for constructing a graphic and text dataset, wherein the system comprises:

[0043] a foreground feature knowledge base construction module, configured to extract entities, entity attributes, and inter-entity relationships based on the original text corresponding to each original image, and to construct a foreground feature knowledge base based on the entities, the entity attributes, and the inter-entity relationships;

[0044] A background feature knowledge base construction module is used to extract background description samples from each of the original texts and construct a background feature knowledge base based on the background description samples;

[0045] A text construction module, configured to construct text based on the foreground feature knowledge base and the background feature knowledge base to obtain a target prompt text;

[0046] The image-text pair construction module is used to generate a corresponding target image using the target prompt text, and form an image-text pair based on the target image and the corresponding target prompt text to construct the image-text dataset.

[0047] In addition, to achieve the above-mentioned purpose, the present application also proposes an electronic device, which includes: a memory, a processor, and a computer program stored on the memory and runnable on the processor, wherein the computer program is configured to implement the steps of the method for constructing a graphic and text data set as described above.

[0048] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium, and when the computer program is executed by the processor, the steps of the method for constructing the graphic and text data set as described above are implemented.

[0049] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps of the method for constructing a graphic and text dataset as described above.

[0050] In an embodiment of the present application, entities, entity attributes and inter-entity relationships are extracted based on the original text corresponding to each original image, and a foreground feature knowledge base is constructed based on the entities, entity attributes and inter-entity relationships. Based on the publicly available images, entities, entity attributes and inter-entity relationships between each entity in the image can be extracted, and the extracted information is used as foreground features to construct a foreground feature knowledge base; then background description samples are extracted from each original text, and a background feature knowledge base is constructed based on the background description samples. Based on the publicly available images, background features in the image can be extracted, and a background feature knowledge base is constructed based on the background features; then text is constructed based on the foreground feature knowledge base and the background feature knowledge base to obtain target prompt text, and known foreground features and background features can be combined to obtain semantically rich and accurate target prompt text; then the target prompt text is used to generate a corresponding target image, and a picture-text pair is formed based on the target image and the corresponding target prompt text for constructing a picture-text dataset, and a high-quality picture-text dataset can be obtained based on the accurate target prompt text.

[0051] In this application, detailed descriptions of objects, overall environment, style, and other information can be extracted from publicly available images. Based on this information, a foreground feature knowledge base and a background feature knowledge base can be established. A large language model can then be used to generate rich prompt text based on the foreground and background feature knowledge bases, thereby constructing high-quality text-based image model prompt text. Furthermore, based on this high-quality text-based image model prompt text, a high-quality target image can be obtained, thereby constructing a high-quality image-text dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0054] Figure 1 This is a flow chart of the first embodiment of the method for constructing a graphic and text dataset of the present application;

[0055] Figure 2 This is a schematic diagram of the target prompt text construction process of an embodiment of the method for constructing a graphic and text dataset of the present application;

[0056] Figure 3 This is a schematic diagram of the image-text pair construction process of an embodiment of the method for constructing an image-text dataset of the present application;

[0057] Figure 4 This is a schematic diagram of the module structure of the system for constructing a graphic and text data set according to an embodiment of the present application;

[0058] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the method for constructing a graphic and text data set in an embodiment of the present application.

[0059] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0060] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0061] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0062] It is understandable that a text-based graph model is a generative AI model that can automatically generate semantically associated images based on user-provided text descriptions. Such models typically use diffusion models, generative adversarial networks, or autoregressive architectures, and are trained on massive amounts of text-image data to learn the mapping relationship between text and visual features. For example, the MidJourney model excels at generating images with a strong artistic style, suitable for illustrations and creative design; the diffusion model, with its latent space diffusion technology, supports user-defined training and generates high-resolution images; and the DALL-E 3 model significantly improves its ability to restore details of complex text by integrating CLIP semantic alignment and diffusion models. However, despite significant technological advances, the core performance of the text-based graph model is still highly dependent on the quality of the input text. In other words, if the prompt words lack precision, logic, or detailed descriptions, the text-based graph model may generate images that deviate from expectations due to semantic ambiguity or insufficient information.

[0063] Therefore, how to construct a text-based graph model prompt text with higher text quality is a technical problem that has yet to be solved by those skilled in the art.

[0064] To solve the above problems, the present application extracts entities, entity attributes and inter-entity relationships based on the original text corresponding to each original image, and constructs a foreground feature knowledge base based on the entities, entity attributes and inter-entity relationships. Based on the publicly available images, the entities, entity attributes and inter-entity relationships between each entity in the image can be extracted, and the extracted information is used as the foreground feature to construct a foreground feature knowledge base; then, background description samples are extracted from each original text, and a background feature knowledge base is constructed based on the background description samples. Based on the publicly available images, background features in the image can be extracted, and a background feature knowledge base is constructed based on the background features; then, text is constructed based on the foreground feature knowledge base and the background feature knowledge base to obtain target prompt text, and known foreground features and background features can be combined to obtain semantically rich and accurate target prompt text; then, the target prompt text is used to generate a corresponding target image, and a picture-text pair is formed based on the target image and the corresponding target prompt text for constructing a picture-text dataset, and a high-quality picture-text dataset can be obtained based on the accurate target prompt text.

[0065] In this application, detailed descriptions of objects, overall environment, style, and other information can be extracted from publicly available images. Based on this information, a foreground feature knowledge base and a background feature knowledge base can be established. A large language model can then be used to generate rich prompt text based on the foreground and background feature knowledge bases, thereby constructing high-quality text-based image model prompt text. Furthermore, based on this high-quality text-based image model prompt text, a high-quality target image can be obtained, thereby constructing a high-quality image-text dataset.

[0066] It should be noted that the execution subject of the method for constructing the image and text dataset of the present application can be an electronic device with data processing, network communication and program running functions, such as a tablet computer, personal computer, server, etc. In the following embodiments, the execution subject is omitted for explanation.

[0067] Based on this, the present application embodiment provides a method for constructing a graphic data set, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the method for constructing a graphic and text dataset of the present application.

[0068] In this embodiment, the method for constructing the image and text dataset includes steps S10 to S40:

[0069] Step S10, extracting entities, entity attributes, and inter-entity relationships based on the original texts corresponding to the respective original images, and constructing a foreground feature knowledge base based on the entities, the entity attributes, and the inter-entity relationships;

[0070] It should be noted that the original image refers to an image that has been made public, the original text refers to the text used to describe the content of the original image, and there is a one-to-one correspondence between the original text and the original image. For example, the original text can be the title, label, and description text corresponding to the original image. An entity refers to an object in an image that has independent meaning or is identifiable, and entity attributes are information descriptions of the entity. The relationship between entities refers to the relationship between entities. For example, in a feasible implementation, the relationship between entities includes at least a spatial position relationship and an interaction relationship. Among them, the spatial position relationship is the position information of the spatial dimension presented by multiple entities in the original image, and the interaction relationship refers to the information of the action dimension presented by multiple entities in the original image. For example, the interaction relationship can be "little boy->wearing->shirt, woman->holding->flag".

[0071] It is understandable that foreground objects are often the core focus of an image. Therefore, accurately understanding the features of foreground objects is crucial for generating image details that meet the requirements. Different objects vary greatly in attributes such as type, shape, and color. Separately constructing these foreground features facilitates targeted storage and access to these foreground features. For example, if you need to generate an image containing flowers, you need to accurately know the foreground features such as the flower type, petal shape, and color gradient to make the generated flowers appear lifelike. Therefore, in this embodiment, the original images and the original text corresponding to the original images can be directly obtained from open source datasets. The original text corresponding to each original image can then be analyzed to extract the entities, entity attributes, and the spatial positional relationships and interactions between the entities in the original text. The entities, entity attributes, and the spatial positional relationships and interactions between the entities can then be summarized to obtain a foreground feature knowledge base. This allows the generation of prompt text to guide the text-based graph model in generating foreground objects based on the features in the foreground feature knowledge base.

[0072] Step S20, extracting background description samples from each of the original texts, and building a background feature knowledge base based on the background description samples;

[0073] It is understandable that the background description sample refers to the text in the original text that describes the background information of the image. In an image, the background information determines the overall atmosphere and style of the image. For example, indoor scenes, outdoor scenes, the brightness and darkness of light and shadow, and the cold and warm tones will generate images with different atmospheres and styles. Therefore, constructing a background knowledge base separately can better control the image style from a macro perspective and meet the needs of diverse scenarios. For example, to generate a retro-style interior home picture, it is necessary to guide image generation based on information such as retro color matching and typical interior layout in the background knowledge base. Therefore, in this embodiment, the background description samples can also be extracted from the original text corresponding to each original image through a large language model, and then the background features represented by the background description samples can be extracted, and the various background features can be summarized to obtain a background feature knowledge base for subsequent calls.

[0074] Step S30, constructing text based on the foreground feature knowledge base and the background feature knowledge base to obtain target prompt text;

[0075] In this embodiment, when a specific image needs to be generated through a text-based graph model, rich prompt text can be constructed based on the features in the foreground feature knowledge base and the features in the background feature knowledge base, and the constructed prompt text can be used as the target prompt text to generate a specific image based on the target prompt text.

[0076] In an embodiment of the present application, entities, entity attributes and inter-entity relationships are extracted based on the original text corresponding to each original image, and a foreground feature knowledge base is constructed based on the entities, entity attributes and inter-entity relationships. Based on the publicly available images, entities, entity attributes and inter-entity relationships between each entity in the image can be extracted, and the extracted information is used as foreground features to construct a foreground feature knowledge base; then background description samples are extracted from each original text, and a background feature knowledge base is constructed based on the background description samples. Based on the publicly available images, background features in the image can be extracted, and a background feature knowledge base is constructed based on the background features; then text is constructed based on the foreground feature knowledge base and the background feature knowledge base to obtain target prompt text. Known foreground features and background features can be combined to obtain semantically rich and accurate target prompt text.

[0077] In this embodiment, detailed descriptions of objects, the overall environment, and the style of the original image can be extracted. Based on this information, a foreground feature knowledge base and a background feature knowledge base are established. A large language model can then be used to generate rich prompt text based on the foreground and background feature knowledge bases, thereby constructing high-quality textual image model prompt text.

[0078] Step S40 : generating a corresponding target image using the target prompt text, and forming an image-text pair based on the target image and the corresponding target prompt text, so as to construct the image-text dataset.

[0079] It can be understood that the target image refers to the image generated by the text-based graph model based on the target prompt text.

[0080] In this embodiment, after obtaining the target prompt text, an image that meets the requirements of the target prompt text can be generated based on the text-image model, and then the target prompt text and its corresponding target image can be combined into a text-image pair, and then a text-image dataset can be constructed through multiple text-image pairs.

[0081] In this embodiment, the application can extract detailed descriptions of objects, overall environment, style, and other information from the original image, and establish a foreground feature knowledge base and a background feature knowledge base based on the extracted information. A large language model can then be used to generate rich prompt text based on the foreground feature knowledge base and the background feature knowledge base, thereby constructing high-quality text-based image model prompt text. Furthermore, based on the high-quality text-based image model prompt text, a high-quality target image can be obtained, thereby constructing a high-quality image-text dataset.

[0082] Furthermore, based on the first embodiment of the method for constructing a graphic and text dataset of the present application, a second embodiment of the method for constructing a graphic and text dataset of the present application is proposed.

[0083] In this embodiment, before the above step S10, the method further includes:

[0084] Step S100, performing image segmentation and classification labeling on each original image to obtain various visual features;

[0085] It should be noted that image segmentation refers to dividing an image into multiple objects, while classification labeling refers to classifying the segmented objects according to existing categories and labeling the corresponding categories on the objects. Visual features refer to the individual objects after category labeling.

[0086] Step S200 : generating original text corresponding to each of the original images based on a preset semantic description knowledge base and the visual features.

[0087] It should be noted that the semantic description knowledge base refers to the template of semantic description. In this embodiment, the visual features can be filled into the semantic description knowledge base, and the templates filled with visual features can be sorted and filled through the large language model, so that the original text corresponding to each original image can be obtained.

[0088] It is understandable that the method for obtaining original text in the first embodiment is to obtain it directly from a public data source. However, in public data sources, original images are generally not accompanied by original text. Therefore, the method of directly obtaining original text from a public data source in the first embodiment will result in a small number of samples, resulting in insufficient content in the foreground feature knowledge base and background feature knowledge base.

[0089] In this embodiment, only the original image needs to be obtained. Specifically, the original image can be processed to automatically generate the original text corresponding to the original image. Therefore, even in scenarios where the original text sample is small, the present application can automatically generate the original text, thereby facilitating the enrichment of the background feature knowledge base and the foreground feature knowledge base, and further facilitating the construction of rich text prompts.

[0090] Furthermore, based on the first embodiment of the method for constructing a graphic and text dataset of the present application, a third embodiment of the method for constructing a graphic and text dataset of the present application is proposed.

[0091] In this embodiment, the method further includes:

[0092] Step S300 , performing data cleaning and spelling correction on the original text corresponding to each original image to obtain a pre-processed text;

[0093] Understandably, raw text obtained from open-source databases often contains noise, such as garbled characters, special symbols, and irrelevant punctuation. Therefore, data cleaning is necessary to remove this noise from the raw text. Error correction refers to correcting spelling errors in the raw text to improve its accuracy. Preprocessed text refers to the raw text obtained after data cleaning and spelling correction.

[0094] Based on this, the above step S10 includes:

[0095] Step S101 : extracting entities, entity attributes and relationships between entities based on the preprocessed texts.

[0096] It can be understood that in this embodiment, more accurate entities, entity attributes, and spatial position relationships can be obtained based on the preprocessed text with higher accuracy, thereby improving the text quality of the target prompt text.

[0097] Furthermore, based on the first embodiment of the method for constructing a graphic and text dataset of the present application, a fourth embodiment of the method for constructing a graphic and text dataset of the present application is proposed.

[0098] In this embodiment, the above step S10 includes:

[0099] Step S102, performing semantic segmentation on the original text corresponding to each original image to obtain each word corresponding to each original text;

[0100] Step S103: Identify entities in each of the words through a large language model, and obtain the relationship between entities and the entity attributes corresponding to each of the entities based on the large language model and the original texts. The entity attributes include: functional attributes, shape attributes, size attributes, color attributes, pattern attributes, texture attributes, material attributes, situation attributes, opacity attributes, direction attributes, action attributes, and text attributes.

[0101] For example, a large language model can be used to segment the original text, breaking down the continuous text sequence into individual words or phrases based on semantics. For example, for the original text "A red, round balloon is floating in the air," the segmentation yields words such as "a," "red," "round," "balloon," "in the air," and "floating," which can be used for subsequent feature extraction and analysis.

[0102] Then, a large language model can be used to perform entity recognition on the segmented text, identifying nouns or noun phrases describing objects as entities. For example, the word "balloon" in the original text above is an entity.

[0103] Then, the entity attributes in the original text can be extracted through the large language model based on the definition of each entity attribute.

[0104] In a feasible implementation, the definitions of the attributes of each entity are as follows:

[0105] Functional attributes: Through semantic analysis, the function of entities in the original text in the image is understood in combination with the context. For example, for the text "There is a desk lamp on the table for lighting", the function of the desk lamp is identified as "lighting"; if the text describes "A worker is hammering a nail with a hammer", the function of the hammer is "hammering".

[0106] Shape attribute: Extract shape description words from the original text to determine the shape of the entity. For example, the entity can be circular, square, or triangular. For example, if the original text is "There is a rectangular painting hanging on the wall", it can be determined that the shape of the painting is "rectangular".

[0107] Size attribute: Look for words related to size comparison in the original text, such as "big," "small," "huge," and "tiny." Combined with common object size references, determine the size of the object relative to other objects or general cognition. For example, if the original text is "There is a big tree beside a small pond," it can be known that the tree is "big" and the pond is "small." If the original text is "Holding a tiny chip in the hand," it can be seen that the chip is "tiny."

[0108] Color attributes: Extract color vocabulary from the original text, including pure color descriptions such as "red," "blue," and "green," as well as mixed color descriptions such as "colorful" and "flowery," to determine the color characteristics of the entity. For example, if the original text is "She is wearing a pink dress," the dress color is "pink"; if the original text is "There are many colorful butterflies flying among the flowers," the butterfly color is "multicolored."

[0109] Pattern attributes: This analyzes the vocabulary related to the surface style of entities in the original text to determine whether the entity is solid, whether it has patterns (such as stripes, plaid, or floral patterns), and whether it has geometric patterns (such as circular patterns or triangular patterns). For example, if the original text reads "This wooden table has exquisite carvings on its surface," the pattern attribute of the table is "carved." If the original text reads "The floor is paved with black and white square tiles," the pattern attribute of the tiles is "black and white square geometric pattern."

[0110] Texture attributes: Based on the descriptive words in the original text, determine whether the surface of an object is smooth (such as glass), rough (such as sandpaper), uneven (such as cobblestone pavement), shiny (such as polished metal), or dull (such as old cloth). If the original text is "a metal ball that feels smooth to the touch," the texture attribute of the metal ball is "smooth"; if the original text is "the walls of this old castle appear uneven," the texture attribute of the castle wall is "uneven."

[0111] Material attributes: Identify material nouns mentioned in the original text to determine the material of the entity. Common materials include wood, metal, glass, plastic, and ceramic. For example, if the original text is "a ceramic vase is placed on the table," the material attribute of the vase is "ceramic." If the original text is "the bench in the park is made of wood," the material attribute of the bench is "wood."

[0112] Condition attribute: This method searches for words that describe an entity's state, such as whether it's new or old, good or bad, or damaged or worn, to determine the object's condition. For example, if the original text is "The old, beat-up car parked on the side of the road," the car's condition attribute is "worn"; if the original text is "The newly opened book exudes the scent of ink," the book's condition attribute is "new."

[0113] Opacity attribute: Determine from the original text whether an object is transparent (such as a glass water cup), translucent (such as frosted glass, certain gauze materials), or opaque (such as a metal block, a wooden board). For example, if the original text is "You can see the scenery outside through the transparent glass window", the opacity attribute of the glass window is "transparent"; if the original text is "Use an opaque curtain to cover the stage", the opacity attribute of the curtain is "opaque".

[0114] Directional attributes: Analyze the original text's descriptions of the orientation of objects, characters, and postures to determine whether the object is upright (e.g., a standing flagpole), horizontal (e.g., a lying plank), inverted (e.g., a bat hanging upside down), or tilted (e.g., a ladder leaning against a wall). For example, if the original text reads, "The upright telephone pole towered into the sky," the pole's direction attribute is "upright"; if the original text reads, "He leaned the picture frame against the wall," the picture frame's direction attribute is "tilted."

[0115] Action attributes: For characters or movable objects, extract words from the original text that describe their actions and postures, such as "running," "jumping," "sitting," and "waving." For example, if the original text is "an athlete running hard on the field," the athlete's action attribute is "running."

[0116] Text attributes: Find information in the original text about whether there is text on the surface of an object, as well as the writing location, font, number of lines, paragraphs, and other information of the text. For example, if the original text is "The brand name is printed on the packaging box in a bold regular script font, arranged in a single line," the text attributes of the packaging box can be clearly defined as: there is text, and the location is "surface", the font is "bold regular script", and the number of lines is "single line".

[0117] Then, the spatial position relationship and / or interaction relationship between different entities in the original text can be analyzed. Specifically, semantic dependency analysis technology can be used to find the interaction and / or relative spatial relationship between entities. For example, when the original text is "the book is placed on the table", the spatial position relationship between the book and the table is "placed on it"; when the original text is "the car is driving on the road", the spatial position relationship between the car and the road is "driving on it".

[0118] Therefore, the present application can obtain various components of the foreground feature knowledge base based on the original text corresponding to the image.

[0119] In a feasible implementation manner, after the above step S103, the method further includes:

[0120] Step S104, classifying each of the entities to obtain an entity category corresponding to each of the entities;

[0121] Step S105 : storing the entities, the relationships between the entities, the entity attributes corresponding to the entities, and the entity categories corresponding to the entities.

[0122] For example, after obtaining the entities, the identified entities can be classified according to a predefined classification system, wherein the entity categories mentioned above are the results of the classification. Specifically, the classification system can cover common object categories, such as: people, animals, plants, vehicles, equipment and tools, etc. For example, "balloons" can be classified as "equipment and tools". Then, the entities, the relationships between entities, the entity attributes of each entity, and the entity categories corresponding to each entity can be stored.

[0123] In one feasible implementation, a knowledge graph can be used to store entities, relationships between entities, entity attributes of each entity, and the entity categories corresponding to each entity. Entity categories can be presented as labels in the knowledge graph. Thus, when a user needs to generate an image of a specific category or entity relationship, the knowledge graph can be used to filter out information such as the target entity and the relationship between target entities based on the user input. Images can then be quickly generated based on this filtered information, thereby improving image generation efficiency.

[0124] In a feasible implementation manner, the above step S20 includes:

[0125] Step S201, extracting various background attributes from the background description sample, wherein the background attributes include scene attributes, background subject attributes, picture style attributes, composition attributes, lighting attributes, color palette attributes, texture attributes, depth of field attributes, theme attributes, emotion attributes, count attributes, and perspective attributes;

[0126] Step S202: constructing a background feature knowledge base according to each of the background attributes.

[0127] For example, sentences and paragraphs describing the background can be screened out from the original text to obtain background description samples. The background description samples can then be normalized, formatted, and corrected for typos to improve their quality.

[0128] Then, the large language model can be used to extract various background attributes from the background description sample based on the definition of each background attribute.

[0129] Specifically, each background attribute is defined as:

[0130] Scene attributes: Determine the scene type of the image from the background description sample. Scene attributes can be: indoor (mentioning indoor elements such as rooms, furniture, and walls), outdoor (appearing natural elements such as the sky, grass, and mountains), landscape (focusing on describing landscapes such as mountains, rivers, and lakes), city streets (including urban facilities such as high-rise buildings, roads, and street lights), etc.; if the background description sample is "sunshine shining on the quiet streets of a small town, with ancient buildings and street lights on the street", it can be known that the scene attribute corresponding to the background description sample is "city streets".

[0131] Background attribute: Identifies the description of the background elements in the background description sample. For example, if the background description sample is "deep in the dark forest, the trees are tall and dense," the background attribute of the background description sample is "forest"; if the background description sample is "with blue sky and white clouds as the background, and endless grassland below," the background attribute is "blue sky and white clouds, grassland."

[0132] Picture style attributes: The picture style is determined based on the keywords in the background description sample; picture style attributes can be: realism (detailed description, close to reality, without exaggeration or distortion, for example, the photo realistically records the street scene), cartoon (cartoon characters, bright colors, and simple lines, for example, "the cartoon castle in the animation is colorful"), oil painting (mentioning oil painting brushstrokes and rich color texture, for example, "this painting has a strong oil painting style, with rich colors and rich layers"), retro (including retro elements, such as old-fashioned furniture and a bygone era atmosphere, for example, "the room is filled with an old gramophone, full of retro atmosphere"), science fiction (involving future technology elements, such as spaceships and laser weapons, for example, "in the vast universe, interstellar battleships shuttle through it, which is very sci-fi"), etc.

[0133] Compositional attributes: Analyze the background description sample's description of the arrangement of elements and focal distribution in the image. For example, compositional attributes could be: symmetrical (e.g., "The palace building is symmetrical and solemn"), balanced (elements are evenly distributed and harmonious, "The figures and scenery in the picture are well matched and visually balanced"), or asymmetrical (e.g., "Modern artwork, elements are randomly combined, presenting an asymmetrical aesthetic").

[0134] Lighting attributes: Search for descriptions of light sources in sample background descriptions. Lighting attributes can include natural light sources (such as sunlight and moonlight, "The morning sun shines through the leaves onto the ground"), artificial light sources (such as lamplight and candlelight, "The room is lit with a dim light"), and information such as the direction and intensity of the light source, such as "Strong light hits the model's face from the side, highlighting the three-dimensional effect."

[0135] Palette attributes: Extract the main colors mentioned in the background description sample and the matching relationship between the colors; for example, the palette attribute can be "the picture is mainly warm-toned, with the orange-red sunset and the golden wheat field complementing each other"; the palette attribute can also be "the cool-toned cyan sea water beats against the black reefs."

[0136] Texture attributes: Determine the texture of the entire image or key elements based on the background description sample. Texture attributes can be: smooth (such as "the smooth marble floor reflects light"), rough ("the surface of the ancient city wall is rough, full of historical vicissitudes"), shiny ("the metal sculptures shine in the sun") or dull ("the dark basement, the walls are dull in color").

[0137] Depth of field attributes: Analyze the descriptions of image focus and clarity range in the background description samples. Depth of field attributes can include: clarity of the entire image (such as a panoramic landscape photo describing "the beautiful scenery before you is unobstructed, from the nearby flowers to the distant mountains, everything is clearly visible"), partial clarity ("close-up of the person, blurred background, highlighting the subject's expression"), and the atmosphere created by the depth of field effect, such as "shallow depth of field makes the subject stand out, and the blurred background creates a dreamy feeling."

[0138] Theme attribute: summarizes the core theme of the image described by the background description sample, as well as the relationship between other elements and the theme. For example, if the background description sample is "with maternal love as the theme, the mother gently holds her child, and the cradle and toys beside her are embellishments", then the theme attribute is "maternal love", and other elements serve as a foil. If the background description sample is "environmental protection theme poster, with withered trees contrasting with new green shoots", then the theme attribute is "environmental protection".

[0139] Emotional attributes: Understand the overall emotion conveyed by the image from the background description samples; emotional attributes can be: cheerful ("Children are laughing and playing in the playground, the picture is full of joy"), tranquil ("The quiet lake is only a breeze blowing across the water, creating ripples"), mysterious ("The ancient castle is looming in the mist, exuding a mysterious atmosphere"), tense ("The battlefield is filled with smoke, the soldiers are nervous, and the situation is on the verge of breaking out"), etc.

[0140] Count attribute: Counts the approximate number of primary and secondary objects mentioned in the background description sample. For example, if the background description sample is "there are three people in the picture, and several birds flying across the sky," the count attribute can be the number of people and birds, which is used as a reference for subsequent composition.

[0141] Perspective attribute: The camera shooting perspective is determined based on the background description sample keywords. If the background description sample is "from a level perspective, showing the daily lives of people on the street", it is eye level; the background description sample is "looking down from a high altitude, the prosperity of the city is in sight", it is a bird's-eye view; the background description sample is "shooting tall monuments from above, making them appear more majestic and spectacular", it is a low angle; the background description sample is "lying on the grass, shooting flowers from a worm's-eye perspective, showing a different microscopic world", which is a worm's-eye perspective, etc.

[0142] Then, the background attributes in each background description sample may be summarized to obtain a background feature knowledge base, so as to obtain a rich prompt text based on the rich background feature knowledge base.

[0143] In a feasible implementation manner, the above step S30 includes:

[0144] Step S301, when it is detected that there is no original prompt text, determining a first random feature in the foreground feature knowledge base and determining a second random feature in the background feature knowledge base, the original prompt text being a previously received prompt text;

[0145] It is understood that the original prompt text refers to the keywords or key sentences provided by the user to generate a specific field or specific scene. The first random feature refers to a feature randomly determined in the foreground feature knowledge base. The second random feature refers to a feature randomly determined in the background feature knowledge base. Specifically, the first random feature includes at least a randomly determined entity, a randomly determined entity attribute within the entity, and a randomly determined spatial position relationship or interaction relationship. The second random feature includes a randomly determined background attribute.

[0146] Step S302: constructing text based on the first random feature and the second random feature through a large language model to obtain a target prompt text.

[0147] It is understandable that when it is detected that the user has not given a key word or phrase, entities, entity attributes, relationships between entities, and background attributes can be randomly determined from the foreground feature knowledge base and the background feature knowledge base, and then the target prompt text can be generated based on the randomly determined features through the large language model.

[0148] In this embodiment, the application randomly selects entities and background information from two knowledge bases and uses the rich text generation imagination of the large language model to generate rich and diverse target prompt texts that may transcend realism. Therefore, in this embodiment, the large language model can be driven by randomly determining features to generate target prompt texts covering various scenarios and styles, thereby meeting the different creative needs of users.

[0149] Please refer to Figure 2 , Figure 2 This is a schematic diagram of the target prompt text construction process of an embodiment of the method for constructing a graphic and text dataset of this application. In a feasible implementation, the above step S30 further includes:

[0150] Step S303: When the original prompt text is detected, the foreground feature knowledge base is filtered for a first target feature corresponding to the original prompt text by using a large language model, and a foreground description text is constructed by using the large language model based on the original prompt text and the first target feature.

[0151] It can be understood that the first target feature refers to information in the foreground feature knowledge base that is relevant to the original prompt text. For example, if the original prompt text is a category, such as "person," the entities, entity attributes, and inter-entity relationships within the person category can be used as the first target feature. The large language model can then construct text describing the image foreground based on the original prompt text and the first target feature.

[0152] Step S304, screening the second target feature corresponding to the original prompt text in the background feature knowledge base by using the large language model, and constructing a background description text based on the original prompt text and the second target feature by using the large language model;

[0153] Similarly, the second target feature refers to information in the background feature knowledge base related to the original prompt text. Specifically, the second target feature includes background attributes related to the original prompt text. Then, the background description text can still be constructed based on the original prompt text and the second target feature through the large language model.

[0154] Step S305: Obtain target prompt text according to the foreground description text and the background description text.

[0155] In this embodiment, the foreground description text and the background description text may be spliced ​​together using a large language model, and the spliced ​​text may be used as the target prompt text.

[0156] It is understandable that Figure 2 The step of semantically enhancing the original prompt text based on the foreground feature knowledge base and the background feature knowledge base refers to the steps of constructing a foreground description text, constructing a background description text, and splicing them together to obtain the target prompt text.

[0157] In addition, in a feasible implementation manner, the specific steps of step S303 may be:

[0158] When the presence of the original prompt text is detected, if it is detected that the original prompt text does not match any category in the foreground feature knowledge base, then a determination is made as to whether the original prompt text is an entity. If the original prompt text is determined to be an entity, information in the foreground feature knowledge base related to the entity represented by the original prompt text can be used as the first target feature. If the original prompt text is determined not to be an entity, each entity in the foreground feature knowledge base can be matched with the original prompt text, and the information related to the entity with the highest degree of match can be used as the first target feature. Then, based on the language combination capability of the large language model, a semantically coherent foreground description text can be generated by the large language model from the first target feature and the original prompt text.

[0159] The specific steps of step S304 may be:

[0160] For each known background attribute in the background feature knowledge base, the natural language understanding ability of the large language model can be used to determine the content in each background attribute that matches the original prompt text with a high degree of match, and the content in the background feature knowledge base that matches the original prompt text with a high degree of match can be used as the second target feature. Then, based on the language combination ability of the large language model, the original prompt text and the second target feature can be combined to obtain the background description text.

[0161] The specific steps of step S305 may be:

[0162] By combining the foreground description text and the background description text through a large language model, the target prompt text can be obtained.

[0163] For example, if the original prompt text is: "Generate a food-themed image," the large language model can search for food-category entities in the foreground feature knowledge base. If a food-category entity, such as "cake," exists in the foreground feature knowledge base, entity attributes of the food-category entity can be extracted. For example, entity attributes could be: the cake's shape is "round and multi-layered," its color is "pink cream with colorful candy decorations," and its ingredients are "flour, cream, sugar, etc." Thus, the first target features are: "cake," "round and multi-layered," "pink cream with colorful candy decorations," and "flour, cream, sugar, etc." The foreground description text generated by the large language model based on "Generate a food-themed image" and the first target features could be: "A beautiful round, multi-layered cake, pink cream adorned with colorful candies, exuding an alluring aroma." Simultaneously, the large language model can identify background attributes from the background feature knowledge base that closely match the original prompt text: "Generate a food-themed image." For example, if the original prompt text closely matches the scene attributes "bright kitchen background" and "restaurant table scene," the style attribute "realistic and warm," and the color palette attribute "primarily warm tones, highlighting the color of the food," then the second target features are: "bright kitchen background," "restaurant table scene," "realistic and warm," and "primarily warm tones, highlighting the color of the food." The background description text generated by the large language model might be: "In a bright and warm kitchen, fresh fruit is placed on the wooden table, and sunlight shines through the window onto the food." The foreground and background description texts can then be combined based on the language organization capabilities of the large language model to generate the target prompt text. For example, the target prompt text might be: "In a bright and warm kitchen, an exquisite round multi-layered cake is placed on the wooden table. Pink cream is dotted with colorful candies, exuding an enticing aroma. Fresh fruit is also placed nearby, and sunlight shines through the window onto the food."

[0164] In this embodiment, the present application can generate high-quality prompt text whether the user gives a prompt statement or not, thereby improving the scenario adaptability of the present application.

[0165] In a feasible implementation manner, the above step S305 includes:

[0166] Step S3051: splicing the foreground description text and the background description text using the large language model to obtain an intermediate prompt text;

[0167] Step S3052: Determine the words to be replaced in the intermediate prompt text according to preset word replacement rules, generate target words based on the words to be replaced through the large language model, and replace the words to be replaced based on the target words to obtain a target prompt text. The word replacement rules include at least: adjective replacement, position relationship replacement, and quantity relationship replacement.

[0168] It is understood that in addition to directly using the text obtained by splicing the large language models as the target prompt text, the text obtained by splicing the large language models can also be used as the intermediate prompt text. In this way, adjectives, quantifiers, and positional words in the intermediate prompt text can be replaced, and multiple target prompt texts and thus multiple images can be obtained based on a single intermediate prompt text.

[0169] In this embodiment, the present application can expand the content of the prompt text by replacing words, thereby not only improving the quality of the target prompt text, but also ensuring the diversity of the target prompt text.

[0170] Furthermore, based on the various embodiments of the method for constructing a graphic and text dataset of the present application described above, a fifth embodiment of the method for constructing a graphic and text dataset of the present application is proposed.

[0171] Please refer to Figure 3 , Figure 3 This is a schematic diagram of the image-text pair construction process of an embodiment of the method for constructing an image-text dataset of this application. Figure 3 In the above step S40, the following steps are included:

[0172] Step S401: input the target prompt text into a preset text-image model to obtain a target image;

[0173] Step S402: inputting the target image and the target prompt text into a comparative language image pre-training model to obtain the similarity between the target image and the target prompt text;

[0174] Step S403 : When it is detected that the similarity is greater than or equal to a preset similarity threshold, the target image and the target prompt text are constructed into an image-text pair, and the image-text pair is stored.

[0175] For example, the image encoder in a contrastive language image pre-training model (such as CLIP) extracts features from the target image, converting it into a high-dimensional image feature vector. Specifically, structures such as convolutional neural networks can be used to perform multi-layer abstraction of the image's pixel information, capturing features such as color, shape, texture, and object layout. Simultaneously, the text encoder encodes the input target prompt text, converting it into a text feature vector. Through word embedding and multi-head attention mechanisms, the encoder understands the text's semantics, lexical relationships, and thematic information. The contrastive language image pre-training model then calculates the similarity between the image feature vector and the text feature vector, filtering out image-text pairs with low similarity and retaining only those with high similarity to ensure the quality of the incoming data.

[0176] In this embodiment, after obtaining the target prompt text, an image can be generated based on the target prompt text, and the similarity between the generated image and the target prompt text can be calculated to obtain a high-quality image-text pair. Thus, this application can not only generate high-quality prompt text, but also generate high-quality image-text pairs, thereby laying the foundation for training an efficient image generation model.

[0177] In a feasible implementation manner, after the above step S403, the method further includes:

[0178] Step S404 : determining a target image type corresponding to the target image among various preset image types, and dividing the image-text pairs into an image-text pair data set corresponding to the target image type.

[0179] For example, after obtaining the image-text pair, the target image may be classified, so that the image-text pair may be divided into an image-text pair dataset corresponding to the type of the target image for subsequent use.

[0180] In another feasible implementation, metadata tags can be added to images. These metadata tags include information such as the image category (e.g., landscape, people, animals), style (e.g., realistic, cartoon, abstract), and so on. These organized and labeled image-text pairs can then be stored in a composite image-text dataset, making subsequent data use and management more convenient for querying, filtering, and classification, thereby improving dataset usability and management efficiency.

[0181] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the method for constructing the graphic and text data set of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0182] This application also provides a system for constructing image and text datasets. Please refer to Figure 4 , the construction system of the image and text dataset includes:

[0183] A foreground feature knowledge base construction module 10 is configured to extract entities, entity attributes, and inter-entity relationships based on the original text corresponding to each original image, and to construct a foreground feature knowledge base based on the entities, the entity attributes, and the inter-entity relationships;

[0184] A background feature knowledge base construction module 20 is used to extract background description samples from each of the original texts and construct a background feature knowledge base based on the background description samples;

[0185] A text construction module 30 is used to construct text based on the foreground feature knowledge base and the background feature knowledge base to obtain a target prompt text;

[0186] The image-text pair construction module 40 is configured to generate a corresponding target image using the target prompt text, and form an image-text pair based on the target image and the corresponding target prompt text, so as to construct the image-text dataset.

[0187] In one embodiment, the system for constructing the image and text dataset further includes:

[0188] Image processing module, used to segment and classify each original image to obtain various visual features;

[0189] The original text generation module is used to generate the original text corresponding to each of the original images based on a preset semantic description knowledge base and each of the visual features.

[0190] In one embodiment, the system for constructing the image and text dataset further includes:

[0191] The text preprocessing module is used to perform data cleaning and spelling correction on the original text corresponding to each original image to obtain a preprocessed text;

[0192] Based on this, the foreground feature knowledge base construction module 10 is further used to:

[0193] Based on the preprocessed texts, entities, entity attributes and relationships between entities are extracted.

[0194] In one embodiment, the foreground feature knowledge base construction module 10 is further configured to:

[0195] Performing semantic segmentation on the original texts corresponding to the original images to obtain the words corresponding to the original texts;

[0196] Entities in each of the words are identified through a large language model, and based on the large language model and the original texts, the relationship between entities and the entity attributes corresponding to each of the entities are obtained, wherein the entity attributes include: functional attributes, shape attributes, size attributes, color attributes, pattern attributes, texture attributes, material attributes, situation attributes, opacity attributes, direction attributes, action attributes, and text attributes.

[0197] In one embodiment, the system for constructing the image and text dataset further includes:

[0198] An entity classification module, configured to classify each of the entities to obtain an entity category corresponding to each of the entities;

[0199] The data storage module is used to store the entities, the relationships between the entities, the entity attributes corresponding to the entities, and the entity categories corresponding to the entities.

[0200] In one embodiment, the background feature knowledge base construction module 20 is further configured to:

[0201] Extracting various background attributes from the background description sample, the background attributes including: scene attributes, background subject attributes, picture style attributes, composition attributes, lighting attributes, color palette attributes, texture attributes, depth of field attributes, theme attributes, emotion attributes, count attributes, and perspective attributes;

[0202] A background feature knowledge base is constructed according to each of the background attributes.

[0203] In one embodiment, the text construction module 30 is used to:

[0204] When it is detected that the original prompt text does not exist, determining a first random feature in the foreground feature knowledge base and determining a second random feature in the background feature knowledge base, the original prompt text being the previously received prompt text;

[0205] A text is constructed based on the first random feature and the second random feature through a large language model to obtain a target prompt text.

[0206] In one embodiment, the text construction module 30 is used to:

[0207] When the original prompt text is detected, the foreground feature knowledge base is filtered using a large language model to select a first target feature corresponding to the original prompt text, and a foreground description text is constructed based on the original prompt text and the first target feature using the large language model;

[0208] screening a second target feature corresponding to the original prompt text in the background feature knowledge base by using the large language model, and constructing a background description text based on the original prompt text and the second target feature by using the large language model;

[0209] A target prompt text is obtained according to the foreground description text and the background description text.

[0210] In one embodiment, the text construction module 30 is used to:

[0211] splicing the foreground description text and the background description text by using the large language model to obtain an intermediate prompt text;

[0212] According to preset word replacement rules, the words to be replaced in the intermediate prompt text are determined, and target words are generated based on the words to be replaced through the large language model, and the words to be replaced are replaced based on the target words to obtain a target prompt text. The word replacement rules include at least: adjective replacement, position relationship replacement and quantity relationship replacement.

[0213] In one embodiment, the image-text pair construction module 40 is used to:

[0214] Inputting the target prompt text into a preset text-based image model to obtain a target image;

[0215] Inputting the target image and the target prompt text into a comparative language image pre-training model to obtain the similarity between the target image and the target prompt text;

[0216] When it is detected that the similarity is greater than or equal to a preset similarity threshold, the target image and the target prompt text are constructed into an image-text pair, and the image-text pair is stored.

[0217] In one embodiment, the system for constructing the image and text dataset further includes:

[0218] The image-text pair classification module is used to determine the target image type corresponding to the target image among various preset image types, and classify the image-text pairs into an image-text pair data set corresponding to the target image type.

[0219] The system for constructing a graphic and text dataset provided in this application, employing the method for constructing a graphic and text dataset in the aforementioned embodiments, can address the technical problem of constructing high-quality text-based graph model prompt text, thereby constructing a high-quality graphic and text dataset. Compared to the prior art, the beneficial effects of the system for constructing a graphic and text dataset provided in this application are the same as those of the method for constructing a graphic and text dataset provided in the aforementioned embodiments. Other technical features of the system for constructing a graphic and text dataset are the same as those disclosed in the aforementioned embodiments and are not further elaborated here.

[0220] The present application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for constructing a graphic and text data set in the above-mentioned embodiment one.

[0221] Reference below Figure 5 , which shows a structural schematic diagram of an electronic device suitable for implementing the embodiments of the present application. Figure 5 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0222] like Figure 5 As shown, the electronic device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory 1002 or programs loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the electronic device. The processing device 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems may be connected to the input / output interface 1006: an input device 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; a storage device 1003 including, for example, a magnetic tape or hard disk; and a communication device 1009. The communication device 1009 may allow the electronic device to communicate with other devices wirelessly or wired to exchange data. Although the figures show electronic devices with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0223] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are performed.

[0224] The electronic device provided in this application, utilizing the method for constructing a graphic-text dataset in the aforementioned embodiment, can solve the technical problem of constructing high-quality text-based model prompt text, thereby constructing a high-quality graphic-text dataset. Compared to the prior art, the beneficial effects of the electronic device provided in this application are the same as those of the method for constructing a graphic-text dataset in the aforementioned embodiment, and the other technical features of this electronic device are the same as those disclosed in the method of the aforementioned embodiment, and are not further described here.

[0225] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0226] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0227] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, a computer program) stored thereon, wherein the computer-readable program instructions are used to execute the method for constructing a graphic and text dataset in the above-mentioned embodiment.

[0228] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0229] The computer-readable storage medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0230] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by an electronic device, the electronic device: extracts entities, entity attributes and relationships between entities based on the original text corresponding to each original image, and constructs a foreground feature knowledge base based on the entities, the entity attributes and the relationships between entities; extracts background description samples from each of the original texts, and constructs a background feature knowledge base based on the background description samples; constructs text based on the foreground feature knowledge base and the background feature knowledge base to obtain the target prompt text.

[0231] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0232] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0233] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0234] The computer-readable storage medium provided in this application stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned method for constructing a text-image dataset. This computer-readable storage medium addresses the technical problem of constructing high-quality text-image model prompt text, thereby constructing a high-quality text-image dataset. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are similar to those of the method for constructing a text-image dataset provided in the aforementioned embodiment, and are not further elaborated here.

[0235] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned method for constructing a graphic and text data set when executed by a processor.

[0236] The computer program product provided in this application can solve the technical problem of constructing high-quality text-based graph model prompt text, thereby constructing a high-quality graph-text dataset. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the graph-text dataset construction method provided in the above-mentioned embodiment, and will not be elaborated here.

[0237] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for constructing a graphic and text dataset, characterized in that: include: Performing semantic segmentation on the original texts corresponding to the original images to obtain the words corresponding to the original texts; Identifying entities in each of the words using a large language model, obtaining inter-entity relationships and entity attributes corresponding to each of the entities based on the large language model and the original texts, and constructing a foreground feature knowledge base based on the entities, the entity attributes, and the inter-entity relationships; Extracting background description samples from each of the original texts, and extracting various background attributes from the background description samples; Constructing a background feature knowledge base according to each of the background attributes; Constructing text based on the foreground feature knowledge base and the background feature knowledge base to obtain target prompt text; Generating a corresponding target image using the target prompt text, and forming an image-text pair based on the target image and the corresponding target prompt text for constructing the image-text dataset; The step of constructing text based on the foreground feature knowledge base and the background feature knowledge base to obtain target prompt text includes: When it is detected that the original prompt text does not exist, a first random feature is determined in the foreground feature knowledge base, and a second random feature is determined in the background feature knowledge base, wherein the original prompt text is a previously received prompt text; a text is constructed based on the first random feature and the second random feature by using a large language model to obtain a target prompt text; When the existence of the original prompt text is detected, a first target feature corresponding to the original prompt text is screened from the foreground feature knowledge base using a large language model, and a foreground description text is constructed based on the original prompt text and the first target feature using the large language model. A second target feature corresponding to the original prompt text is screened from the background feature knowledge base using the large language model, and a background description text is constructed based on the original prompt text and the second target feature using the large language model. The foreground description text and the background description text are concatenated using the large language model to obtain an intermediate prompt text. According to a preset word replacement rule, a word to be replaced in the intermediate prompt text is determined, and a target word is generated based on the word to be replaced using the large language model, and the word to be replaced is replaced based on the target word to obtain a target prompt text. The word replacement rule includes at least: adjective replacement, positional relationship replacement, and quantitative relationship replacement.

2. The method for constructing a graphic and text dataset according to claim 1, wherein: Before the step of extracting entities, entity attributes, and relationships between entities based on the original text corresponding to each original image, the method for constructing the image-text dataset further includes: Perform image segmentation and classification labeling on each original image to obtain various visual features; Based on a preset semantic description knowledge base and each of the visual features, original text corresponding to each of the original images is generated.

3. The method for constructing a graphic and text dataset according to claim 1, wherein: The method further comprises: Perform data cleaning and spelling correction on the original text corresponding to each original image to obtain preprocessed text; Based on the preprocessed texts, entities, entity attributes and relationships between entities are extracted.

4. The method for constructing a graphic and text dataset according to claim 1, wherein: After the step of obtaining the relationship between entities and the entity attributes corresponding to each of the entities based on the large language model and the original texts, the method further includes: Classifying each of the entities to obtain an entity category corresponding to each of the entities; The entities, the relationships between the entities, the entity attributes corresponding to the entities, and the entity categories corresponding to the entities are stored.

5. The method for constructing a graphic and text dataset according to claim 1, wherein: The step of generating a corresponding target image using the target prompt text, and forming an image-text pair based on the target image and the corresponding target prompt text, includes: Inputting the target prompt text into a preset text-based image model to obtain a target image; Inputting the target image and the target prompt text into a comparative language image pre-training model to obtain the similarity between the target image and the target prompt text; When it is detected that the similarity is greater than or equal to a preset similarity threshold, the target image and the target prompt text are constructed into an image-text pair, and the image-text pair is stored.

6. The method for constructing a graphic and text data set according to claim 5, wherein: After the step of constructing the target image and the target prompt text into an image-text pair, the method further includes: Among various preset image types, a target image type corresponding to the target image is determined, and the image-text pairs are divided into an image-text pair data set corresponding to the target image type.

7. A system for constructing a graphic and text data set, characterized in that: The construction system of the image and text dataset includes: a foreground feature knowledge base construction module, configured to perform semantic segmentation on the original text corresponding to each original image to obtain the words corresponding to each original text; identify entities in each of the words using a large language model, and obtain inter-entity relationships and entity attributes corresponding to each of the entities based on the large language model and the original texts; and construct a foreground feature knowledge base based on the entities, the entity attributes, and the inter-entity relationships; A background feature knowledge base construction module is used to extract background description samples from each of the original texts and construct a background feature knowledge base based on the background description samples; the background feature knowledge base construction module is also used to extract various background attributes from the background description samples; a text construction module configured to construct text based on the foreground feature knowledge base and the background feature knowledge base to obtain a target prompt text; the text construction module further configured to, when detecting that no original prompt text exists, determine a first random feature in the foreground feature knowledge base and a second random feature in the background feature knowledge base, wherein the original prompt text is a previously received prompt text; and construct text based on the first random feature and the second random feature using a large language model to obtain a target prompt text; The text construction module is further configured to, upon detecting the presence of the original prompt text, screen the foreground feature knowledge base for a first target feature corresponding to the original prompt text using the large language model, and construct a foreground description text based on the original prompt text and the first target feature using the large language model; screen the background feature knowledge base for a second target feature corresponding to the original prompt text using the large language model, and construct a background description text based on the original prompt text and the second target feature using the large language model; concatenate the foreground description text and the background description text using the large language model to obtain an intermediate prompt text; determine a to-be-replaced word in the intermediate prompt text according to a preset word replacement rule, generate a target word based on the to-be-replaced word using the large language model, and replace the to-be-replaced word with the target word to obtain a target prompt text, wherein the word replacement rule includes at least adjective replacement, positional replacement, and quantitative replacement. The image-text pair construction module is used to generate a corresponding target image using the target prompt text, and form an image-text pair based on the target image and the corresponding target prompt text to construct the image-text dataset.

8. An electronic device, characterized in that: The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the method for constructing a graphic and text dataset according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the method for constructing a graphic and text dataset according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the method for constructing a graphic and text dataset according to any one of claims 1 to 6 are implemented.

Images