Image data generation method and device, electronic equipment and readable storage medium

Abstract

The invention relates to the technical field of image processing, and provides an image data generation method and device, electronic equipment and a readable storage medium. The method comprises the steps of obtaining to-be-processed text data, wherein the to-be-processed text data comprises at least one piece of object data; inputting the to-be-processed text data into the trained large language model to process the to-be-processed text data to obtain layout information data corresponding to each piece of object data; according to the layout information data corresponding to the object data, determining a fusion feature vector corresponding to the layout information data; performing hierarchical mapping processing on the fusion feature vector corresponding to the layout information data to obtain a layout feature vector set corresponding to the to-be-processed text data; and inputting the layout feature vector set corresponding to the to-be-processed text data into the trained diffusion model for processing to obtain target image data containing at least one object, thereby improving the fidelity in the multi-subject image generation process and improving the accuracy of the multi-subject layout.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image data generating method, device, electronic device and readable storage medium. Background Art

[0002] In the field of text-to-image generation, the progress of stable diffusion models has promoted the diversity and fidelity of image generation results. However, when dealing with complex natural scenes, existing models have problems with alignment errors, insufficient understanding of spatial relationships, and invalid semantic information in the generation process of multi-subject images, which reduces the high fidelity and semantic consistency of text-to-image generation. In the current existing technology, the controllability of generated images for multi-subject layout results can be improved through user guidance, but this requires users to manually provide guidance information, thereby reducing the intelligence level of image data generation technology.

[0003] It can be seen that the existing technology has problems such as low fidelity of multi-subject image generation and low precision of multi-subject layout due to alignment errors in the multi-subject image generation process, insufficient understanding of spatial relationships, and low accuracy of layout results. Summary of the invention

[0004] In view of this, the embodiments of the present disclosure provide an image data generation method, device, electronic device and readable storage medium to solve the problems in the prior art of low fidelity of multi-subject image generation and low precision of multi-subject layout due to alignment errors in the multi-subject image generation process, insufficient understanding of spatial relationships, and low accuracy of layout results.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided a method for generating image data, comprising: obtaining text data to be processed, the text data to be processed comprising at least one object data; inputting the text data to be processed into a trained large language model to process the text data to be processed, and obtaining layout information data corresponding to each object data, wherein the large language model is trained based on the text data to be trained, real layout information data corresponding to each text data to be trained, and preset low-rank matrix adjustment parameters; determining a fused feature vector corresponding to the layout information data according to the layout information data corresponding to each object data; performing hierarchical mapping processing on the fused feature vector corresponding to the layout information data, and obtaining a set of layout feature vectors corresponding to the text data to be processed; inputting the set of layout feature vectors corresponding to the text data to be processed into a trained diffusion model for processing, and obtaining target image data containing at least one object, wherein the diffusion model is trained based on the large language model, the text data to be trained, reference image data corresponding to each text data to be trained, and a label corresponding to each text data to be trained.

[0006] According to a second aspect of an embodiment of the present disclosure, there is provided an image data generating device, comprising: an acquisition module, for acquiring text data to be processed, the text data to be processed comprising at least one object data; a first processing module, for inputting the text data to be processed into a trained large language model to process the text data to be processed, and obtaining layout information data corresponding to each object data, wherein the large language model is trained based on the text data to be trained, the real layout information data corresponding to each text data to be trained, and a preset low-rank matrix adjustment parameter; a determination module, for determining a fused feature vector corresponding to the layout information data according to the layout information data corresponding to each object data; a second processing module, for performing hierarchical mapping processing on the fused feature vector corresponding to the layout information data, and obtaining a set of layout feature vectors corresponding to the text data to be processed; and a third processing module, for inputting the set of layout feature vectors corresponding to the text data to be processed into a trained diffusion model for processing, and obtaining target image data containing at least one object, wherein the diffusion model is trained based on the text data to be trained, the reference image data corresponding to each text data to be trained, and the label corresponding to each text data to be trained.

[0007] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0008] According to a fourth aspect of an embodiment of the present disclosure, a readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0009] Compared with the prior art, the disclosed embodiment has the following beneficial effects: by acquiring text data to be processed containing at least one object data, and processing the acquired text data to be processed in a trained large language model through a low-rank matrix adjustment method, layout information data of each object data in the text data to be processed is obtained, and feature fusion processing can be performed on the layout information data of each object data in the text data to be processed to obtain a fused feature vector corresponding to the layout information data, and then hierarchical mapping processing can be performed on the fused feature vector corresponding to the layout information data through a multi-layer perceptron to obtain a set of layout feature vectors corresponding to the text data to be processed, and the set of layout feature vectors corresponding to the text data to be processed is processed through a trained diffusion model to obtain target image data containing at least one object, thereby improving the fidelity of the multi-subject image generation process, improving the accuracy of the multi-subject layout, improving the efficiency and accuracy of image data generation, ensuring the quality and authenticity of the image data, and enhancing the understanding of complex layout information and multiple object information. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0011] Figure 1 is a schematic diagram of an application scenario of an embodiment of the present disclosure;

[0012] Figure 2 is a flowchart of an image data generating method provided by an embodiment of the present disclosure;

[0013] Figure 3 is a flowchart of a diffusion model training method provided by an embodiment of the present disclosure;

[0014] Figure 4 It is a flowchart of a segmentation model training method provided by an embodiment of the present disclosure;

[0015] Figure 5 is a structural schematic diagram of an image data generating device provided by an embodiment of the present disclosure;

[0016] Figure 6 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0018] A method and device for generating image data according to an embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0019] Figure 1 Schematic diagram of an application scenario of an embodiment of the present disclosure. The application scenario may include terminal devices 1, 2 and 3, a server 4 and a network 5.

[0020] Terminal devices 1, 2 and 3 can be hardware or software. When terminal devices 1, 2 and 3 are hardware, they can be various electronic devices with display screens and supporting communication with server 4, including but not limited to smart phones, tablet computers, laptop portable computers and desktop computers, etc.; when terminal devices 1, 2 and 3 are software, they can be installed in the above electronic devices. Terminal devices 1, 2 and 3 can be implemented as multiple software or software modules, or as a single software or software module, and the embodiments of the present disclosure are not limited to this. Furthermore, various applications can be installed on terminal devices 1, 2 and 3, such as data processing applications, instant messaging tools, social platform software, search applications, shopping applications, etc.

[0021] The server 4 may be a server that provides various services, for example, a background server that receives a request sent by a terminal device that establishes a communication connection with the server, and the background server may receive and analyze the request sent by the terminal device, and generate a processing result. The server 4 may be a single server, or a server cluster composed of several servers, or a cloud computing service center, which is not limited in the embodiments of the present disclosure.

[0022] It should be noted that the server 4 can be hardware or software. When the server 4 is hardware, it can be various electronic devices that provide various services for the terminal devices 1, 2, and 3. When the server 4 is software, it can be multiple software or software modules that provide various services for the terminal devices 1, 2, and 3, or it can be a single software or software module that provides various services for the terminal devices 1, 2, and 3, and the embodiments of the present disclosure are not limited to this.

[0023] The network 5 can be a wired network connected by coaxial cable, twisted pair and optical fiber, or it can be a wireless network that can interconnect various communication devices without wiring, such as Bluetooth, Near Field Communication (NFC), infrared, etc., which is not limited in the embodiments of the present disclosure.

[0024] The user can establish a communication connection with the server 4 via the network 5 through the terminal devices 1, 2 and 3 to receive or send information, etc. Specifically, the server 4 can obtain the text data to be processed containing at least one object data through the terminal devices 1, 2 and 3, and process the obtained text data to be processed in the large language model that has been trained through the low-rank matrix adjustment method to obtain the layout information data of each object data in the text data to be processed, and can perform feature fusion processing on the layout information data of each object data in the text data to be processed to obtain the fused feature vector corresponding to the layout information data, and then can perform hierarchical mapping processing on the fused feature vector corresponding to the layout information data through a multi-layer perceptron to obtain a set of layout feature vectors corresponding to the text data to be processed, and process the set of layout feature vectors corresponding to the text data to be processed through the trained diffusion model to obtain the target image data containing at least one object.

[0025] It should be noted that the specific types, quantities and combinations of the terminal devices 1, 2 and 3, the server 4 and the network 5 can be adjusted according to the actual needs of the application scenario, and the embodiments of the present disclosure are not limited to this.

[0026] Figure 2 It is a flowchart of an image data generating method provided by an embodiment of the present disclosure. Figure 2 The image data generation method can be Figure 1 The server executes. Figure 2 As shown, the image data generating method includes:

[0027] Step 201: Acquire text data to be processed, where the text data to be processed includes at least one object data.

[0028] Specifically, the text data to be processed that is submitted by the user or automatically generated can be received through the input interface, which is not limited here. The text data to be processed can exist in data formats such as strings or files, which is not limited here, and the text data to be processed contains at least one object data, which can be a specific entity or concept described in the text data to be processed, including but not limited to names of people, places, time, or objects, etc. The object data can be identified and extracted from the text data to be processed through natural language processing methods or regular expressions, etc., which are not limited here, and can be used for semantic understanding and content mapping in the subsequent image data generation process, thereby enhancing the data source flexibility of the image data generation process, improving the accuracy and richness of image data content generation, and improving the image data generation method's understanding ability and expression effect for complex text descriptions.

[0029] Among them, the text data to be processed can be the original text information to be imaged or visually presented, the text data to be processed can be the text content input by the user, including but not limited to stories, news, or instruction documents, etc. The object data can be identifiable entities or concepts in the text data to be processed, which can be identified and extracted through natural language processing methods, and can provide the necessary semantic and content basis for image data generation.

[0030] For example, the text data to be processed submitted by the user may be received through the input interface. The text data to be processed may be "a beautiful girl is reading a book". The text data to be processed includes two object data, namely "girl" and "book".

[0031] Step 202, input the text data to be processed into the trained large language model to process the text data to obtain layout information data corresponding to each object data. The large language model is trained based on the text data to be trained, the real layout information data corresponding to each text data to be trained, and the preset low-rank matrix adjustment parameters.

[0032] Specifically, the text data to be processed can be used as input and input into a trained large language model. The large language model can be obtained by training and optimizing through preset low-rank matrix adjustment parameters, and then when processing the text data to be processed, layout information data of each object data can be automatically generated and output. The layout information data includes but is not limited to text data, coordinate point data, category information, and / or attribute information, etc. The object data can be a paragraph, a title, a role, or a list, etc., which is not limited here. This enhances the level of intelligence in processing the text data to be processed, improves the automatic conversion capability from text to visual layout, and improves the efficiency and accuracy of image data generation.

[0033] Among them, the large language model can be an advanced artificial intelligence model that can understand and generate natural language text. The large language model can be a large-scale pre-trained model based on deep learning methods, including but not limited to bidirectional encoder representations from Transformer (Bidirectional Encoder Representations from Transformers, BERT) or generative pre-trained Transformer (Generative Pre-trained Transformer, GPT), etc., and can have the ability to process complex language tasks. The layout information data corresponding to the object data can be parameter information related to visual presentation such as text data, coordinate point data, category information, and / or attribute information of each text object. The text objects include but are not limited to roles, paragraphs, titles, or pictures, etc. The text data to be trained can be a data set containing a variety of text contents, which can be used to train the above-mentioned large language model, so that the large language model can learn language rules and text features. The real layout information data corresponding to the text data to be trained can be the real layout information data corresponding to the text data to be trained during the training process. , which can be used to represent the real data of the layout of the text data to be trained in the actual document or page, and can be used to guide the large language model to learn the ability to infer layout information from the text content. The layout method includes but is not limited to position or format, etc. The preset low-rank matrix adjustment parameters can be an optimization method introduced when training the large language model. The complexity and computational complexity of the large language model can be reduced by performing low-rank approximation or adjustment on the matrix inside the large language model. The preset low-rank matrix adjustment parameters can be obtained through hyperparameter search or optimization algorithm, etc., and are not limited here. They can be used to improve the generalization ability of the large language model, accelerate the training process or improve the output quality.

[0034] For example, the text data to be processed, "A beautiful girl is reading a book", can be input into a large language model trained based on the low-rank matrix adjustment method, and then processed together with the prompt template. The prompt template can be: Now you are an assistant, help me design an image layout for a given description. Specifically, the layout represents a set of "object: bounding box, object description" items. "Object" can be the name of any entity, and "bounding box" is expressed as [x, y, w, h], where "x, y" represents the coordinates of the upper left corner of the bounding box. "w, h" represents the corresponding width and height. The data are normalized to between 0 and 1. Please return at least 2 objects.

[0035] The result can be: [Entity1:{name:"Girl"},layout:[0.3,0.3,0.4,0.6],attributes:{Color:"Colorful",Posture:"Sitting"}];

[0036] [Entity2:{name:"Book"},layout:[0.4,0.55,0.2,0.15],attributes:{Color:"Brown",Type:"Hardcover"}].

[0037] Step 203: Determine the fusion feature vector corresponding to the layout information data according to the layout information data corresponding to each object data.

[0038] Specifically, feature extraction can be performed on the layout information data corresponding to each object data. The extracted features may include text data, coordinate point data, category information, and / or attribute information, etc., which can describe multi-dimensional data of the layout information, which is not limited here. Then, through the feature fusion method, the extracted feature information can be added according to weights or rules to obtain a fused feature vector that can be used to characterize the characteristics of the layout information data, that is, the fused feature vector corresponding to the layout information data, thereby enhancing the representation ability of the layout information data, improving the comprehensiveness and accuracy of the feature vector, thereby improving the efficiency and effect of subsequent processing, and by fusing the feature vector, the accuracy of obtaining the internal laws and features of the layout information data is improved.

[0039] For example, feature extraction processing can be performed on the text data, coordinate point data, category information, and attribute information corresponding to "girl" and "book" respectively to obtain the text feature vectors, masks, category vectors, and attribute vectors corresponding to the girl and book respectively, and the above feature vectors are added together to obtain the fused feature vector corresponding to the layout information data.

[0040] Step 204 , performing hierarchical mapping processing on the fused feature vector corresponding to the layout information data to obtain a set of layout feature vectors corresponding to the text data to be processed.

[0041] Specifically, the fused feature vectors corresponding to each layout information data can be converted and combined through the hierarchical mapping processing of the multi-layer perceptron. The hierarchical mapping processing can include multiple levels of mapping operations, which can be mapping from low-level detail feature vectors to high-level abstract feature vectors, or the fused feature vectors of different objects can be combined and integrated according to their hierarchical relationship in the structure of the text data to be processed. There is no limitation here. The mapping processing can be used to obtain a set of layout feature vectors corresponding to the text data to be processed, thereby enhancing the hierarchy and structure of the layout feature vectors, improving the ability to express the complexity of the document layout, and thus improving the accuracy of document layout analysis, comparison and optimization. The hierarchical mapping processing makes the layout feature vector set corresponding to the text data to be processed not only contain the detailed information of the layout of a single object, but also contain the abstract information of the layout relationship between objects, providing data support for subsequent document processing tasks.

[0042] Among them, the set of layout feature vectors corresponding to the text data to be processed can be a set of vectors obtained by further combining the fused feature vectors corresponding to the layout information data of each object in the text data to be processed through hierarchical mapping processing of a multi-layer perceptron. Each vector in the set of layout feature vectors corresponding to the text data to be processed represents the layout features in the text data to be processed. The set of layout feature vectors corresponding to the text data to be processed as a whole represents the overall layout characteristics of the text data to be processed, and can be used in subsequent application scenarios such as document layout analysis, layout optimization, document comparison, or automatic typesetting, which are not limited here.

[0043] For example, the fused feature vector corresponding to the layout information data may be input into a multi-layer perceptron, and the multi-layer perceptron may perform hierarchical mapping processing on the fused feature vector corresponding to the layout information data to obtain a set of layout feature vectors corresponding to the text data to be processed.

[0044] Step 205, input the layout feature vector set corresponding to the text data to be processed into the trained diffusion model for processing to obtain target image data containing at least one object, and the diffusion model is trained based on the large language model, the text data to be trained, the reference image data corresponding to each text data to be trained, and the label corresponding to each text data to be trained.

[0045] Specifically, the set of layout feature vectors corresponding to the text data to be processed can be input into a pre-trained diffusion model. The diffusion model can be trained based on the output results of a large language model, a large amount of text data to be trained, reference image data corresponding to each text data to be trained, and a label corresponding to each text data to be trained. Through the gradual denoising or generation process of the diffusion model, according to the set of layout feature vectors corresponding to the text data to be processed, target image data containing at least one object can be constructed and generated, thereby enhancing the cross-modal conversion capability of the image data generation model from text layout features to visual image generation, improving the diversity and accuracy of image generation, and improving the efficiency and quality of document automatic generation and visual presentation. Through the diffusion model, text layout information can be converted into intuitive image display, and a variety of visual styles can be generated according to the text content.

[0046] Among them, the diffusion model can be obtained by training with a large language model, text data to be trained, reference image data corresponding to each text data to be trained, and labels corresponding to each text data to be trained. It can generate a clear image by simulating the inverse process of data diffusion to random noise, that is, removing noise. The diffusion model can generate high-quality and diverse images. The target image data can be the image data obtained by processing the layout feature vector set corresponding to the text data to be processed by the diffusion model. The target image data contains at least one object, and the layout and visual style of the target image data match the description and requirements of the text data to be processed. The reference image data corresponding to the text data to be trained can be the image sample associated with each text data to be trained when training the diffusion model. The reference image data corresponding to the text data to be trained can be used to characterize the visual display method of the text data to be trained, and can provide a mapping example from text to image for the image data generation model. The label corresponding to the text data to be trained can be the real image data corresponding to the text data to be trained. The label corresponding to the text data to be trained can help the diffusion model understand the context and semantic information of the text data to be trained during the training process. The label corresponding to the text data to be trained can be obtained by manual annotation or automatic classification, and can be used to guide the training process of the diffusion model.

[0047] For example, the text data to be processed may be "a beautiful girl is reading a book", and then the layout feature vector set corresponding to the text data to be processed may be processed through the diffusion model to obtain the target image data, which may contain two objects, "girl" and "book".

[0048] According to the technical solution provided by the embodiment of the present disclosure, by acquiring text data to be processed containing at least one object data, and processing the acquired text data to be processed in a trained large language model through a low-rank matrix adjustment method, layout information data of each object data in the text data to be processed is obtained, and feature fusion processing can be performed on the layout information data of each object data in the text data to be processed to obtain a fused feature vector corresponding to the layout information data, and then hierarchical mapping processing can be performed on the fused feature vector corresponding to the layout information data through a multi-layer perceptron to obtain a set of layout feature vectors corresponding to the text data to be processed, and the set of layout feature vectors corresponding to the text data to be processed is processed through a trained diffusion model to obtain target image data containing at least one object, thereby improving the fidelity in the multi-subject image generation process, improving the accuracy of the multi-subject layout, improving the efficiency and accuracy of image data generation, ensuring the quality and authenticity of the image data, and enhancing the understanding of complex layout information and multiple object information.

[0049] In some embodiments, the layout information data corresponding to the object data includes coordinate point data corresponding to the object data, classification label data corresponding to the object data, and attribute data corresponding to the object data; according to the layout information data corresponding to each object data, the fused feature vector corresponding to the layout information data is determined, including: performing feature extraction processing on the text data to be processed to obtain the text feature vector corresponding to the text data to be processed; performing mask generation processing on the coordinate point data corresponding to each object data to obtain the mask corresponding to each object data; processing the mask corresponding to each object data through Fourier encoding to obtain the mask feature vector corresponding to each object data; inputting the classification label data corresponding to each object data into the text encoder for processing to obtain the category feature vector set corresponding to the object data; performing text encoding processing on the attribute data corresponding to multiple object data to obtain the attribute feature vector set corresponding to the object data; adding the mask feature vector corresponding to each object data, the category feature vector set corresponding to the object data, and the attribute feature vector set corresponding to the object data to obtain the fused feature vector corresponding to the layout information data.

[0050] Specifically, a text feature vector can be extracted from the text data to be processed through a text encoder, and a mask can be generated for the coordinate point data corresponding to each object data. The mask corresponding to the object data can be generated based on the coordinate point data of the object data, and can be used to identify the position of the object in the target image data. The mask corresponding to the object data can then be processed through Fourier encoding to obtain a mask feature vector corresponding to the object data. The mask feature vector corresponding to the object data can be used to characterize the spatial position and shape characteristics of the object. The classification label data corresponding to the object data can be input into the text encoder to obtain a set of category feature vectors corresponding to the object data, which can be used to characterize the classification information of the object. The attribute data corresponding to the object data can be text encoded to obtain a set of attribute feature vectors. The attribute data corresponding to the object data may include shape or color, etc., which are not limited here and can be used to characterize the visual attributes of the object. The mask feature vector corresponding to the object data, the set of category feature vectors corresponding to the object data, and the set of attribute feature vectors corresponding to the object data can be added together to obtain a fusion feature vector corresponding to the layout information data.

[0051] Among them, the coordinate point data corresponding to the object data can be data used to characterize the specific position of the object in the target image data, such as the x and y coordinate values ​​of the upper left corner, which can be used to determine the spatial position of the object. The classification label data corresponding to the object data can be label information used to identify the category to which the object belongs, which can be a name of a person or an object, etc., which is not limited here, and can be used to classify and identify different objects. The attribute data corresponding to the object data can be data used to characterize the visual attributes of the object, including but not limited to the shape or color of the object, etc., which can be used to define the appearance style of the object. The text feature vector corresponding to the text data to be processed can be a vector representation obtained from the text data to be processed through feature extraction processing, which can be used to obtain the overall features and semantic information of the text data to be processed. The mask corresponding to the object data can be a binary image generated according to the coordinate point data of the object data. The mask corresponding to the object data can be used to identify the position and shape of the object in the target image data, and can also be used for mask operations in image processing. Fourier coding can be a signal processing method. Fourier coding can be used to process the mask image and extract the spatial frequency characteristics of the mask to generate a mask feature vector. The mask feature vector corresponding to the object data can be a vector representation obtained by processing the mask through Fourier coding. The mask feature vector corresponding to the object data can be used to characterize the spatial frequency characteristics and shape information of the mask. The text encoder can be a model or algorithm for converting text data into a vector representation, which can be based on a deep learning method, including but not limited to a recurrent neural network, a convolutional neural network or a Transformer, etc., and can be used to obtain the semantic features and context information of the text data to be processed. The set of category feature vectors corresponding to the object data can be a set of vectors obtained after processing the classification label data corresponding to the object data through a text encoder. Each vector represents the category name to which the object belongs, and can be used to classify and identify different object categories.

[0052] For example, the input text data to be processed can be processed by a text encoder to obtain a text feature vector corresponding to the text data to be processed, mask generation processing can be performed based on the coordinate point data of each object data provided by the large language model to obtain a mask corresponding to each object data, Fourier encoding can be performed on the mask corresponding to each object data to obtain a mask feature vector corresponding to each object data, the category name corresponding to each object data can be input into the text encoder to obtain a category feature vector set corresponding to the object data, the attribute information corresponding to each object data can be input into the text encoder to obtain an attribute feature vector set corresponding to the object data, and then the mask feature vector corresponding to each object data, the category feature vector set corresponding to the object data, and the attribute feature vector set corresponding to the object data can be added to obtain a fusion feature vector corresponding to the layout information data.

[0053] According to the technical solution provided by the embodiment of the present disclosure, a text feature vector is extracted from the text data to be processed through a text encoder, a mask can be generated for the coordinate point data corresponding to each object data, and then the mask corresponding to the object data can be processed through Fourier encoding to obtain the mask feature vector corresponding to the object data, and the classification label data corresponding to the object data can be input into the text encoder to obtain a set of category feature vectors corresponding to the object data, and the attribute data corresponding to the object data can be text encoded to obtain a set of attribute feature vectors, and then the mask feature vector corresponding to the object data, the set of category feature vectors corresponding to the object data, and the set of attribute feature vectors corresponding to the object data can be added to obtain a fused feature vector corresponding to the layout information data, thereby enhancing the richness and accuracy of the layout information data, improving the comprehensive expression ability of the feature vector, and thus improving the accuracy and diversity of image data generation, and by fusing the coordinate point data, the classification label data and the attribute data, enhancing the comprehensiveness of the layout and visual features of the object in the document.

[0054] In some embodiments, the diffusion model includes a segmentation model; before obtaining the text data to be processed, it also includes: obtaining multiple text data to be trained, reference image data corresponding to each text data to be trained, and labels corresponding to each text data to be trained, the text data to be trained includes at least one object training data, and the label corresponding to the text data to be trained is the real image data corresponding to the text data to be trained; inputting each text data to be trained into the trained large language model to process the text data to be trained, and obtain layout training data corresponding to each object training data, the layout training data corresponding to the object training data includes coordinate point data corresponding to each object training data and classification label data corresponding to each object training data; according to the layout training data corresponding to each object training data, determining the fusion training vector corresponding to each layout training data; performing hierarchical mapping processing on the fusion training vector corresponding to each layout information data, and obtaining a plurality of layout training vector sets corresponding to the text data to be trained; The reference image data corresponding to the text data is denoised to obtain the noise image data corresponding to each text data to be trained; the segmentation model is trained based on the real image data corresponding to each text data to be trained, the coordinate point data corresponding to each object training data, the mask corresponding to each object training data, the noise image data corresponding to each text data to be trained, and the classification label data corresponding to each object training data to obtain the attention map comparison loss corresponding to the segmentation model; the noise image data corresponding to each text data to be trained is input into the diffusion model for processing to obtain the training image data corresponding to each text data to be trained; the reconstruction loss corresponding to the diffusion model is determined according to the training image data corresponding to each text data to be trained and the label corresponding to each text data to be trained; the total loss corresponding to the diffusion model is determined according to the attention map comparison loss corresponding to the segmentation model and the reconstruction loss corresponding to the diffusion model; the parameters in the diffusion model are updated according to the total loss corresponding to the diffusion model through loop iteration.

[0055] Specifically, in the image data generation method, in the stage of training the diffusion model, multiple text data to be trained, reference image data corresponding to the text data to be trained, and labels corresponding to each text data to be trained can be obtained. The text data to be trained can be processed by the trained large language model to extract layout training data corresponding to the object training data, including coordinate point data corresponding to the object training data, classification label data corresponding to the object training data, and attribute data corresponding to the object training data. Then, a fused training vector corresponding to the layout training data can be generated according to the layout training data corresponding to the object training data, and hierarchical mapping processing can be performed on the fused training vectors corresponding to each layout training data. A set of layout training vectors corresponding to each text data to be trained is obtained, and the reference image data corresponding to the text data to be trained can be denoised to generate noisy image data corresponding to the text data to be trained. Then, the segmentation model can be trained, and the attention map comparison loss can be calculated by comparing the real image with the attention map obtained by the segmentation model. The noisy image data corresponding to the text data to be trained can be input into the diffusion model for denoising to generate training image data, and the reconstruction loss can be calculated by comparing with the real image label. Then, the total loss can be determined according to the attention map comparison loss corresponding to the segmentation model and the reconstruction loss corresponding to the diffusion model, and the parameters of the diffusion model can be optimized through cyclic iteration.

[0056] Among them, the segmentation model can be a model that can identify different objects or regions in an image, can be used for image segmentation tasks, and can divide an image into multiple regions with specific semantic meanings, including but not limited to U-Net modules, etc. The object training data can be the object data used for training in the text data to be trained, and can contain specific information of each object data in the text data to be trained. The real image data corresponding to the text data to be trained can be the actual corresponding image data associated with the text data to be trained, and can be used as comparison information for determining the loss during the training process. The layout training data corresponding to the object training data can be data extracted from the object training data, which can be used to describe the layout features of the training object in the training image data, and can include coordinate point data corresponding to the object training data and classification label data corresponding to the object training data. The coordinate point data corresponding to the object training data can be used to characterize a set of data points at a specific position of the training object in the training image data, and can be used to determine the spatial position of the training object. The classification label data corresponding to the object training data can be label information used to identify the category to which the training object belongs, and can be used to classify and identify different training objects. The fused training vector corresponding to the layout training data can be a vector representation obtained by fusing multiple features in the layout training data. The features in the layout training data include but are not limited to coordinate point data. and / or classification label data, etc., which can be used to describe the layout features of the training objects. The layout training vector set corresponding to the text data to be trained can refer to a vector set obtained by hierarchical mapping of the fused training vectors corresponding to the training data of multiple objects, which can be used to characterize the layout features of the text data to be trained. The noise image data corresponding to the text data to be trained can be image data obtained by adding noise to the reference image data, which can be used to simulate noise interference in the image generation process. The attention map comparison loss corresponding to the segmentation model can be a loss value calculated by comparing the difference between the attention map obtained by the segmentation model and the attention map of the training object in the real image, which can be used Evaluate the segmentation model's ability to understand the layout information of the training text data. The training image data corresponding to the training text data refers to the image data obtained after denoising the noisy image data through the diffusion model, which can be used as the generated samples in the training process. The reconstruction loss corresponding to the diffusion model can be the loss value obtained by calculating the difference between the training image data and the real image data through the mean square error loss, which can be used to evaluate the reconstruction ability of the diffusion model in the image data generation model. The total loss corresponding to the diffusion model can be the total loss value obtained after the attention map comparison loss corresponding to the segmentation model and the reconstruction loss corresponding to the diffusion model, which can be used to guide the optimization process of the diffusion model parameters.

[0057] For example, according to Figure 3As shown in the flowchart of the diffusion model training method, the text data to be trained is input into the trained large language model 301 to obtain layout training data corresponding to each object training data, wherein the layout training data corresponding to the object training data includes coordinate point data corresponding to the object training data, classification label data corresponding to each object training data, and attribute data corresponding to each object training data, and then the layout training data corresponding to each object training data can be trained by contrastive language-image pre-training. The CLIP encoder 302 processes the input text data to be trained to obtain the text feature vector corresponding to the text data to be trained. The mask corresponding to each object training data can be determined according to the coordinate point data corresponding to each object training data generated by the trained large language model 301, and then the mask corresponding to each object training data can be Fourier encoded to obtain the mask feature vector corresponding to each object training data. The classification label data corresponding to each object training data can be input into the CLIP encoder 302 for processing to obtain the category feature vector set corresponding to the object training data. The attribute data corresponding to each object training data can be input into the CLIP encoder 302 to obtain the attribute feature vector set corresponding to the object training data. Then, the mask feature vector corresponding to the object training data, the category feature vector set corresponding to the object training data, and the attribute feature vector set corresponding to the object training data can be added to obtain the fused training vector corresponding to each layout training data. The fused training vector corresponding to each layout training data can be input into a multi-layer perceptron (MLP). 303 is processed to obtain a plurality of layout training vector sets corresponding to the text data to be trained, and then the reference image data corresponding to each text data to be trained can be denoised by the diffusion model 304 to obtain the noise image data corresponding to each text data to be trained, and the text feature vector corresponding to the text data to be trained, the mask feature vector corresponding to each object training data, and the layout training vector set corresponding to the text data to be trained can be input into the segmentation model 305 contained in the diffusion model 304. After multiple iterations, the diffusion model 304 gradually removes the noise and generates the target image data corresponding to the text data to be trained, and then the difference between the target image data corresponding to the text data to be trained and the label corresponding to the text data to be trained can be calculated by the mean square error loss to obtain the reconstruction loss corresponding to the diffusion model 304. At the same time, by training the segmentation model 305, the attention map comparison loss corresponding to the segmentation model 305 can be obtained. The reconstruction loss corresponding to the diffusion model 304 and the attention map comparison loss corresponding to the segmentation model 305 can be added to obtain the total loss corresponding to the diffusion model 304, and the parameters in the diffusion model 304 are updated according to the total loss corresponding to the diffusion model 304.

[0058] According to the technical solution provided by the embodiment of the present disclosure, by obtaining multiple text data to be trained, reference image data corresponding to the text data to be trained, and labels corresponding to each text data to be trained, the text data to be trained can be processed by the trained large language model, and the layout training data corresponding to the object training data can be extracted. Then, a fusion training vector corresponding to the layout training data can be generated according to the layout training data corresponding to the object training data, and the fusion training vectors corresponding to each layout training data can be hierarchically mapped to obtain a set of layout training vectors corresponding to each text data to be trained. The reference image data corresponding to the text data to be trained can be denoised to generate noise image data corresponding to the text data to be trained. Then, the segmentation model can be trained by comparing the real image with the attention vector obtained by the segmentation model. The attention map comparison loss is calculated by using the atlas. The noisy image data corresponding to the text data to be trained can be input into the diffusion model for denoising to generate training image data, and the reconstruction loss is calculated by comparing with the real image label. Then, the total loss can be determined according to the attention map comparison loss corresponding to the segmentation model and the reconstruction loss corresponding to the diffusion model, and the parameters of the diffusion model can be optimized through cyclic iteration. This enhances the robustness and accuracy of the diffusion model in processing complex text-to-image conversion tasks, improves the understanding and application ability of the image data generation model for text layout information, and thus improves the fidelity and consistency of the generated image data. By calculating the attention map comparison loss corresponding to the segmentation model, the attention to the key areas of the text layout is improved. Combined with the reconstruction loss, the overall similarity between the generated image data and the real image data is guaranteed.

[0059] In some embodiments, before inputting the noise image data corresponding to each text data to be trained into the diffusion model for processing to obtain the training image data corresponding to each text data to be trained, it also includes: performing image encoding processing on the real image data corresponding to each text data to be trained to obtain the training vector corresponding to each real image data; performing mask generation processing on the coordinate point data corresponding to each object training data to obtain the mask corresponding to each object training data; processing the mask corresponding to each object training data by Fourier encoding to obtain the mask training vector corresponding to each object training data; inputting the noise image data corresponding to each text data to be trained into the segmentation model for self-attention processing to obtain the self-attention vector corresponding to each text data to be trained; The self-attention vector corresponding to this data and the set of layout training vectors corresponding to each text data to be trained are gated for attention processing to obtain the gated attention vector corresponding to each text data to be trained; the mask training vector corresponding to each object training data and the gated attention vector corresponding to each text data to be trained are visually extracted to obtain the visual training vector corresponding to each object training data; the cross-attention image vector corresponding to each text data to be trained is determined according to the classification label data corresponding to each object training data and the visual training vector corresponding to each object training data; the mean square error loss calculation is performed on the cross-attention image vector corresponding to each text data to be trained and the training vector corresponding to each real image data to obtain the attention map comparison loss corresponding to the segmentation model.

[0060] Specifically, the real image data corresponding to each text data to be trained can be converted into a training vector corresponding to each real image data through image coding processing, and the coordinate point data corresponding to each object training data can be subjected to mask generation processing to obtain the mask corresponding to each object training data, and then the mask corresponding to each object training data can be processed through Fourier coding to obtain the mask training vector corresponding to each training data, and then the noisy image data corresponding to the training text data can be subjected to self-attention processing through the segmentation model to obtain the self-attention vector corresponding to the text data to be trained, and the self-attention vector corresponding to the text data to be trained and the layout training vector set corresponding to each text data to be trained can be subjected to gated attention processing to generate the gated attention vector corresponding to the text data to be trained, and the mask training vector and the gated attention vector can be combined to perform visual feature extraction to obtain the visual training vector corresponding to the object training data, and the cross-attention image vector corresponding to the text data to be trained can be determined according to the classification label data corresponding to each object training data and the visual training vector corresponding to each object training data, and the mean square error loss calculation is performed with the training vector corresponding to the real image data to obtain the attention map comparison loss corresponding to the segmentation model.

[0061] Among them, the training vector corresponding to the real image data can be a vector representation obtained by converting the real image data corresponding to the text data to be trained through image coding processing, which can be used in subsequent training and loss calculation processes. The mask corresponding to the object training data can be a binary image generated according to the coordinate point data corresponding to the object training data, which can be used to identify the position and shape of the training object in the target training image data. The mask training vector corresponding to the object training data can be a vector obtained by processing the mask corresponding to the object training data through Fourier coding, which can be used to characterize the spatial frequency characteristics and shape information of the mask corresponding to the object training data, and can be used for subsequent visual feature extraction and attention calculation. The self-attention vector corresponding to the text data to be trained can be a vector obtained by performing self-attention processing on the noise image data corresponding to the training text data through a segmentation model, which can be used to learn the dependency between different regions within the image data and can be used for subsequent gating. Attention processing, the gated attention vector corresponding to the text data to be trained can be a vector obtained by gated attention processing of the self-attention vector corresponding to the text data to be trained and the set of layout training vectors corresponding to the text data to be trained. The key areas in the image data can be weighted in combination with the text layout information to enhance the attention of the image data generation model to the layout information. The visual training vector corresponding to the object training data can be a vector obtained by visual feature extraction processing, which can be used to characterize the visual features of the training object in the image data, including but not limited to color or texture, etc., and can be used for subsequent calculation of cross-attention image vectors. The cross-attention image vector corresponding to the text data to be trained can be a vector calculated based on the classification label data corresponding to the object training data and the visual training vector corresponding to the object training data, which can be used to learn the classification information and visual features of the training object, and can also be used for comparison and loss calculation with the training vector of the real image data.

[0062] For example, according to Figure 4 As shown in the flow chart of the segmentation model training method, the segmentation model can be a U-Net module 400, and the noise image data corresponding to each text data to be trained can be self-attention processed in the U-Net module 400 to obtain the self-attention vector corresponding to each text data to be trained, and then the self-attention vector corresponding to each text data to be trained and the layout training vector set corresponding to each text data to be trained can be gated attention processed to obtain the gated attention vector corresponding to each text data to be trained, and the mask training vector corresponding to each object training data and the gated attention vector corresponding to each text data to be trained are processed by the visual feature extraction method to obtain the visual training vector corresponding to each object training data. The process can be expressed by an expression, for example,

[0063] oi =∑M i ⊙V′

[0064] Among them, M i It can be the mask training vector corresponding to the object training data obtained by processing the coordinate point data corresponding to the i-th object training data, o i It can be the visual training vector corresponding to the i-th object training data, V′ can be the gated attention vector corresponding to the text data to be trained, and ⊙ can be the dot product processing.

[0065] The classification label data corresponding to each object training data and the visual training vector corresponding to each object training data can be cross-attention calculated to obtain a cross-attention image vector corresponding to each text data to be trained. The difference between the cross-attention image vector corresponding to each text data to be trained and the training vector corresponding to each real image data can be calculated through the mean square error loss to obtain the attention map comparison loss corresponding to the segmentation model.

[0066] According to the technical solution provided by the embodiment of the present disclosure, the real image data corresponding to each text data to be trained is converted into a training vector corresponding to each real image data through image coding processing, and the coordinate point data corresponding to each object training data is subjected to mask generation processing to obtain the mask corresponding to each object training data, and then the mask corresponding to each object training data can be processed through Fourier coding to obtain the mask training vector corresponding to each training data, and then the noisy image data corresponding to the training text data can be subjected to self-attention processing through the segmentation model to obtain the self-attention vector corresponding to the text data to be trained, and the self-attention vector corresponding to the text data to be trained and the layout training vector set corresponding to each text data to be trained are subjected to gated attention processing to generate the gated attention vector corresponding to the text data to be trained, and the mask training vector can be combined with the mask training vector to obtain the gated attention vector corresponding to the text data to be trained. The mask training vector and the gated attention vector are used to extract visual features, and the visual training vector corresponding to the object training data is obtained. The cross-attention image vector corresponding to the text data to be trained can be determined according to the classification label data corresponding to each object training data and the visual training vector corresponding to each object training data. The mean square error loss is calculated with the training vector corresponding to the real image data to obtain the attention map comparison loss corresponding to the segmentation model. This enhances the segmentation model's ability to jointly understand the text layout and image visual features, improves the image data generation accuracy of the image data generation model in complex scenes, and improves the consistency between the generated image data and the real image data in layout and visual features. Through the mask training vector and the gated attention mechanism, the precision of controlling the degree of attention to text layout and visual features when generating images is improved.

[0067] In some embodiments, before obtaining the text data to be processed, it also includes: obtaining multiple text data to be trained, real layout information data corresponding to each text data to be trained, and preset low-rank matrix adjustment parameters; integrating the large language model and the preset low-rank matrix adjustment parameters to obtain the large language model corresponding to the preset low-rank matrix adjustment parameters; inputting each text data to be trained into the large language model corresponding to the preset low-rank matrix adjustment parameters for forward propagation processing to obtain the layout training data corresponding to each text data to be trained; determining the low-rank matrix adjustment loss corresponding to the large language model according to the layout training data corresponding to each text data to be trained and the real layout information data corresponding to each text data to be trained; and updating the parameters in the large language model according to the low-rank matrix adjustment loss corresponding to the large language model through loop iteration.

[0068] Specifically, multiple text data to be trained, real layout information data corresponding to each text data to be trained, and preset low-rank matrix adjustment parameters (Low-Rank Adaptation of Large Language Models, LoRA) can be obtained, and the LoRA parameters can be integrated into the large language model to obtain an adjusted model, and then the text data to be trained can be input into the adjusted large language model for forward propagation to obtain layout training data corresponding to each text data to be trained, and the low-rank matrix adjustment loss corresponding to the large language model can be calculated by comparing the layout training data corresponding to the text data to be trained with the real layout information data corresponding to the text data to be trained, and the parameters in the large language model can be updated according to the low-rank matrix adjustment loss corresponding to the large language model through a cyclic iteration method to optimize the large language model's understanding and generation capabilities of text layout information.

[0069] Among them, the large language model corresponding to the preset low-rank matrix adjustment parameters can be an adjusted model obtained by integrating LoRA parameters on the basis of the large language model. LoRA parameters can be used to adjust and optimize the internal structure and weight of the large language model when processing text layout information to improve the performance and accuracy of the large language model, the layout training data corresponding to the text data to be trained can be data describing the layout characteristics of the text, the layout training data corresponding to the text data to be trained can be obtained after the text data to be trained is processed and analyzed by the large language model, the layout training data corresponding to the text data to be trained can include the position, size, and / or shape of each training object in the text data to be trained, etc., which is not limited here, and can be used to guide the layout arrangement in the image data generation process, the low-rank matrix adjustment loss corresponding to the large language model can be the loss value calculated according to the difference between the layout training data corresponding to the text data to be trained and the real layout information data corresponding to the text data to be trained during the training process of the large language model, and the low-rank matrix adjustment loss corresponding to the large language model can be used to evaluate the accuracy and performance of the large language model when processing text layout information, and can be used as the basis for updating the parameters of the large language model.

[0070] For example, the pre-trained large language model can be fine-tuned through the LoRA method based on the pre-trained large language model, the LoRA parameters can be integrated into the large language model to obtain a large language model with LoRA parameters, and each text data to be trained is input into the large language model with LoRA parameters for forward propagation processing to obtain the layout training data corresponding to the text data to be trained, and the loss calculation is performed on the layout training data corresponding to the text data to be trained and the real layout information data corresponding to the text data to be trained to obtain the low-rank matrix adjustment loss corresponding to the large language model, and the parameters in the large language model are updated according to the low-rank matrix adjustment loss.

[0071] According to the technical solution provided by the embodiment of the present disclosure, by obtaining multiple text data to be trained, real layout information data corresponding to each text data to be trained, and preset low-rank matrix adjustment parameters, the LoRA parameters can be integrated into the large language model to obtain an adjusted model, and then the text data to be trained can be input into the adjusted large language model for forward propagation to obtain layout training data corresponding to each text data to be trained, and the low-rank matrix adjustment loss corresponding to the large language model can be calculated by comparing the layout training data corresponding to the text data to be trained with the real layout information data corresponding to the text data to be trained. The parameters in the large language model can be updated according to the low-rank matrix adjustment loss corresponding to the large language model in a cyclic iteration manner, thereby enhancing the accuracy and adaptability of the large language model in the text-to-image layout conversion task, improving the image data generation model's ability to capture and process complex text layout information, and improving the consistency between the generated layout training data and the real text layout data. The large language model is trained and optimized through low-rank matrix adjustment, and the rationality and accuracy of the layout of the large language model when generating image data are enhanced.

[0072] In some embodiments, the cross-attention image vector corresponding to each text data to be trained is determined based on the classification label data corresponding to each object training data and the visual training vector corresponding to each object training data, including: performing text encoding processing on the classification label data corresponding to each object training data to obtain the classification label vector corresponding to each object training data; performing weight conversion processing on the classification label vector corresponding to each object training data to obtain the text label vector corresponding to each object training data; performing cross-attention calculation processing on the text label vector corresponding to each object training data and the visual training vector corresponding to each object training data to obtain the cross-attention image vector corresponding to each text data to be trained.

[0073] Specifically, the classification label data corresponding to each object training data can be text encoded and converted into a classification label vector corresponding to each object training data. The classification label vector corresponding to the object training data can be weighted converted to obtain a text label vector corresponding to the object training data. Then, the text label vector corresponding to the object training data and the visual training vector corresponding to the object training data can be cross-attention calculated to obtain a cross-attention image vector corresponding to the text data to be trained.

[0074] Among them, the classification label vector corresponding to the object training data can be a vector obtained by converting the classification label data corresponding to the object training data through text encoding processing, the classification label vector corresponding to the object training data can be used to retain the semantic information of the classification label and facilitate subsequent calculation and processing, and the text label vector corresponding to the object training data can be a vector obtained after weight conversion processing of the classification label vector corresponding to the object training data, and the importance of different classification labels can be emphasized through weight conversion. The text label vector corresponding to the object training data contains the semantic information of the classification label, and the text label vector corresponding to the object training data can be used as input data for subsequent cross-attention calculation.

[0075] For example, the classification label data corresponding to the object training data can be processed by the CLIP text encoder to obtain the classification label vectors corresponding to each object training data, and then the classification label vectors corresponding to each object training data can be converted into text tag vectors corresponding to each object training data through weight conversion, and then the text tag vectors corresponding to each object training data and the visual training vectors corresponding to each object training data can be cross-attention calculated to obtain the cross-attention image vectors corresponding to each text data to be trained.

[0076] According to the technical solution provided by the embodiments of the present disclosure, by performing text encoding processing on the classification label data corresponding to each object training data and converting them into classification label vectors corresponding to each object training data, the classification label vectors corresponding to the object training data can be weighted converted to obtain text label vectors corresponding to the object training data, and then the text label vectors corresponding to the object training data and the visual training vectors corresponding to the object training data can be cross-attention calculated to obtain cross-attention image vectors corresponding to the text data to be trained, thereby enhancing the joint processing capability of object classification information and visual features in the image data generation process, improving the classification accuracy and visual consistency of the generated image data, and improving the fidelity and effect of image data generation.

[0077] In some embodiments, the noise image data corresponding to each text data to be trained is input into the segmentation model for self-attention processing to obtain the self-attention vector corresponding to each text data to be trained, including: performing linear transformation processing on the noise image data corresponding to each text data to be trained to obtain the key vector corresponding to each noise image data, the value vector corresponding to each noise image data, and the query vector corresponding to each noise image data; performing attention weight calculation processing on the key vector corresponding to each noise image data and the query vector corresponding to each noise image data to obtain the attention weight matrix corresponding to each noise image data; performing weighted sum processing on the attention weight matrix corresponding to each noise image data and the value vector corresponding to each noise image data to obtain the self-attention vector corresponding to each text data to be trained.

[0078] Specifically, the noise image data corresponding to each text data to be trained can be converted into a key vector corresponding to each noise image data, a value vector corresponding to each noise image data, and a query vector corresponding to each noise image data through linear transformation processing, and then the attention weight matrix can be obtained by performing attention weight calculation on the key vector corresponding to each noise image data and the query vector corresponding to each noise image data. The value vector corresponding to the noise image data can be weighted and summed according to the attention weight matrix to obtain the self-attention vector corresponding to the text data to be trained.

[0079] Among them, the key vector, value vector, and query vector corresponding to the noisy image data can be vectors obtained from the noisy image data through linear transformation processing, wherein the key vector can be used to match the query vector, the value vector can be used to provide the original data for weighted summation, the query vector can be used to guide the calculation of attention weights, and the attention weight matrix corresponding to the noisy image data can be a matrix obtained by calculating the similarity between the key vector and the query vector, and each element of the attention weight matrix corresponding to the noisy image data can be used to characterize the attention weights between different regions in the noisy image data.

[0080] According to the technical solution provided by the embodiments of the present disclosure, the noise image data corresponding to each text data to be trained are converted into a key vector corresponding to each noise image data, a value vector corresponding to each noise image data, and a query vector corresponding to each noise image data through linear transformation processing, and then the attention weights can be calculated for the key vectors corresponding to each noise image data and the query vectors corresponding to each noise image data to obtain an attention weight matrix, and the value vectors corresponding to the noise image data can be weighted summed according to the attention weight matrix to obtain the self-attention vector corresponding to the text data to be trained, thereby enhancing the segmentation model's understanding and processing capabilities of the noise image data, improving the accuracy of the image data generation model in capturing the internal dependencies of the image, and improving the precision and realism of the image data generation.

[0081] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0082] The following are embodiments of the device disclosed herein, which can be used to execute the method embodiments disclosed herein. For details not disclosed in the device embodiments disclosed herein, please refer to the method embodiments disclosed herein.

[0083] Figure 5 Schematic diagram of an image data generating device provided by an embodiment of the present disclosure. Figure 4 As shown, the image data generating device includes:

[0084] An acquisition module 501 is used to acquire text data to be processed, where the text data to be processed includes at least one object data;

[0085] A first processing module 502 is used to input the text data to be processed into the trained large language model to process the text data to be processed, and obtain layout information data corresponding to each object data, wherein the large language model is trained based on the text data to be trained, the real layout information data corresponding to each text data to be trained, and the preset low-rank matrix adjustment parameters;

[0086] A determination module 503, configured to determine a fusion feature vector corresponding to the layout information data according to the layout information data corresponding to each object data;

[0087] The second processing module 504 is used to perform hierarchical mapping processing on the fused feature vector corresponding to the layout information data to obtain a set of layout feature vectors corresponding to the text data to be processed;

[0088] The third processing module 505 is used to input the layout feature vector set corresponding to the text data to be processed into the trained diffusion model for processing to obtain target image data containing at least one object. The diffusion model is trained based on the text data to be trained, the reference image data corresponding to each text data to be trained, and the label corresponding to each text data to be trained.

[0089] According to the technical solution provided by the embodiment of the present disclosure, by acquiring text data to be processed containing at least one object data, and processing the acquired text data to be processed in a trained large language model through a low-rank matrix adjustment method, layout information data of each object data in the text data to be processed is obtained, and feature fusion processing can be performed on the layout information data of each object data in the text data to be processed to obtain a fused feature vector corresponding to the layout information data, and then hierarchical mapping processing can be performed on the fused feature vector corresponding to the layout information data through a multi-layer perceptron to obtain a set of layout feature vectors corresponding to the text data to be processed, and the set of layout feature vectors corresponding to the text data to be processed is processed through a trained diffusion model to obtain target image data containing at least one object, thereby improving the fidelity in the multi-subject image generation process, improving the accuracy of the multi-subject layout, improving the efficiency and accuracy of image data generation, ensuring the quality and authenticity of the image data, and enhancing the understanding of complex layout information and multiple object information.

[0090] In some embodiments, the determination module 503 is specifically used to perform feature extraction processing on the text data to be processed to obtain a text feature vector corresponding to the text data to be processed; perform mask generation processing on the coordinate point data corresponding to each object data to obtain a mask corresponding to each object data; process the mask corresponding to each object data through Fourier encoding to obtain a mask feature vector corresponding to each object data; input the classification label data corresponding to each object data into the text encoder for processing to obtain a set of category feature vectors corresponding to the object data; perform text encoding processing on the attribute data corresponding to multiple object data to obtain a set of attribute feature vectors corresponding to the object data; add the mask feature vectors corresponding to each object data, the set of category feature vectors corresponding to the object data, and the set of attribute feature vectors corresponding to the object data to obtain a fusion feature vector corresponding to the layout information data.

[0091] In some embodiments, the image data generating device is further used to obtain a plurality of text data to be trained, reference image data corresponding to each text data to be trained, and a label corresponding to each text data to be trained, wherein the text data to be trained includes at least one object training data, and the label corresponding to the text data to be trained is the real image data corresponding to the text data to be trained; each text data to be trained is input into the trained large language model to process the text data to be trained, and the layout training data corresponding to each object training data is obtained, and the layout training data corresponding to the object training data includes the coordinate point data corresponding to each object training data, the classification label data corresponding to each object training data, and the attribute data corresponding to each object training data; according to the layout training data corresponding to each object training data, the fusion training vector corresponding to each layout training data is determined; the fusion training vector corresponding to each layout information data is hierarchically mapped to obtain the layout training vectors corresponding to the plurality of text data to be trained The invention relates to a method for preparing a set of text data to be trained and a method for obtaining a set of reference image data corresponding to each text data to be trained; performing noise processing on the reference image data corresponding to each text data to be trained to obtain the noise image data corresponding to each text data to be trained; performing training processing on the segmentation model based on the real image data corresponding to each text data to be trained, the coordinate point data corresponding to each object training data, the noise image data corresponding to each text data to be trained, and the classification label data corresponding to each object training data to obtain the attention map comparison loss corresponding to the segmentation model; inputting the noise image data corresponding to each text data to be trained into the diffusion model for processing to obtain the training image data corresponding to each text data to be trained; determining the reconstruction loss corresponding to the diffusion model according to the training image data corresponding to each text data to be trained and the label corresponding to each text data to be trained; determining the total loss corresponding to the diffusion model according to the attention map comparison loss corresponding to the segmentation model and the reconstruction loss corresponding to the diffusion model; updating the parameters in the diffusion model according to the total loss corresponding to the diffusion model through loop iteration.

[0092] In some embodiments, the above-mentioned image data generating device is also used to perform image encoding processing on the real image data corresponding to each text data to be trained, so as to obtain the training vector corresponding to each real image data; perform mask generation processing on the coordinate point data corresponding to each object training data, so as to obtain the mask corresponding to each object training data; process the mask corresponding to each object training data by Fourier encoding, so as to obtain the mask training vector corresponding to each object training data; input the noise image data corresponding to each text data to be trained into the segmentation model for self-attention processing, so as to obtain the self-attention vector corresponding to each text data to be trained; and perform self-attention processing on the self-attention vector corresponding to each text data to be trained and the self-attention vector corresponding to each text data to be trained. The corresponding layout training vector set is gated for attention processing to obtain the gated attention vector corresponding to each text data to be trained; the mask training vector corresponding to each object training data and the gated attention vector corresponding to each text data to be trained are processed for visual feature extraction to obtain the visual training vector corresponding to each object training data; the cross-attention image vector corresponding to each text data to be trained is determined according to the classification label data corresponding to each object training data and the visual training vector corresponding to each object training data; the cross-attention image vector corresponding to each text data to be trained and the training vector corresponding to each real image data are calculated for mean square error loss to obtain the attention map comparison loss corresponding to the segmentation model.

[0093] In some embodiments, the above-mentioned image data generating device is also used to obtain multiple text data to be trained, real layout information data corresponding to each text data to be trained, and preset low-rank matrix adjustment parameters; integrate the large language model and the preset low-rank matrix adjustment parameters to obtain the large language model corresponding to the preset low-rank matrix adjustment parameters; input each text data to be trained into the large language model corresponding to the preset low-rank matrix adjustment parameters for forward propagation processing to obtain the layout training data corresponding to each text data to be trained; determine the low-rank matrix adjustment loss corresponding to the large language model according to the layout training data corresponding to each text data to be trained and the real layout information data corresponding to each text data to be trained; and update the parameters in the large language model according to the low-rank matrix adjustment loss corresponding to the large language model through loop iteration.

[0094] In some embodiments, the cross-attention image vector corresponding to each text data to be trained is determined based on the classification label data corresponding to each object training data and the visual training vector corresponding to each object training data. The cross-attention image vector corresponding to each text data to be trained is specifically used to perform text encoding processing on the classification label data corresponding to each object training data to obtain the classification label vector corresponding to each object training data; perform weight conversion processing on the classification label vector corresponding to each object training data to obtain the text label vector corresponding to each object training data; perform cross-attention calculation processing on the text label vector corresponding to each object training data and the visual training vector corresponding to each object training data to obtain the cross-attention image vector corresponding to each text data to be trained.

[0095] In some embodiments, the noise image data corresponding to each text data to be trained is input into the segmentation model for self-attention processing to obtain the self-attention vector corresponding to each text data to be trained. Specifically, the noise image data corresponding to each text data to be trained is linearly transformed to obtain the key vector corresponding to each noise image data, the value vector corresponding to each noise image data, and the query vector corresponding to each noise image data; the key vector corresponding to each noise image data and the query vector corresponding to each noise image data are calculated with attention weights to obtain the attention weight matrix corresponding to each noise image data; the attention weight matrix corresponding to each noise image data and the value vector corresponding to each noise image data are weighted summed to obtain the self-attention vector corresponding to each text data to be trained.

[0096] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure.

[0097] Figure 6 Schematic diagram of an electronic device 6 provided in an embodiment of the present disclosure. Figure 6 As shown, the electronic device 6 of this embodiment includes: a processor 601, a memory 602, and a computer program 603 stored in the memory 602 and executable on the processor 601. When the processor 601 executes the computer program 603, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 601 executes the computer program 603, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0098] The electronic device 6 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 6 may include, but is not limited to, a processor 601 and a memory 602. Those skilled in the art will appreciate that Figure 6The electronic device 6 is merely an example and does not limit the electronic device 6 . The electronic device 6 may include more or less components than those shown in the figure, or different components.

[0099] The processor 601 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0100] The memory 602 may be an internal storage unit of the electronic device 6, for example, a hard disk or memory of the electronic device 6. The memory 602 may also be an external storage device of the electronic device 6, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 6. The memory 602 may also include both an internal storage unit of the electronic device 6 and an external storage device. The memory 602 is used to store computer programs and other programs and data required by the electronic device.

[0101] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0102] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium (e.g., a computer-readable storage medium). Based on this understanding, the present disclosure implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable storage medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0103] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should all be included in the protection scope of the present disclosure.

Claims

1. A method for generating image data, characterized in that: include: Acquire text data to be processed, wherein the text data to be processed includes at least one object data; Inputting the text data to be processed into a trained large language model to process the text data to be processed, and obtaining layout information data corresponding to each of the object data, wherein the large language model is trained based on the text data to be trained, the real layout information data corresponding to each of the text data to be trained, and a preset low-rank matrix adjustment parameter; Determine, according to the layout information data corresponding to each of the object data, a fusion feature vector corresponding to the layout information data; Performing hierarchical mapping processing on the fused feature vectors corresponding to the layout information data to obtain a set of layout feature vectors corresponding to the text data to be processed; The layout feature vector set corresponding to the text data to be processed is input into a trained diffusion model for processing to obtain target image data containing at least one object, wherein the diffusion model is trained based on the large language model, the text data to be trained, the reference image data corresponding to each of the text data to be trained, and the label corresponding to each of the text data to be trained.

2. The image data generating method according to claim 1, characterized in that: The layout information data corresponding to the object data includes coordinate point data corresponding to the object data, classification label data corresponding to the object data, and attribute data corresponding to the object data; The step of determining the fused feature vector corresponding to the layout information data according to the layout information data corresponding to each of the object data comprises: Performing feature extraction processing on the text data to be processed to obtain a text feature vector corresponding to the text data to be processed; Performing mask generation processing on the coordinate point data corresponding to each of the object data to obtain a mask corresponding to each of the object data; Processing the mask corresponding to each of the object data by Fourier coding to obtain the mask feature vector corresponding to each of the object data; Inputting the classification label data corresponding to each of the object data into a text encoder for processing to obtain a set of category feature vectors corresponding to the object data; Performing text encoding processing on the attribute data corresponding to the plurality of object data to obtain a set of attribute feature vectors corresponding to the object data; The mask feature vectors corresponding to each of the object data, the set of category feature vectors corresponding to the object data, and the set of attribute feature vectors corresponding to the object data are added together to obtain a fused feature vector corresponding to the layout information data.

3. The image data generating method according to claim 1, characterized in that: The diffusion model includes a segmentation model; Before obtaining the text data to be processed, the method further includes: Acquire a plurality of text data to be trained, reference image data corresponding to each of the text data to be trained, and a label corresponding to each of the text data to be trained, wherein the text data to be trained includes at least one object training data, and the label corresponding to the text data to be trained is the real image data corresponding to the text data to be trained; Inputting each of the text data to be trained into the trained large language model to process the text data to be trained, and obtaining layout training data corresponding to each of the object training data, wherein the layout training data corresponding to the object training data includes coordinate point data corresponding to each of the object training data, classification label data corresponding to each of the object training data, and attribute data corresponding to each of the object training data; Determining, according to the layout training data corresponding to each of the object training data, a fusion training vector corresponding to each of the layout training data; Performing hierarchical mapping processing on the fused training vectors corresponding to each of the layout information data to obtain a plurality of layout training vector sets corresponding to the text data to be trained; Performing noise processing on the reference image data corresponding to each of the text data to be trained, to obtain noise image data corresponding to each of the text data to be trained; The segmentation model is trained based on the real image data corresponding to each of the text data to be trained, the coordinate point data corresponding to each of the object training data, the noise image data corresponding to each of the text data to be trained, and the classification label data corresponding to each of the object training data, to obtain the attention map comparison loss corresponding to the segmentation model; Inputting the noise image data corresponding to each of the text data to be trained into the diffusion model for processing to obtain the training image data corresponding to each of the text data to be trained; Determine the reconstruction loss corresponding to the diffusion model according to the training image data corresponding to each of the to-be-trained text data and the label corresponding to each of the to-be-trained text data; Determine the total loss corresponding to the diffusion model according to the attention map comparison loss corresponding to the segmentation model and the reconstruction loss corresponding to the diffusion model; Through loop iteration, the parameters in the diffusion model are updated according to the total loss corresponding to the diffusion model.

4. The image data generating method according to claim 3, characterized in that: Before inputting the noise image data corresponding to each of the to-be-trained text data into the diffusion model for processing to obtain the training image data corresponding to each of the to-be-trained text data, the method further includes: Performing image coding processing on the real image data corresponding to each of the text data to be trained to obtain a training vector corresponding to each of the real image data; Performing mask generation processing on the coordinate point data corresponding to each of the object training data to obtain a mask corresponding to each of the object training data; Processing the mask corresponding to each of the object training data by Fourier coding to obtain the mask training vector corresponding to each of the object training data; Inputting the noise image data corresponding to each of the text data to be trained into the segmentation model for self-attention processing to obtain the self-attention vector corresponding to each of the text data to be trained; Performing gated attention processing on the self-attention vector corresponding to each of the text data to be trained and the layout training vector set corresponding to each of the text data to be trained to obtain the gated attention vector corresponding to each of the text data to be trained; Performing visual feature extraction processing on the mask training vector corresponding to each of the object training data and the gated attention vector corresponding to each of the text data to be trained to obtain a visual training vector corresponding to each of the object training data; Determine the cross-attention image vector corresponding to each of the text data to be trained according to the classification label data corresponding to each of the object training data and the visual training vector corresponding to each of the object training data; The mean square error loss calculation is performed on the cross-attention image vectors corresponding to each of the text data to be trained and the training vectors corresponding to each of the real image data to obtain the attention map comparison loss corresponding to the segmentation model.

5. The image data generating method according to claim 1, characterized in that: Before obtaining the text data to be processed, the method further includes: Acquire a plurality of the text data to be trained, real layout information data corresponding to each of the text data to be trained, and the preset low-rank matrix adjustment parameters; Integrate the large language model and the preset low-rank matrix adjustment parameters to obtain a large language model corresponding to the preset low-rank matrix adjustment parameters; Inputting each of the text data to be trained into the large language model corresponding to the preset low-rank matrix adjustment parameter for forward propagation processing to obtain layout training data corresponding to each of the text data to be trained; Determine the low-rank matrix adjustment loss corresponding to the large language model according to the layout training data corresponding to each of the text data to be trained and the real layout information data corresponding to each of the text data to be trained; Through loop iteration, the parameters in the large language model are updated according to the low-rank matrix adjustment loss corresponding to the large language model.

6. The image data generating method according to claim 4, characterized in that: The step of determining the cross-attention image vector corresponding to each of the text data to be trained according to the classification label data corresponding to each of the object training data and the visual training vector corresponding to each of the object training data comprises: Performing text encoding processing on the classification label data corresponding to each of the object training data to obtain the classification label vector corresponding to each of the object training data; Performing weight conversion processing on the classification label vector corresponding to each of the object training data to obtain the text label vector corresponding to each of the object training data; A cross-attention calculation process is performed on the text tag vector corresponding to each of the object training data and the visual training vector corresponding to each of the object training data to obtain a cross-attention image vector corresponding to each of the text data to be trained.

7. The image data generating method according to claim 4, characterized in that: The step of inputting the noise image data corresponding to each of the text data to be trained into the segmentation model for self-attention processing to obtain the self-attention vector corresponding to each of the text data to be trained includes: Performing linear transformation processing on the noise image data corresponding to each of the to-be-trained text data to obtain a key vector corresponding to each of the noise image data, a value vector corresponding to each of the noise image data, and a query vector corresponding to each of the noise image data; Performing attention weight calculation processing on the key vector corresponding to each of the noise image data and the query vector corresponding to each of the noise image data to obtain an attention weight matrix corresponding to each of the noise image data; The attention weight matrix corresponding to each of the noise image data and the value vector corresponding to each of the noise image data are weighted and summed to obtain the self-attention vector corresponding to each of the text data to be trained.

8. An image data generating device, characterized in that: include: An acquisition module, used for acquiring text data to be processed, wherein the text data to be processed includes at least one object data; A first processing module is used to input the text data to be processed into a trained large language model to process the text data to be processed, and obtain layout information data corresponding to each of the object data, wherein the large language model is trained based on the text data to be trained, the real layout information data corresponding to each of the text data to be trained, and a preset low-rank matrix adjustment parameter; A determination module, used to determine the fused feature vector corresponding to the layout information data according to the layout information data corresponding to each of the object data; A second processing module is used to perform hierarchical mapping processing on the fused feature vector corresponding to the layout information data to obtain a set of layout feature vectors corresponding to the text data to be processed; The third processing module is used to input the layout feature vector set corresponding to the text data to be processed into the trained diffusion model for processing to obtain target image data containing at least one object. The diffusion model is trained based on the text data to be trained, the reference image data corresponding to each of the text data to be trained, and the labels corresponding to each of the text data to be trained.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images