Multi-agent collaborative end-to-end text and image generation method and system

Abstract

The application discloses a kind of multi-agent collaborative end-to-end text and image generation method and system.The system includes user input module, semantic feature first agent, logic constraint second agent, format constraint third agent, image-text generation fourth agent and document generation fifth agent.Semantic feature first agent extracts text semantic features;Logic constraint second agent extracts style features;Format constraint third agent identifies layout rules;Image-text generation fourth agent performs double-constraint decoding based on the improved conditional generative adversarial network, introduces semantic style collaborative loss and multi-scale consistency constraint loss, and synchronously generates semantic-consistent and style-matched image-text content;Document generation fifth agent automatically completes image-text layout according to format features, and outputs editable documents.The application can avoid modal fragmentation, detail loss and manual layout problems, and improve the quality and efficiency of image-text document generation.

Description

Technical Field

[0001] This invention belongs to the field of computer data processing and artificial intelligence technology, specifically relating to an end-to-end text and image generation method and system for multi-agent collaboration. Background Technology

[0002] With the rapid development of artificial intelligence technology, the fields of text and image generation have ushered in unprecedented development opportunities. Text generation systems, as important tools for improving writing efficiency and quality, have been widely applied in numerous scenarios. The core objective is to ensure the authenticity and integrity of generated text and images. The multi-agent generation of style, semantic, and format features ensures the integrity and authenticity of documents. The emergence of generative adversarial networks has driven advancements in image generation technology, making high-resolution and realistic image generation a reality. These technological advancements provide strong support for meeting diverse needs in text creation and graphic design, demonstrating enormous application potential.

[0003] In existing technologies, multi-agent text generation systems are an advanced application of artificial intelligence. They involve multiple agents collaborating to generate text and image content, typically integrating natural language processing, computer vision, and machine learning techniques to achieve highly automated and personalized content creation. This area has become a research hotspot in the fields of natural language processing and cross-modal generation. Chinese patent document CN118917294B provides a smart writing method and system based on role-based intelligent agents. While it plays a significant role in improving writing efficiency and quality, it still faces the following core technical challenges when dealing with complex document generation tasks involving mixed text and images: First, there is modal fragmentation and style conflict. Existing systems typically treat text generation and image generation as two separate pipelines. The lack of interaction between the text generation model and the image generation model at the underlying feature level leads to a mismatch between the style of the generated illustrations and the context of the text description.

[0004] Secondly, there is the issue of detail loss and structural distortion. Traditional conditional generative adversarial networks (GANs) often focus only on global structure while neglecting local texture when generating high-resolution images, or vice versa. A single-scale discriminator struggles to simultaneously guarantee the semantic structural integrity of an image and the clarity of its texture details.

[0005] Third, it lacks end-to-end typesetting capabilities. Most existing technologies only output plain text or independent image files, and cannot automatically complete "text and image mixed layout" according to specific industry standard templates. Users still need to spend a lot of time manually adjusting the format.

[0006] Therefore, there is an urgent need for a system that can collaboratively handle semantic, style, and format constraints and achieve one-stop editing document generation to overcome the aforementioned technical deficiencies. Summary of the Invention

[0007] To address the issues of inconsistent content styles, low efficiency in text and image layout, and poor detail fidelity in related technologies, this invention provides an end-to-end text and image generation method and system based on multi-agent collaboration, in order to improve the consistency of text and image content, image fidelity, and document automation.

[0008] The technical solution of this invention is implemented as follows: a multi-agent collaborative end-to-end text and image generation system, comprising: User input module: Used to receive text content, style transfer templates, and target specification templates input by users; The first intelligent agent for semantic features: connected to the user input module, it receives the text content, performs syntactic analysis on the text using a semantic role labeling model and a conditional random field, and generates a high-dimensional semantic feature vector. ; The second intelligent agent, bound by logical constraints, is connected to the user input module and is used to receive the style transfer template. After centering and whitening the template image data, it extracts the first d principal components using principal component analysis to generate a style feature vector. ; The third intelligent agent for format constraint is connected to the user input module and is used to receive the target specification template, identify non-text element regions and layout rules in the template using edge detection and contour extraction algorithms, and generate a format feature vector. ; The fourth agent for image and text generation connects the first agent for semantic features and the second agent for logical constraints, respectively, and is used for dual-constraint decoding based on an improved conditional generative adversarial network to simultaneously generate and... Semantic consistency and possess Image content with stylistic characteristics With text content ; The fifth agent for document generation: connects the fourth agent for image and text generation and the third agent for format constraints, respectively, and is used to receive... , and Based on a document layout optimization algorithm, the generated text and images are filled into a formatting frame, and an editable document is output.

[0009] This invention employs a semantic feature first agent, a logical constraint second agent, and a format constraint third agent to extract semantic features of text, style features of style transfer templates, and format features of target specification templates in parallel. A fourth agent for image-text generation uses an improved conditional generative adversarial network for dual-constraint decoding, simultaneously generating images and text consistent with semantic features and possessing specified style characteristics. This forces cross-modal alignment of image and text content at the underlying feature level, effectively avoiding modal fragmentation and style conflicts. Simultaneously, a fifth agent for document generation automatically fills the generated image and text content into a format framework based on pre-extracted format feature vectors and outputs an editable document. This achieves end-to-end automated typesetting from user input to the final document, significantly reducing the workload of manual format adjustments and improving the generation efficiency and content consistency of mixed image and text documents.

[0010] As a further improvement to the above scheme, the fourth intelligent agent for generating images and text includes a generator network and a discriminator network; The generator network employs a dual-stream encoder-decoder structure, processing the high-dimensional semantic feature vectors separately. and style feature vector Encoding fusion is performed; the loss function of the fourth agent for image and text generation is... Including basic combat losses Semantic-style collaborative loss and multi-scale consistency constraint loss .

[0011] Semantic and style features are encoded and fused using a dual-stream encoder-decoder structure, and semantic features are introduced. Style co-alignment loss and multi-scale consistency constraint loss can force the alignment of cross-modal features while ensuring the fidelity of image texture, structure and semantics, significantly improving the content consistency of generated text and the quality of image details.

[0012] As a further improvement to the above scheme, the semantic-style collaborative loss... The formula used to constrain the similarity relationship of generated content in the feature space is as follows: ; in, As a cooperating factor, from the formula =0.8 Dynamic calculation; The cosine similarity function is used. and These are pre-trained image encoders and text encoders, respectively. Describing the L2 norm, For image content, This refers to the text content.

[0013] By dynamically adjusting the constraint strength of semantics and style through collaborative factors and using cosine similarity to force alignment of the image and text feature space, the problem of cross-modal content style fragmentation and semantic conflict can be effectively solved, thereby improving the consistency of generated images and text.

[0014] As a further improvement to the above scheme, the multi-scale consistency constraint loss... The formula used to constrain the fidelity of the generated image in terms of shallow texture, mid-level structure, and deep semantics is as follows: ; Where K=3, This represents the feature mapping of the k-th layer of the feature extraction network. The weight coefficients at the k-th scale satisfy... =1, Describing the L1 norm, For image content, These are real images from a standard image dataset. Represents a real image Data distribution Find the expected value.

[0015] By constraining the L1 norm at three scales—shallow texture, mid-level structure, and deep semantics—the generated image is forced to maintain consistency with the real image in the feature distribution of each layer. This ensures both the clarity of image details and the integrity of semantic structure, effectively avoiding the loss of details or structural distortion.

[0016] As a further improvement to the above scheme, the fourth intelligent agent for image and text generation is equipped with an adaptive weight adjustment mechanism to dynamically balance the loss weights. The specific formula is as follows: ; in, and These are the weight coefficients for the semantic-style collaborative loss and the multi-scale consistency constraint loss, respectively.

[0017] By dynamically allocating loss weights based on the magnitude of semantic and style feature vectors, the system can adapt to different input characteristics without manual parameter tuning, thus enhancing the generalization ability of image and text generation.

[0018] An end-to-end text and image generation method based on multi-agent collaboration, applied to the system described above, includes the following steps: S1. Obtain text content, style transfer template, and target specification template through the user interface; S2. Parallel Feature Extraction: Extract semantic feature vectors from the text using the first agent of semantic features. The style feature vector of the style transfer template is extracted by the second intelligent agent through logical constraints. The third-party agent extracts the format feature vector of the target specification template through format constraints. ; S3, the fourth intelligent agent for image and text generation is based on and Perform dual-constraint decoding by alternately optimizing the generator and discriminator parameters until the semantic-style co-operation loss and multi-scale consistency constraint loss converge, and output the image content. With text content ; S4, Document generation, the fifth intelligent agent, based on... The limited layout structure will and Reorganize and format the document to generate the final document.

[0019] By extracting semantic, style, and format features in parallel and combining dual-constraint decoding with automatic layout reorganization, end-to-end generation from input to editable documents is achieved, significantly improving the efficiency of mixed text and image layout and content consistency.

[0020] As a further improvement to the above scheme, in step S2, the logically constrained second agent extracts the style feature vector of the style transfer template. Specifically: The style template collection is processed by grayscale conversion and normalization. Calculate the covariance matrix of the pixel matrix; Perform singular value decomposition on the covariance matrix; The top d eigenvectors with the largest eigenvalues ​​are selected as the principal component projection directions to obtain the dimensionality-reduced style feature vectors. .

[0021] By selecting principal components after grayscale conversion, normalization, covariance matrix calculation, and singular value decomposition, the core style features of the style transfer template can be effectively extracted and dimensionality reduced, reducing data redundancy and improving the efficiency and stability of subsequent image and text generation.

[0022] As a further improvement to the above scheme, in step S3, the convergence criterion is as follows: <0.01 and < scale , ( scale =0.05); in, Let be the semantic-style collaborative loss function value in the t-th iteration. Let be the semantic-style collaborative loss function value in the (t-100th)th iteration. For multi-scale consistency constraint loss, scale This is the convergence threshold for the multi-scale consistency constraint loss.

[0023] By setting dual judgment criteria of relative change rate and absolute threshold, the training convergence state can be accurately determined, avoiding early stopping or underfitting caused by loss fluctuations, thereby ensuring the training stability and generation quality of the image and text generation model.

[0024] Compared with traditional processing methods, the present invention has the following advantages: (1) Existing multi-agent architectures only realize the division of labor and cooperation in the task process, while this invention uses semantic-style collaborative loss. This establishes a low-level feature association between text and images, ensuring that the generated content maintains a high degree of consistency in semantics and style, thereby avoiding the problems of modal fragmentation and style conflict.

[0025] (2) Existing technologies do not have a dedicated optimization mechanism for balancing details and structure in image generation, while this invention uses multi-scale consistency constraint loss. By constraining the generated image in three dimensions—shallow texture, mid-level structure, and deep semantics—the loss of detail and structural distortion are avoided, thereby improving the fidelity of the text and image content.

[0026] (3) The system of the present invention integrates a full-link intelligent agent from feature analysis to final typesetting, which can automatically complete the mixed text and image typesetting according to industry standard templates, realize end-to-end generation from input materials to output editable documents, and reduce the workload of manual typesetting.

[0027] (4) The adaptability of existing multi-agent systems depends on the task allocation logic of the agents, while the present invention dynamically balances semantics through an adaptive weight adjustment mechanism. The weights of style collaboration loss and multi-scale constraint loss enable the system to automatically adapt to text inputs of different lengths and style templates of different complexities without the need for manual adjustment of hyperparameters. Attached Figure Description

[0028] Figure 1 A schematic diagram of the overall architecture of an end-to-end text and image generation system with multi-agent collaboration provided by the present invention; Figure 2 This is a schematic diagram of the network structure of the dual-constraint decoding mechanism in the fourth intelligent agent for image and text generation in this invention; Figure 3 The flowchart is provided by the present invention for an end-to-end text and image generation method based on multi-agent collaboration. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] Example 1 like Figure 1 As shown, this embodiment provides a multi-agent collaborative end-to-end text and image generation system, including a user input module, a semantic feature first agent, a logical constraint second agent, a format constraint third agent, an image and text generation fourth agent, and a document generation fifth agent. In this embodiment, the system runs on a high-performance computing server, and the hardware environment includes: a CPU (Intel Xeon Gold), a GPU (NVIDIA Tesla A100, used for deep learning model inference), and a large capacity of memory.

[0031] The user input module is used to receive text content, style transfer templates, and target specification templates input by the user.

[0032] The first intelligent agent for semantic features is connected to the user input module. It receives the text content, performs syntactic analysis on the text using a semantic role labeling model and a conditional random field, and generates a high-dimensional semantic feature vector. .

[0033] The core of the semantic feature-based first agent is a semantic role labeling network fine-tuned from a pre-trained BERT model. In this embodiment, the input is natural language text T input by the user. First, word segmentation and part-of-speech tagging are performed. Then, a Transformer encoder is used to extract context vectors. Finally, a CRF layer outputs semantic labels for each word (such as agent, patient, time, and location), and the label sequence is mapped to a fixed-dimensional semantic feature vector. .

[0034] The second intelligent agent, logically constrained and connected to the user input module, receives the style transfer template, performs centering and whitening processing on the template image data, and then extracts the first d principal components using principal component analysis to generate a style feature vector. .

[0035] The second agent, bound by logical constraints, primarily executes a statistical dimensionality reduction algorithm: first, it reads the style template image set. Then scale all images to a uniform size (e.g.) The mean vector of the data matrix is ​​flattened into a vector; then the mean vector of the data matrix is ​​calculated and centered; finally, the covariance matrix is ​​calculated. , where n is the number of style template images, and X is the decentralized image pixel data matrix (dimension n×p, where p is the total number of pixels in each image after flattening). get The covariance matrix is ​​used for subsequent eigenvalue decomposition. Perform eigenvalue decomposition and select the top d eigenvectors with the largest eigenvalues ​​(in this embodiment) Output style feature vector .

[0036] A third-party intelligent agent, connected to the user input module, receives the target specification template, uses edge detection and contour extraction algorithms to identify non-text element regions and layout rules in the template, and generates a format feature vector. .

[0037] The fourth agent for image and text generation connects the first agent for semantic features and the second agent for logical constraints. It is used for dual-constraint decoding based on an improved conditional generative adversarial network, simultaneously generating and... Semantic consistency and possess Image content with stylistic characteristics With text content .

[0038] like Figure 2 As shown, the fourth intelligent agent for image and text generation includes a generator G and a discriminator D. The generator network adopts a two-stream encoder-decoder structure (specifically, the U-Net architecture), and an adaptive instance normalization layer AdaIN is introduced at the bottleneck layer Bottleneck to... Injected into the feature map, respectively for and Encode fusion is performed. The loss function for generating the fourth agent from images and text is also considered. Including basic combat losses Semantic-style collaborative loss and multi-scale consistency constraint loss .

[0039] The semantic-style collaborative loss The formula used to constrain the similarity relationship of generated content in the feature space is as follows: ; in, As a cooperating factor, from the formula =0.8 Dynamic calculation is used when the magnitude of the input semantic features is much larger than that of the style features. Increase the size of the model to force it to maintain stylistic consistency. The cosine similarity function; and In this embodiment, a pre-trained image encoder and a text encoder are respectively used. Extracting image content using ResNet-50 Use BERT to extract text content; Describing the L2 norm, For image content, This refers to the text content.

[0040] The multi-scale consistency constraint loss The formula used to constrain the fidelity of the generated image in terms of shallow texture, mid-level structure, and deep semantics is as follows: ; Where K=3, This represents the feature mapping of the k-th layer of the feature extraction network. The weight coefficients at the k-th scale satisfy... =1, Describing the L1 norm, For image content, These are real images from a standard image dataset. Represents a real image Data distribution Find the expected value.

[0041] To ensure that the generated document illustrations have both clear edges (shallow features) and correct object structure (deep features), this embodiment defines K=3 to represent constraints at three scales: k=1 (shallow layer), corresponding to the ReLU1_2 layer of the VGG-19 network, focusing on color and texture; k=2 (middle layer), corresponding to the ReLU3_4 layer, focusing on geometry; and k=3 (deep layer), corresponding to the ReLU5_4 layer, focusing on semantic categories.

[0042] In this embodiment, the fourth intelligent agent for generating images and text is configured with an adaptive weight adjustment mechanism to dynamically balance the loss weights. The specific formula is as follows: ; in, and These are the weight coefficients for the semantic-style collaborative loss and the multi-scale consistency constraint loss, respectively.

[0043] The fifth agent for document generation connects to the fourth agent for image and text generation and the third agent for format constraints, and is used to receive... , and Based on a document layout optimization algorithm, the generated text and images are filled into a formatting frame, and an editable document is output.

[0044] Example 2 like Figure 3 As shown, this embodiment provides an end-to-end text and image generation method for multi-agent collaboration, applied to the system described in Embodiment 1. The method includes the following steps: S1. Obtain text content, style transfer template, and target specification template through the user interface.

[0045] S2. Parallel Feature Extraction: Extract semantic feature vectors from the text using the first agent of semantic features. The style feature vector of the style transfer template is extracted by the second intelligent agent through logical constraints. The third-party agent extracts the format feature vector of the target specification template through format constraints. ; Specifically, the logical constraint second agent extracts the style feature vector of the style transfer template. The specific steps are as follows: The style template collection is processed by grayscale conversion and normalization. Calculate the covariance matrix of the pixel matrix; Perform singular value decomposition on the covariance matrix; The top d eigenvectors with the largest eigenvalues ​​are selected as the principal component projection directions to obtain the dimensionality-reduced style feature vectors. .

[0046] S3, the fourth intelligent agent for image and text generation is based on and Perform dual-constraint decoding by alternately optimizing the generator and discriminator parameters until the semantic-style co-operation loss and multi-scale consistency constraint loss converge, and output the image content. With text content .

[0047] S4, Document generation, the fifth intelligent agent, based on... The limited layout structure will and Reorganize and format the document to generate the final document.

[0048] In this embodiment, the convergence criterion is: <0.01 and < scale , ( scale =0.05); in, Let be the semantic-style collaborative loss function value in the t-th iteration. Let be the semantic-style collaborative loss function value in the (t-100th)th iteration. For multi-scale consistency constraint loss, scale This is the convergence threshold for the multi-scale consistency constraint loss.

[0049] Example 3 This embodiment uses the generation of an "environmental protection report" as an example to specifically illustrate the system described in Embodiment 1 and the method described in Embodiment 2. The user uploads a text file "Report on Environmental Protection.txt" to the front-end interface, along with a style reference image "Minimalist Blue Tones.jpg" and a template file "Company Standard Report Template.docx".

[0050] The system background starts three parallel threads: Thread 1 calls the semantic feature first agent to parse the text; Thread 2 calls the logical constraint second agent to parse the image style; Thread 3 calls the format constraint third agent to parse the document layout, identifying the logo position, title area, and body text area. Subsequently, the image-text generation fourth agent loads pre-trained weights and... and The input generator, during the training phase (performed offline), continuously updates the network parameters through backpropagation until... The relative rate of change is less than 1%. During the inference phase (conducted online), the image content corresponding to the text paragraph is directly output. and style-polished text content The document generation process involves a fifth intelligent agent reading the template's XML description file and employing a greedy algorithm to extract the text content. Enter the text box and paste the image content. Enter image placeholders. If the generated content exceeds the page limit, the agent will automatically adjust the font size or crop the image. Finally, it will call the Office Open XML SDK to generate a .docx file for users to download.

[0051] Based on the disclosure and teachings of the foregoing specification, those skilled in the art can make changes and modifications to the above embodiments. Therefore, the present invention is not limited to the specific embodiments disclosed and described above, and some modifications and changes to the invention should also fall within the protection scope of the claims of the present invention. Furthermore, although some specific terms are used in this specification, these terms are only for convenience of explanation and do not constitute any limitation on the present invention.

Claims

1. A multi-agent collaborative end-to-end text and image generation system, characterized in that, include: User input module: Used to receive text content, style transfer templates, and target specification templates input by users; The first intelligent agent for semantic features: connected to the user input module, it receives the text content, performs syntactic analysis on the text using a semantic role labeling model and a conditional random field, and generates a high-dimensional semantic feature vector. ; The second intelligent agent, bound by logical constraints, is connected to the user input module and is used to receive the style transfer template. After centering and whitening the template image data, it extracts the first d principal components using principal component analysis to generate a style feature vector. ; The third intelligent agent for format constraint is connected to the user input module and is used to receive the target specification template, identify non-text element regions and layout rules in the template using edge detection and contour extraction algorithms, and generate a format feature vector. ; The fourth agent for image and text generation connects the first agent for semantic features and the second agent for logical constraints, respectively, and is used for dual-constraint decoding based on an improved conditional generative adversarial network to simultaneously generate and... Semantic consistency and possess Image content with stylistic characteristics With text content ; The fifth agent for document generation: connects the fourth agent for image and text generation and the third agent for format constraints, respectively, and is used to receive... , and Based on a document layout optimization algorithm, the generated text and images are filled into a formatting frame, and an editable document is output.

2. The multi-agent collaborative end-to-end text and image generation system according to claim 1, characterized in that, The fourth intelligent agent for image and text generation includes a generator network and a discriminator network; The generator network employs a dual-stream encoder-decoder structure, processing the high-dimensional semantic feature vectors separately. and style feature vector Encoding fusion is performed; the loss function of the fourth agent for image and text generation is... Including basic combat losses Semantic-style collaborative loss and multi-scale consistency constraint loss .

3. The multi-agent collaborative end-to-end text and image generation system according to claim 2, characterized in that, The semantic-style collaborative loss The formula used to constrain the similarity relationship of generated content in the feature space is as follows: ； in, As a cooperating factor, from the formula =0.8 Dynamic calculation; The cosine similarity function is used. and These are pre-trained image encoders and text encoders, respectively. Describing the L2 norm, For image content, This refers to the text content.

4. The multi-agent collaborative end-to-end text and image generation system according to claim 2, characterized in that, The multi-scale consistency constraint loss The formula used to constrain the fidelity of the generated image in terms of shallow texture, mid-level structure, and deep semantics is as follows: ； Where K=3, This represents the feature mapping of the k-th layer of the feature extraction network. The weight coefficients at the k-th scale satisfy... =1, Describing the L1 norm, For image content, These are real images from a standard image dataset. Represents a real image Data distribution Find the expected value.

5. The multi-agent collaborative end-to-end text and image generation system according to claim 2, characterized in that, The fourth intelligent agent for image and text generation is equipped with an adaptive weight adjustment mechanism to dynamically balance the loss weights. The specific formula is as follows: ； in, and These are the weight coefficients for the semantic-style collaborative loss and the multi-scale consistency constraint loss, respectively.

6. A multi-agent collaborative end-to-end text and image generation method, applied to the system described in claims 1-5, characterized in that, Includes the following steps: S1. Obtain text content, style transfer template, and target specification template through the user interface; S2. Parallel Feature Extraction: Extract semantic feature vectors from the text using the first agent of semantic features. The style feature vector of the style transfer template is extracted by the second intelligent agent through logical constraints. ; Extracting the format feature vector of the target specification template using a third-party intelligent agent based on format constraints. ; S3, the fourth intelligent agent for image and text generation is based on and Perform dual-constraint decoding by alternately optimizing the generator and discriminator parameters until the semantic-style co-operation loss and multi-scale consistency constraint loss converge, and output the image content. With text content ; S4, Document generation, the fifth intelligent agent, based on... The limited layout structure will and Reorganize and format the document to generate the final document.

7. The multi-agent collaborative end-to-end text and image generation method according to claim 6, characterized in that, In step S2, the logically constrained second agent extracts the style feature vector of the style transfer template. Specifically: The style template collection is processed by grayscale conversion and normalization. Calculate the covariance matrix of the pixel matrix; Perform singular value decomposition on the covariance matrix; The top d eigenvectors with the largest eigenvalues ​​are selected as the principal component projection directions to obtain the dimensionality-reduced style feature vectors. .

8. The multi-agent collaborative end-to-end text and image generation method according to claim 6, characterized in that, In step S3, the convergence criterion is as follows: <0.01 and < scale , ( scale =0.05); in, Let be the semantic-style collaborative loss function value in the t-th iteration. Let be the semantic-style collaborative loss function value in the (t-100th)th iteration. For multi-scale consistency constraint loss, scale This is the convergence threshold for the multi-scale consistency constraint loss.

Images