Image-text mixed layout generation method and system capable of highlighting key content
By integrating Wikipedia datasets and combining user intentions, using graphic and text generation models and graphic and text mixed layout generation models, the problem of difficult to effectively utilize historical business documents and lack of user intention considerations in the existing technology is solved, and the key content is highlighted The mixed layout of pictures and texts has been generated to improve the accuracy and visual effect of document generation.
Patent Information
- Application Number
- CN202411614150.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-13
- Publication Date
- 2025-02-18
AI Technical Summary
The prior art is difficult to effectively use historical business documents to assist in the generation of new business documents, and in the generation of non-standard graphic and text mixed documents, the lack of consideration of user intentions, resulting in the layout of the generated document difficult to meet the user's subjective requirements.
By integrating the Wikipedia data set, the entire structure information, graphic and text data pairs, keyword sentences and consensus information of the mixed graphic and text document are extracted, and the graphic and text generation model is input for encoding to generate graphic and text content encoding information. Combining user intention words, document keywords and text styles, coding and feature fusion, input a graphic and text mixed layout generation model, and generate a graphic and text mixed layout that highlights key content.
It realizes the generation of document layout based on user intentions, highlights the content of attention, improves the accuracy and visual effects of document generation, and meets the user's subjective requirements.
Smart Images
Figure CN119478137A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image and text generation, and more particularly to a method and system for generating an image and text mixed layout that highlights key contents. Background Art
[0002] In recent years, with the rapid development of artificial intelligence technology, text and image generation technology based on deep neural networks has made significant breakthroughs. In actual scenarios, image generation technology has been relatively maturely applied in some sub-fields. However, except for the wide application of text generation technology in traditional advantageous fields such as translation and speech recognition, it is difficult to achieve high accuracy in other aspects, especially when it comes to semantic generation applications. Business documents are different from other creative documents. They are not only standardized in format and high in content quality, but also have great relevance and similarity in content. Therefore, in the case of a large number of business documents accumulated, how to effectively use these historical business documents to assist in completing new business document creation tasks is the focus and key. Furthermore, although non-standard business documents do not have fixed format requirements, their layout also follows certain implicit rules such as line spacing, title size, image-text ratio, style and style according to the document type. How to learn such implicit rules through historical document data to achieve better visual effects is also the key to the success of business document generation. Finally, the generation of non-standard mixed text and graphics documents is essentially a complex system engineering project. The constraint information of the task is proposed by the user, which makes it difficult to have a solidified objective evaluation standard for problem solving. How to build a human-computer collaborative environment, give full play to people's experience, knowledge and creativity, and generate documents that meet subjective requirements is the key to ensuring that the system continues to be "easy to use".
[0003] For the automated mixed layout of graphic and text documents, traditional layout methods such as heuristic rule-based and template-based are commonly used, as well as deep learning layout methods that are gradually becoming a research hotspot.
[0004] While heuristic rule-based approaches can capture the rich raster properties of layout images, they do not model the layout structure, resulting in noisy proposals in contextual search on layout datasets. Early work on synthesizing 2D layouts relied on example- and rule-based heuristics that fail to capture complex element distributions.
[0005] Template-based methods, such as Merrell et al., apply some simple design guidelines to solve similar problems. These works usually provide an optimization framework as well as some design guidelines to synthesize and evaluate reasonable layouts. However, these works focus more on graphic elements (such as photos, furniture) and put visual aesthetics first.
[0006] Deep learning layout methods have gradually become the mainstream of research with the rise of artificial intelligence technology. They have not only achieved great results in 2D graphics layout, but also can perform 3D scene synthesis. Wang recently proposed a sequential decision method for indoor scene synthesis. In each step, a CNN is trained to predict the location or determine the category of an object by looking at the top-down view presented. This is similar to the wireframe rendering discriminator, in the sense that convolutions are used to capture the spatial patterns of the layout. SHED is another fine-grained substructure similarity measure for comparing two 3D shapes. These works provide valuable clues for developing effective structural measures for layout similarity.
[0007] Graph Neural Networks (GNNs) model node dependencies in graphs through message passing and are perfect tools for learning structured data. GNNs provide coarse graph embeddings, which, although useful for many tasks, may lose useful structural information in contextual searches if each graph is processed separately. Summary of the invention
[0008] In view of this, the present invention provides a method and system for generating a mixed layout of images and texts that highlights key content, which can generate a document layout according to user intentions and highlight the content of concern.
[0009] In order to achieve the above object, the present invention adopts the following technical solution:
[0010] In a first aspect, the present invention provides a method for generating a mixed layout of images and texts that highlights key content, comprising the following steps:
[0011] Based on the Wikipedia dataset, a mixed text-image document is integrated, and the entire structure information, text-image data pairs, key words and phrases, and consensus information are extracted from the mixed text-image document;
[0012] Inputting the entire structure information, image-text data pairs, key words and sentences, and consensus information of the image-text hybrid document into the image-text generation model, encoding the image and text respectively, and obtaining image-text content encoding information;
[0013] The keywords of the title and the keywords of the abstract in the mixed text-image document are respectively used as user intention words and document keywords. After encoding and feature fusion of the user intention words, document keywords and text style, they are input into the mixed text-image layout generation model together with the text-image content encoding information to generate a mixed text-image layout.
[0014] Furthermore, the generation process of the mixed text and image document includes:
[0015] Classify the Wikipedia dataset, pack similar images into image packages, and semantically concatenate the corresponding texts to form synthetic texts;
[0016] The image package and the synthetic text are aggregated through data collection to form an image and text data package;
[0017] According to the positional relationship between each image and text in the image and text data package, each image and text is placed in the corresponding position and integrated into a mixed image and text document.
[0018] Furthermore, the image-text generation model includes a global attention network, an image generator, a text generator and a discriminator;
[0019] During the training phase of the image-text generation model, the global attention network combines the entire structure information and consensus information of the image-text hybrid document to generate global attention information;
[0020] The image generator encodes the image data in combination with the global attention information, cross-attentively fuses the encoded image features with the text features encoded by the text generator, and then decodes them;
[0021] The text generator encodes the text data by combining the global attention information and the keywords, cross-attentively fuses the encoded text features with the image features encoded by the image generator, and then decodes them;
[0022] The discriminator measures whether the decoded image data and text data meet preset requirements.
[0023] Furthermore, the image generator includes: an image convolutional network, an image encoder, an image cross-attention network and an image decoder;
[0024] The image convolution network performs convolution operation on the image data to extract image features, and the extracted image features are fused with the global attention information and then input into the image encoder for encoding;
[0025] The image cross-attention module cross-attentionally fuses the encoded image features with the text features encoded by the text generator, and inputs the fused information into the image decoder for decoding.
[0026] Furthermore, the text generator includes: an attention network, a text convolutional network, a text encoder, a text cross attention network and a text decoder;
[0027] The attention network focuses on marking the keywords of the mixed text and image document;
[0028] The text convolution network performs convolution operation on the text data to extract text features, and the extracted text features, keyword focus identification information and global attention information are fused and input into the text encoder for encoding;
[0029] The text cross-attention network cross-attentionally fuses the encoded text features with the image features encoded by the image generator, and inputs the fused information into the text decoder for decoding.
[0030] Furthermore, the discriminator includes: an image discriminator, a text discriminator and a cross-modal discriminator;
[0031] The image discriminator determines whether the difference between the image data finally decoded by the image generator and the original image data features is within a preset range;
[0032] The text discriminator determines whether the difference between the text data finally decoded by the text generator and the original text data features is within a preset range;
[0033] The cross-modal discriminator determines whether the cross-attention fusion features output by the image generator match the cross-attention fusion features output by the text generator.
[0034] Furthermore, the image-text mixed layout generation model includes: an image-text generator, an image-text encoder and an image-text discriminator;
[0035] In the training phase of the image-text mixed layout generation model, the image-text generator generates an image-text mixed layout according to the fusion feature encoding information of the user's intended words, document keywords and text style and the image-text content encoding information;
[0036] The image-text encoder encodes the generated image-text mixed layout and the original image-text mixed layout respectively to obtain vectors Z' and Z respectively;
[0037] The image-text discriminator determines whether the difference between vectors Z' and Z is within a preset range.
[0038] In a second aspect, the present invention provides a system for generating a mixed layout of images and texts that highlights key contents, which is applicable to the method described above, and includes:
[0039] An input module, used to input a mixed text-and-picture document to be edited, and extract the entire structure information, text-and-picture data pairs, key words and phrases, and consensus information from the mixed text-and-picture document;
[0040] A picture-text generation module, which is used to use a picture-text generation model to combine the entire structure information, picture-text data pairs, key words and sentences, and consensus information of the picture-text hybrid document to encode the image and text respectively, and obtain picture-text content encoding information;
[0041] The graphic-text mixed layout module is used to use the keywords of the title and the keywords of the abstract in the graphic-text mixed document as user intention words and document keywords respectively, encode the user intention words, document keywords and text style and perform feature fusion, and then input them together with the graphic-text content encoding information into the graphic-text mixed layout generation model to generate a graphic-text mixed layout.
[0042] It can be seen from the above technical solutions that, compared with the prior art, the present invention has the following beneficial effects:
[0043] 1. Starting from the overall structure of the document, the present invention integrates the key information of the contextual environment information and the text content into the training of the image-text generation model, constructs a multimodal common feature space with additional contextual feature information, and expects to implicitly add cross-modal search conditions to improve search accuracy and add more search result information. Even if the retrieval condition data is far away from the condition data space used during training, it can prevent the retrieval results from deviating from the real results.
[0044] 2. In view of the fact that current document layout methods lack consideration of user intent, the present invention uses the article title and key content as prompt information of user intent, integrates it with the text style, and then uses the integrated information and the image-text content encoding information generated by the image-text generation model to jointly train and learn the image-text mixed layout generation model. The trained model can highlight the content of interest when generating the document layout according to the user's intent. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0046] Figure 1 A flow chart of a method for generating a mixed layout of images and texts that highlights key contents provided by the present invention;
[0047] Figure 2 A flowchart of preprocessing the Wikipedia dataset provided by the present invention;
[0048] Figure 3 A schematic diagram of the structure of the image-text generation model provided by the present invention;
[0049] Figure 4 A flowchart of preprocessing the arXiv corpus provided by the present invention;
[0050] Figure 5A schematic diagram of the structure of the mixed text and image layout generation model provided by the present invention. DETAILED DESCRIPTION
[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0052] like Figure 1 As shown, the embodiment of the present invention discloses a method for generating a mixed layout of images and texts that highlights key contents, comprising the following steps:
[0053] S1. Based on the Wikipedia dataset, we integrate text-image hybrid documents and extract the entire structure information, text-image data pairs, key words and sentences, and consensus information from the text-image hybrid documents.
[0054] S2, inputting the entire structure information, image-text data pairs, key words and sentences, and consensus information of the image-text hybrid document into the image-text generation model, encoding the image and text respectively, and obtaining the image-text content encoding information;
[0055] S3. The keywords of the title and the keywords of the abstract in the mixed text and image document are used as user intention words and document keywords respectively. After encoding and feature fusion of the user intention words, document keywords and text style, they are input into the mixed text and image layout generation model together with the encoding information of the text and image content to generate a mixed text and image layout.
[0056] The above steps are further explained below.
[0057] S1. Wikipedia dataset preprocessing.
[0058] The present invention trains the image-text generation model based on the Wikipedia data set. Since the data in the original Wikipedia data set are organized into 10 categories such as art, biology, geography, etc., and the data are "image-text" pairs, the data set needs to be preprocessed to simulate most cases of multiple images and multiple texts in real documents. At the same time, in order to simplify the processing, the default document has images and texts appearing in pairs. After integrating the Wikipedia data into multiple image-text data packets similar to image-text documents, the data needs to be split into image-text data pairs again, but in the splitting process, it is also necessary to extract the entire structure information of the document, key word sentence information, and consensus information that can analyze the expression of the image-text data in the document. At this point, all data set preprocessing tasks before training have been completed.
[0059] The specific data set preprocessing is as follows Figure 2 As shown, specifically including:
[0060] First, by default, there is a one-to-one matching relationship between images and texts. The Wikipedia dataset is classified into 10 categories, such as art, biology, and geography, to simulate the situation of multiple images and multiple texts in real documents. Then, image extraction is performed on the images to group similar images into a package. Similarly, semantic splicing is performed on the corresponding texts to form synthetic texts.
[0061] Finally, the image package and synthetic text are aggregated to form a picture and text data package.
[0062] Since we know the positional relationship between each image and text in the image and text data package (derived from the image and text pairing), we can put these images and text in corresponding positions as a mixed image and text document, and this mixed document can be regarded as a real document.
[0063] Then, through traditional image-text separation technology, we can obtain the image-text relative layout information and image-text data pairs of the entire document; by analyzing the text through semantic analysis, we can separate key words and phrases, as well as semantically similar sentences in different texts (if similar sentences appear at the same time in different text paragraphs, they are considered to need to be emphasized in common), that is, consensus information.
[0064] S2. Training of image-text generation model.
[0065] The image-text generation model of the present invention is designed based on the GAN concept, and the image-text generation model includes a global attention network, an image generator, a text generator and a discriminator;
[0066] In the training phase of the image-text generation model, the global attention network combines the entire structural information and consensus information of the image-text hybrid document to generate global attention information;
[0067] The image generator encodes the image data in combination with the global attention information, cross-attunes and fuses the encoded image features with the text features encoded by the text generator, and then decodes them.
[0068] The text generator encodes the text data by combining global attention information and keywords, cross-attentively fuses the encoded text features with the image features encoded by the image generator, and then decodes them;
[0069] The discriminator measures whether the decoded image data and text data meet the preset requirements.
[0070] Specifically, if Figure 3 As shown, the image generator includes: an image convolutional network, an image encoder, an image cross-attention network and an image decoder;
[0071] The image convolution network performs convolution operations on the image data to extract image features. The extracted image features are fused with the global attention information and then input into the image encoder for encoding.
[0072] The image cross-attention module cross-attentionally fuses the encoded image features with the text features encoded by the text encoder, and inputs the fused information into the image decoder for decoding.
[0073] The text generator includes: attention network, text convolutional network, text encoder, text cross attention network and text decoder;
[0074] The attention network focuses on marking keywords in mixed text and image documents;
[0075] The text convolution network performs convolution operations on text data to extract text features. The extracted text features, keyword key identification information and global attention information are fused and input into the text encoder for encoding;
[0076] The text cross-attention network cross-attentionally fuses the encoded text features with the image features encoded by the image encoder, and inputs the fused information into the text decoder for decoding.
[0077] Among them, the parameters of the image encoder and the text encoder are the same, so the features of the encoded image data and text data are represented in the same space.
[0078] The discriminators include: image discriminator, text discriminator and cross-modal discriminator;
[0079] The image discriminator determines whether the difference between the image data finally decoded by the image decoder and the original image data features (i.e., the image features captured by the image convolution network) is within a preset range;
[0080] The text discriminator determines whether the difference between the text data finally decoded by the text decoder and the original text data features is within a preset range. The original text data features here refer to the key words and sentences that are marked by the attention network and then fused with the global information, and then fused with the text feature data captured by the text convolution network;
[0081] The cross-modal discriminator determines whether the cross-attention fusion features output by the image generator match the cross-attention fusion features output by the text generator. After passing through the cross-modal discriminator, if the image and text match, the difference value output by the cross-modal discriminator is within a fixed range, otherwise it can be considered that the image and text do not match. The cross-attention fusion features output by the image cross-attention network use the image encoding as a reference to judge other inputs and their relevance, while the cross-attention fusion features output by the text cross-attention network are the opposite. The feature value output by the cross-attention network is actually a spatial vector. The difference value of the two vectors can be judged by calculating the Euler distance. If the Euler distance is less than a certain range, the text and image are considered to match.
[0082] In the training process of the image-text generation model, just like the traditional GAN network, the generator and discriminator are fixed separately, and only one generator and discriminator are trained at the same time. When the cross-modal discriminator, image discriminator, and text discriminator objective functions are all within the preset reasonable range, the training can be considered complete. After training, the image-text generation model only needs to use the generator, and the final output of the decoder is a one-hot encoding or hash value.
[0083] S3. Training of the image-text mixed layout generation model.
[0084] like Figure 4 As shown in the figure, the mixed layout of text and images is studied and tested based on the arXiv corpus. The arXiv corpus includes not only the original arXiv paper documents, but also its supporting paper id, submitter, author, paper title, annotation, doi, abstract, category, version and other metadata information. Most of the business document data only have general classification, author, title and key information summary, so the original arXiv data can be extracted by category, author, and graded, and the original PDF can be cut out of the single page information, and the single page PDF is graphically processed, and the title, image, text paragraph, bold and other formats are marked with different colors, and finally the layout data used by the mixed layout model of text and images is formed.
[0085] like Figure 5As shown, the mixed text-image layout generation model of the present invention is also constructed based on the GAN concept, and the mixed text-image layout generation model includes: a text-image generator, a text-image encoder, and a text-image discriminator. In order to enable the generated document layout to capture the user's intention and highlight the key content, at the input end of the text-image generator, it is necessary not only to input the information encoded in the text-image content of the original mixed text-image document (i.e., the output of the encoder in the text-image generation model), but also to input the intention words extracted from the title, the keywords extracted from the summary information, and the document layout style and other user-required information after fusion. Feature information; when the document layout image is generated, it is encoded simultaneously with the original layout image and input into the discriminator for true or false judgment. The generator can be constructed using network structures such as CNN and attention mechanism, and the discriminator is mainly used to calculate the KL divergence or Wasserstein distance of the distribution of two document layout images.
[0086] Specifically:
[0087] The original mixed text and image document (PDF document) is encoded by the encoder and input into the image and text generator. At the same time, the keywords in the title and the keywords in the abstract of the document are used as user intention words and document keywords. Combined with the document style (layout style, such as formal, compact, etc.), they are input into the intention encoder, keyword encoder and user requirement encoder for encoding. After encoding, feature fusion (realized by convolutional network) is performed to achieve unification and normalization, and then input into the image and text generator. The image and text generator generates the mixed text and image layout y' based on the fusion feature encoding information of user intention words, document keywords and text style, as well as the image and text content encoding information; the user intention words actually determine the part that needs to be highlighted (after training, it can intelligently associate related text or pictures), the document keywords determine the generated image and text content, and the text style determines the typesetting style.
[0088] The image-text encoder encodes the generated mixed image-text layout y' and the original mixed image-text layout y respectively, and obtains vectors Z' and Z respectively; the original mixed image-text document is taken as an image, and the layout position information of the text and the picture is obtained by using the traditional image segmentation method, and then the original mixed image-text layout y is obtained.
[0089] The image-text discriminator determines whether the difference between vectors Z' and Z is within a preset range. If the difference between the two vectors is within a reasonable range, the generated layout y' is considered to be consistent with the original layout y. After multiple trainings, if the true judgment rate of the generated layout y' is stable at more than 92%, it is considered that the image-text mixed layout generation model training is successful. In actual use, there is no need to enter the original PDF document information, only the intent words, keywords and document style.
[0090] In general, the arXiv dataset is used to train mixed text and image layout models; while the Wikipedia dataset is used to train text and image generation models that generate the required images and text data; because the arXiv dataset is more scientific research-oriented and has insufficient corpus, its typesetting styles are diverse and can reflect the relationship between layout and user intent (keywords, etc.); while the Wikipedia data corpus is rich, but layout information cannot be extracted (because it is not a standard document), so it can be used for text and image generation; the graphical processing of the arXiv dataset is actually to mark the text part with one color and the icon with another color, which is a layout data. In the eyes of the mixed text and image layout model, it is just a picture. Under the training of these layout data and the text and image corpus provided by the mixed text and image documents, the mixed text and image layout generation model can intelligently associate related text or pictures and highlight the key parts.
[0091] In other embodiments, the present invention further provides a system for generating a mixed layout of images and texts that highlights key content, which is applicable to the above method and includes:
[0092] An input module is used to input a mixed text-image document to be edited, and extract the entire structure information, text-image data pairs, key words and phrases, and consensus information from the mixed text-image document;
[0093] The image-text generation module is used to use the image-text generation model to combine the entire structure information, image-text data pairs, key words and sentences, and consensus information of the image-text hybrid document to encode the image and text respectively, and obtain the image-text content encoding information;
[0094] The mixed text and image layout module is used to use the keywords of the title and the keywords of the abstract in the mixed text and image document as user intention words and document keywords respectively, encode the user intention words, document keywords and text style and fuse their features, and then input them into the mixed text and image layout generation model together with the encoding information of the text and image content to generate a mixed text and image layout.
[0095] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0096] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for generating a mixed layout of images and texts that highlights key content, characterized in that: The following steps are involved: Based on the Wikipedia dataset, a mixed text-image document is integrated, and the entire structure information, text-image data pairs, key words and phrases, and consensus information are extracted from the mixed text-image document; Inputting the entire structure information, image-text data pairs, key words and sentences, and consensus information of the image-text hybrid document into the image-text generation model, encoding the image and text respectively, and obtaining image-text content encoding information; The keywords of the title and the keywords of the abstract in the mixed text-image document are respectively used as user intention words and document keywords. After encoding and feature fusion of the user intention words, document keywords and text style, they are input into the mixed text-image layout generation model together with the text-image content encoding information to generate a mixed text-image layout.
2. The method for generating a mixed layout of images and texts that highlights key contents according to claim 1, characterized in that: The generation process of the mixed text and image document includes: Classify the Wikipedia dataset, pack similar images into image packages, and semantically concatenate the corresponding texts to form synthetic texts; The image package and the synthetic text are aggregated through data collection to form an image and text data package; According to the positional relationship between each image and text in the image and text data package, each image and text is placed in the corresponding position and integrated into a mixed image and text document.
3. The method for generating a mixed layout of images and texts that highlights key contents according to claim 1, characterized in that: The image-text generation model includes a global attention network, an image generator, a text generator and a discriminator; In the training phase of the image-text generation model, the global attention network combines the entire structure information and consensus information of the image-text hybrid document to generate global attention information; The image generator encodes the image data in combination with the global attention information, cross-attentively fuses the encoded image features with the text features encoded by the text generator, and then decodes them; The text generator encodes the text data by combining the global attention information and the keywords, cross-attentively fuses the encoded text features with the image features encoded by the image generator, and then decodes them; The discriminator measures whether the decoded image data and text data meet preset requirements.
4. The method for generating a mixed layout of images and texts that highlights key contents according to claim 3 is characterized in that: The image generator includes: an image convolution network, an image encoder, an image cross attention network and an image decoder; The image convolution network performs convolution operation on the image data to extract image features, and the extracted image features are fused with the global attention information and then input into the image encoder for encoding; The image cross-attention module cross-attentionally fuses the encoded image features with the text features encoded by the text generator, and inputs the fused information into the image decoder for decoding.
5. The method for generating a mixed layout of images and texts that highlights key contents according to claim 3 is characterized in that: The text generator includes: an attention network, a text convolutional network, a text encoder, a text cross attention network and a text decoder; The attention network focuses on marking the keywords of the mixed text and image document; The text convolution network performs convolution operation on the text data to extract text features, and the extracted text features, keyword focus identification information and global attention information are fused and input into the text encoder for encoding; The text cross-attention network cross-attentionally fuses the encoded text features with the image features encoded by the image generator, and inputs the fused information into the text decoder for decoding.
6. The method for generating a mixed layout of images and texts that highlights key contents according to claim 3, characterized in that: The discriminator includes: an image discriminator, a text discriminator and a cross-modal discriminator; The image discriminator determines whether the difference between the image data finally decoded by the image generator and the original image data features is within a preset range; The text discriminator determines whether the difference between the text data finally decoded by the text generator and the original text data features is within a preset range; The cross-modal discriminator determines whether the cross-attention fusion features output by the image generator match the cross-attention fusion features output by the text generator.
7. The method for generating a mixed layout of images and texts that highlights key contents according to claim 1, characterized in that: The image-text mixed layout generation model includes: an image-text generator, an image-text encoder and an image-text discriminator; In the training phase of the image-text mixed layout generation model, the image-text generator generates an image-text mixed layout according to the fusion feature encoding information of the user's intended words, document keywords and text style and the image-text content encoding information; The image-text encoder encodes the generated image-text mixed layout and the original image-text mixed layout respectively to obtain vectors Z' and Z respectively; The image-text discriminator determines whether the difference between vectors Z' and Z is within a preset range.
8. A system for generating a mixed layout of images and texts that highlights key content, characterized in that: It is applicable to the method according to any one of claims 1 to 7, comprising: An input module, used to input a mixed text-and-picture document to be edited, and extract the entire structure information, text-and-picture data pairs, key words and phrases, and consensus information from the mixed text-and-picture document; A picture-text generation module, which is used to use a picture-text generation model to combine the entire structure information, picture-text data pairs, key words and sentences, and consensus information of the picture-text hybrid document to encode the image and text respectively, and obtain picture-text content encoding information; The graphic-text mixed layout module is used to use the keywords of the title and the keywords of the abstract in the graphic-text mixed document as user intention words and document keywords respectively, encode the user intention words, document keywords and text style and perform feature fusion, and then input them together with the graphic-text content encoding information into the graphic-text mixed layout generation model to generate a graphic-text mixed layout.