Document generation method and device based on multi-modal large model, equipment and medium

Abstract

The invention provides a document generation method and device based on a multi-modal large model, equipment and a medium, and relates to the field of data processing. According to the method, the multi-modal large model and the text analyzer are combined, automatic outline extraction and structured processing of the input text are achieved, then the document generation model intelligently matches the structured text and the attributes of the structured text to the most suitable demonstration document template through the adaptive element matching technology, and the document generation efficiency is improved. The time for manually adjusting the template and the content is shortened, and the diversity of the template of the demonstration document is increased, so that the generation efficiency and the result quality of the demonstration document are remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a document generation method, device, equipment and medium based on a multimodal large model. Background Art

[0002] In the financial field, PPTs for marketing scenarios are generated based on the PPT template structure method. Users structure the PPT copy content, fill in the corresponding content in specific fields such as main title, subtitle, content, data, etc., and then match the corresponding PPT template according to the name and number of fields.

[0003] Although the existing PPT generation method saves users most of the process of making PPTs, the content of the PPT needs to be manually input; and since the number of templates is limited, while the forms of PPT content are diverse, and also limited by the number of words and the form of the PPT content, it is easy to cause the PPT template and PPT content to not match, and the PPT template or PPT content needs to be manually adjusted, resulting in low PPT generation efficiency.

[0004] Therefore, how to improve the generation efficiency of presentation documents has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides a document generation method, apparatus, device and medium based on a multimodal large model, aiming to improve the generation efficiency of presentation documents.

[0006] In a first aspect, the present application provides a document generation method based on a multimodal large model, the document generation method based on a multimodal large model comprising the following steps:

[0007] Based on the multimodal large model, the outline content of the input text is extracted, and the extracted outline content is structured to output structured text;

[0008] Based on a text parser, the structured text is parsed to obtain text attributes corresponding to the structured text;

[0009] Based on the document generation model, adaptive element matching is performed on the structured text and the text attributes corresponding to the structured text to generate a target document.

[0010] In a second aspect, the present application further provides a document generation device based on a multimodal large model, wherein the document generation device based on the multimodal large model comprises:

[0011] The structured text output module extracts the outline content of the input text based on the multimodal large model, structures the extracted outline content, and outputs the structured text;

[0012] A text attribute parsing module, used to perform content parsing on the structured text based on a text parser to obtain text attributes corresponding to the structured text;

[0013] The presentation generation module is used to perform adaptive element matching on the structured text and the text attributes corresponding to the structured text based on the document generation model to generate a target document.

[0014] In a third aspect, the present application also provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the document generation method based on the multimodal large model as described above are implemented.

[0015] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the document generation method based on the multimodal large model as described above are implemented.

[0016] The present application provides a document generation method, device, computer equipment and storage medium based on a multimodal large model. The method of the present application includes extracting outline content from an input text based on a multimodal large model, and structuring the extracted outline content to output a structured text; based on a text parser, performing content parsing on the structured text to obtain text attributes corresponding to the structured text; based on a document generation model, performing adaptive element matching on the structured text and the text attributes corresponding to the structured text to generate a target document. Through the above-mentioned method, the present application realizes automated outline extraction and structuring of input text by combining a multimodal large model and a text parser, and then the document generation model intelligently matches the structured text and its attributes to the most suitable presentation document template through adaptive element matching technology, which not only reduces the time of manual adjustment of templates and content, but also increases the diversity of presentation document templates, thereby significantly improving the generation efficiency and quality of presentation documents. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A flowchart of a first embodiment of a document generation method based on a multimodal large model provided by the present application;

[0019] Figure 2 A flowchart of a second embodiment of a document generation method based on a multimodal large model provided by the present application;

[0020] Figure 3 A flowchart of a third embodiment of a document generation method based on a multimodal large model provided by the present application;

[0021] Figure 4 It is a structural schematic diagram of a first embodiment of a document generation device based on a multimodal large model provided by the present application;

[0022] Figure 5 It is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0023] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0025] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0026] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0027] Please refer to Figure 1 , Figure 1 A flowchart of a first embodiment of a document generation method based on a multimodal large model provided in this application.

[0028] like Figure 1 As shown, the document generation method based on the multimodal large model includes steps S101 to S103.

[0029] S101, extracting outline content from input text based on the multimodal large model, and structuring the extracted outline content to output structured text;

[0030] In one embodiment, the multimodal large model VL may include an image classification model and a language model. Through the multimodal large model, the outline content in the input text is extracted, and the extracted outline content is structured according to preset data structuring rules to output structured text.

[0031] Furthermore, the input text is obtained, wherein the input text includes a picture text and a first text; based on a text recognition OCR model, content recognition and extraction are performed on the picture text to generate a second text text; based on the multimodal large model and a preset structured data format, key element recognition and content matching are performed on the first text text and the second text text to generate the structured text.

[0032] In one embodiment, the structured data format may be a json format.

[0033] In one embodiment, the input text includes one or more of picture text, text text and prompt text.

[0034] Generally, a prompt is a piece of text or a sentence that instructs a machine learning model to generate output of a specific type, theme, or format. In the field of natural language processing, a prompt usually consists of a question or task description, such as "extract PPT-related content from the input text", "translate this English sentence into Chinese", etc. In the field of image recognition, a prompt can be a picture description, label, or classification information.

[0035] Preferably, the input text may be prompt text, or a combination of prompt text and other text.

[0036] In one embodiment, in a text generation task, the prompt can be regarded as a description or topic, such as "generate a PPT content about pension instructions", etc. Given such a prompt, the machine learning model can generate text that meets the requirements and ensure that the output result is related to the prompt.

[0037] In one embodiment, in the image generation task, the prompt can be regarded as an image description, label or classification information, such as the task description of "generate a PPT cover image" or "generate a PPT subject directory image". By giving such a prompt, the machine learning model can generate an image that meets the requirements and ensure that the output result is related to the prompt.

[0038] In one embodiment, the multimodal large model VL and the OCR model are used together to realize the generation or extraction of PPT content. At the same time, the generated or extracted PPT content is structured and generated into a json format through prompt text, eliminating the process of manually writing text content and filling the content into corresponding fields.

[0039] Among them, OCR (Optical Character Recognition) is a technology that uses automatic data extraction to quickly convert text images into machine-readable formats. The OCR model can recognize the text in the image and return the content in text form.

[0040] Among them, the JSON format is a lightweight text data exchange format that can be used to store and exchange text information.

[0041] In one embodiment, the input text provided by the user is obtained. If there is image text in the input text, the source data is input into the OCR model to extract the text in the image, which is recorded as Word_ocr; if the input is document text, the text in the document is directly read, which is recorded as Word_doc. Word_ocr and Word_doc together constitute the input of the multimodal large model VL.

[0042] For example, according to the [Word_ocr] content, the content outline suitable for PPT is listed in json format {attribute: content}. For example: {name: pension description, title: [{main title: how to manage your own pension}, {subtitle: personal pension part, bold: personal}, {subtitle: enterprise pension part, bold: enterprise}], time: July 2024, speaker: Zhang San}.

[0043] For example, according to the content of [Word_doc], the content outline suitable for PPT is listed in json format {attribute: content}. For example: {name: pension description, title: [{main title: how to manage your own pension}, {subtitle: personal pension part, bold: personal}, {subtitle: enterprise pension part, bold: enterprise}], time: July 2024, speaker: Zhang San}.

[0044] For example, based on a [topic] of pure prompt text, a content outline suitable for building a PPT is listed in json format {attribute: content}, for example: {name: Pension description, title: [{main title: How to manage your own pension}, {subtitle: Personal pension part, bold: Personal}, {subtitle: Enterprise pension part, bold: Enterprise}], time: July 2024, speaker: Zhang San}.

[0045] In-transit accident library payment, the multimodal large model VL outputs the specified content and a PPT outline in json format, which can be recorded as PPT.js.

[0046] S102: Based on a text parser, the structured text is parsed to obtain text attributes corresponding to the structured text;

[0047] In one embodiment, the text parser may be a JSON reader, or other algorithms that can be used to read and parse JSON format text.

[0048] In one embodiment, a json reader reads the PPT outline content, and counts the attributes and the number of attributes of the outline content, as well as the number of words in the attributes corresponding to the content. The output is also in a json format, recorded as statistics.js.

[0049] In one embodiment, the text parser traverses the structured text to identify text attributes related to PPT construction in the structured text, which may include but are not limited to text hierarchy (such as title, subtitle, content, etc.), number of words in the text, frame size, position, font, etc. of each text.

[0050] S103: Based on the document generation model, adaptive element matching is performed on the structured text and the text attributes corresponding to the structured text to generate a target document.

[0051] In one embodiment, the target document may be in the form of a text document, an image document, or a document combining text and graphics, such as a word document, a PPT presentation, and the like.

[0052] In one embodiment, the statistical outline content and the text attributes corresponding to the outline content can be input into the PptSD model in the form of prompt, that is, [prompt: Generate PPT in units of [title] according to the [PPT.js] outline content and the statistical information [statistics.js] corresponding to the outline content, and keep the style consistent. ], where [title] can be expressed in other units.

[0053] In one embodiment, the PptSD model generates a PPT template image based on the input structured text, text attributes and prompt, and then fills the outline content into the PPT template image, and according to the text attributes, performs adaptive element matching adjustment on the filled content of the PPT template image, such as font size, color, layout and format.

[0054] This embodiment provides a document generation method based on a multimodal big model. The method of this application realizes automatic outline extraction and structural processing of input text by combining the multimodal big model and the text parser. Then, the document generation model intelligently matches the structured text and its attributes to the most suitable presentation document template through adaptive element matching technology, which not only reduces the time of manual adjustment of templates and content, but also increases the diversity of presentation document templates, thereby significantly improving the generation efficiency and quality of presentation documents.

[0055] Please refer to Figure 2 , Figure 2 A flowchart of a second embodiment of a document generation method based on a multimodal large model provided in this application.

[0056] In this embodiment, Figure 2 As shown, based on the above Figure 1 In the illustrated embodiment, before step S101, the following steps are further included:

[0057] S201, performing image single modality training on the image classification model based on the first data set until the loss value of the image classification model converges;

[0058] In one embodiment, a picture-text pairing dataset may be prepared as model training data, and the picture-text pairing dataset may include but is not limited to a general picture-text pairing dataset (accounting for 90%) and a marketing picture-text pairing dataset (accounting for 10%).

[0059] Furthermore, a first data set is obtained, wherein the first data set includes image-text pairing data; based on a preset image processing process, data enhancement processing is performed on at least one group of image data collected from the first data set to obtain a third data set; based on the first data set and the third data set, image single-modal iterative training is performed on the image classification model until the loss value of the image classification model converges.

[0060] In one embodiment, a number of pictures (eg, 32 pictures) may be randomly extracted from the picture-text pairing data set as a training batch to construct a first data set, and the pictures used after extraction will not be put back into the training data set.

[0061] In one embodiment, the images in a batch are randomly enhanced by using a random method such as rotation, flipping, cropping, adding Gaussian noise, etc. to obtain enhanced images as the third data set.

[0062] In one embodiment, the enhanced image (i.e., the third data set) and the original image (i.e., the first data set) are first patchified, that is, the image is cut into small images of equal size and without overlapping pixels. Then, the enhanced image and the original image cut into small images are respectively input into a ViT model, and ViT outputs the feature vectors of the two images.

[0063] Among them, the patchify function divides the image into multiple overlapping patches with the specified patch size and step size.

[0064] Among them, the ViT (Vision Transformer) model is a deep learning model based on the Transformer architecture for image recognition tasks. ViT divides the image into a series of small blocks (called patches), and regards these image patches as elements in a sequence. These sequences are processed through the Transformer model to capture the global dependencies between image patches.

[0065] In one embodiment, the InfoNCE (contrastive learning) loss function of the feature vectors of both the enhanced image and the original image is calculated, and then the gradient of the loss is back-propagated to train the ViT model until the loss value of the InfoNCE loss function converges.

[0066] In one embodiment, the InfoNCE (Noise Contrastive Estimation) loss function is a loss function for self-supervised learning, which is usually used for learning feature representation or representation learning. It is based on the idea of ​​information theory and learns model parameters by comparing the similarity of positive samples and negative samples. By maximizing the similarity of similar sample pairs and minimizing the similarity of dissimilar sample pairs, meaningful representations are learned.

[0067] S202, performing image-text multimodal training on the picture classification model and the pre-trained first language model based on the second data set until the loss values ​​of the picture classification model and the first language model converge;

[0068] In one embodiment, the trained ViT model and the pre-trained ChatGLM language model are used for image-text multimodal training.

[0069] Among them, ChatGLM (General Language Model) is a downstream dialogue application based on the GLM large model, an open source dialogue language model that supports both Chinese and English.

[0070] Furthermore, based on the image classification model, feature extraction is performed on the image data in the second data set to obtain a first feature vector; based on the first language model, feature extraction is performed on the text data in the second data set to obtain a second feature vector; based on the first feature vector and the second feature vector, a contrast loss function of the image classification model and the first language model is calculated; based on the contrast loss function, model parameters of the image classification model and the first language model are iteratively optimized until the loss values ​​of the image classification model and the first language model converge.

[0071] In one embodiment, 32 pairs of data are randomly extracted from the picture-text pairing dataset, including picture data and text data.

[0072] In one embodiment, the image data is input into the ViT model, and the text data is input into the ChatGLM language model. The first feature vector in the image data is extracted by the ViT model, and the second feature vector in the text data is extracted by the ChatGLM language model.

[0073] In one embodiment, the InfoNCE loss function is calculated based on the first eigenvector and the second eigenvector, and gradient backpropagation is performed to train model parameters of the ViT model and the ChatGLM language model until the loss value of the InfoNCE loss function converges.

[0074] In one embodiment, backpropagation of loss gradients is a core algorithm for training neural networks in deep learning. It calculates the gradient of the loss function with respect to the network parameters and uses these gradients to update the network weights to minimize the loss function.

[0075] Exemplarily, gradient backpropagation training generally includes but is not limited to forward propagation, loss calculation, backpropagation, parameter update, and iterative training. Generally, the input data is calculated through each layer of the network until the last layer (output layer). At each layer, the input data is linearly transformed by the weight matrix and bias vector, and then usually processed by a nonlinear activation function. According to the output of the network and the true label, the loss function is calculated, which is an indicator of the difference between the model prediction and the actual result. Starting from the output layer, the gradient of the loss function relative to the network parameters (weights and biases) is calculated. Then, these gradients are propagated backward layer by layer using the chain rule until the input layer is reached. Use the calculated gradients and the pre-set learning rate to update the weights and biases of the network. Repeat the above steps until the loss value of the loss function converges.

[0076] S203, training the multilayer perceptron and the second language model based on the third data set until the loss values ​​of the multilayer perceptron and the second language model converge;

[0077] In one embodiment, the model parameters in the ViT model and the ChatGLM model are frozen, and a MLP (Multilayer Perceptron) layer with a two-layer fully connected layer FC structure is added after each of the two models.

[0078] Among them, MLP is a specific type of artificial neural network, which consists of multiple neurons and usually includes an input layer, one or more hidden layers, and an output layer.

[0079] In one embodiment, through the above-mentioned model training process, the feature representation spaces of the ViT model and the ChatGLM model are uniformly mapped to the same representation space through InfoNCE of contrastive learning. Therefore, the feature output of ViT can be used as the input of ChatGLM, and the ViT and ChatGLM models can be connected in series to form a new computing link. The ChatGLM output generates a vector, and the output generation vector is input into the MLP layer connected after the ChatGLM model, and finally the feature vector V_G is output.

[0080] In one embodiment, the output vector of the ViT model is not only used as the input of the ChatGLM model, but also input into the MLP layer connected after the ViT model, and finally outputs the feature vector V_T.

[0081] In one embodiment, 32 pairs of image-text pairing data are randomly sampled from a picture-text pairing dataset, the picture data are input into a ViT model, and after calculation by the ViT model and the ChatGLM model, the feature vector V_T output by the ViT model and the feature vector V_G output by the ChatGLM model are concatenated, and the concatenated vectors are input into a ChatGLM2 model, and the ChatGLM2 model outputs text, and the cross entropy loss is calculated for this output text and the text data in the previously sampled image-text pairing dataset, and the parameters of the ChatGLM2 model and the two MLP layers are trained by the gradient of the cross entropy loss until the loss value converges.

[0082] Among them, cross entropy loss is an indicator to measure the difference between two probability distributions. In classification problems, there is usually a true probability distribution P (usually a one-hot encoding vector, representing the true label distribution of the sample) and a probability distribution Q predicted by the model. Cross entropy loss is used to measure the difference between these two probability distributions.

[0083] Generally, cross entropy loss measures the error between the true distribution and the model's predicted distribution. When the model's prediction is consistent with the actual situation, the cross entropy loss will approach 0. In other words, the optimization goal of the cross entropy loss function is to make the model's predicted probability distribution as close as possible to the probability distribution of the true label to minimize the error.

[0084] S204: construct the multimodal large model based on the trained image classification model, the first language model, the multilayer perceptron and the second language model.

[0085] In one embodiment, each component model in the multimodal large model is trained separately, and finally the multimodal large model VL is constructed with the trained component models.

[0086] In one embodiment, when applying the multimodal large model VL, if there is image text in the input text, the image text is input into the ViT model, and the image text is input into the OCR model to obtain the text data in the image. The text and prompt text in the input text are always input into the ChatGLM model. If there is no image text in the input text, the input of the ViT model is empty, and the ChatGLM model can be used directly.

[0087] This application provides a two-stage marketing PPT generation method, which utilizes a multimodal large model and a language large model, as well as a diffusion model. Based on the contrastive learning framework, the features of the ViT model and the text ChatGLM model can be mapped to the same representation space, which is beneficial for different input types during use, and can contain pictures or plain text. Then, the DiT model is used to generate the corresponding PPT template, and the copy in json format is attached to the template to generate the final PPT, which greatly saves the PPT production time. The generative model can also make the PPT style template diverse.

[0088] Please refer to Figure 3 , Figure 3 A flowchart of a third embodiment of a document generation method based on a multimodal large model provided in the present application.

[0089] In this embodiment, Figure 3 As shown, based on the above Figure 1 In the illustrated embodiment, before step S101, the following steps are further included:

[0090] S301, encoding a preset image text based on an image encoder to generate a sparse spatial image;

[0091] In one embodiment, the image encoder may be a VAE (Variational Autoencoder).

[0092] In one embodiment, VAE is a generative model that aims to learn the latent representation of input data and generate new samples from it. It combines the ideas of autoencoders and probabilistic coding, and achieves a more flexible and controllable sample generation process by modeling the latent representation.

[0093] Generally, VAE assumes that the potential representation obeys a prior distribution (usually a Gaussian distribution) and maps the input data to the distribution parameters of the latent space through the encoder. The decoder samples from the latent space and maps the sampled results back to the original input space to generate new samples. By learning the parameters of the latent distribution, VAE can learn the main features of the input data during training and generate new samples similar to the input data.

[0094] In one embodiment, after the PPT image is encoded by VAE, a sparse spatial representation z is obtained. Assuming that the PPT image is 256×256×3, where 3 is the number of RGB color channels and 256*256 is the PPT size, the shape of z can be obtained to be 32×32×4.

[0095] S302: Based on the sparse spatial image and the preset prompt text, a preset diffusion model is trained to obtain the document generation model.

[0096] Optionally, the document generation model (PptSD model) can use a preset diffusion model DiT (Diffusion Transformer) model framework.

[0097] In one embodiment, the DiT model is a diffusion model combined with a Transformer architecture for image and video generation tasks, which can efficiently capture dependencies in data and generate high-quality results.

[0098] In one embodiment, during training, the input of the DiT model is a sparse spatial representation z after passing through a VAE (variational autoencoder). The first layer of DiT is "patchify", which converts the spatial input into a series of T tokens, each of dimension d, by linearly embedding each patch into the input.

[0099] Among them, patchify is a concise and efficient library that provides two core functions: patchify function and unpatchify function. This library can divide 2D or 3D images of any size into overlapping small blocks (called patches), and can stitch these patches back to the original image to achieve perfect restoration.

[0100] The patchify function divides the image into multiple overlapping patches with the specified patch size and step size, while the unpatchify function is responsible for recombining these patches to restore the original image structure.

[0101] In one embodiment, after patchify, standard ViT frequency-based positional embedding (sine-cosine version) is applied to all input tokens.

[0102] Generally, the number of tokens T created by patchify is determined by the patch size hyperparameter p. In this embodiment, p is set to 16.

[0103] In one embodiment, after patching, the input token is processed by a series of N Transformer blocks. In addition to the noise image input, the diffusion model also processes other conditional information, namely the prompt text (for example: [prompt: Generate ppt in units of [title] based on the content of [ppt.js] and the corresponding statistical information of the content [statistics.js], and keep the style consistent.]).

[0104] In one embodiment, the prompt text vector emb (obtained through ChatGLM) is used as an additional token in the input sequence and is input into the DiT Block together with the input image token.

[0105] In one embodiment, a PPT template is generated according to the input prompt text. At this time, the PPT template will adaptively generate corresponding PPT elements, such as text boxes, picture boxes, and font sizes, according to the statistical information and content information contained in the input prompt text.

[0106] In one embodiment, when the document generation model is used, different forms can be input, such as picture + prompt, document + prompt and pure prompt. The json format of the PPT outline is generated by the first stage, and the output of the first stage is used as the input of the second stage DiT model to output the PPT template. Since this template is generated based on json, it is only necessary to directly fill the content of the outline json into the PPT template.

[0107] Generally, when using prompts for training, it is not necessary to label each sample, but to guide the model to generate the corresponding output through the given prompt. This method can greatly reduce the workload of data labeling and better control the quality and diversity of the model generation results.

[0108] See also Figure 4 , Figure 4It is a structural schematic diagram of a first embodiment of a document generation device based on a multimodal large model provided in the present application. The document generation device based on a multimodal large model is used to execute the aforementioned document generation method based on a multimodal large model.

[0109] like Figure 4 As shown, the document generation device 400 based on the multimodal large model includes: a structured text output module 401, a text attribute parsing module 402 and a presentation generation module 403.

[0110] The structured text output module 401 extracts the outline content of the input text based on the multimodal large model, structures the extracted outline content, and outputs the structured text;

[0111] The text attribute parsing module 402 is used to perform content parsing on the structured text based on a text parser to obtain text attributes corresponding to the structured text;

[0112] The presentation generation module 403 is used to perform adaptive element matching on the structured text and the text attributes corresponding to the structured text based on the document generation model to generate a target document.

[0113] In one embodiment, the structured text output module 401 includes:

[0114] An input text acquisition unit, used to acquire the input text, wherein the input text includes a picture text and a first text;

[0115] A picture text recognition unit, used to perform content recognition and extraction on the picture text based on a text recognition OCR model to generate a second text;

[0116] A structured text generation unit is used to perform key element recognition and content matching on the first text and the second text based on the multimodal large model and a preset structured data format to generate the structured text.

[0117] In one embodiment, the document generation device 400 based on the multimodal large model further includes a first model training module, including:

[0118] A unimodal training unit, configured to perform unimodal image training on the image classification model based on the first data set until a loss value of the image classification model converges;

[0119] A multimodal training unit, configured to perform image-text multimodal training on the image classification model and the pre-trained first language model based on a second data set until loss values ​​of the image classification model and the first language model converge;

[0120] An output model training unit, used to train the multilayer perceptron and the second language model based on the third data set until the loss values ​​of the multilayer perceptron and the second language model converge;

[0121] A multimodal large model construction unit is used to construct the multimodal large model based on the trained image classification model, the first language model, the multilayer perceptron and the second language model.

[0122] In one embodiment, the single-modality training unit comprises:

[0123] A first data set acquisition subunit, configured to acquire a first data set, wherein the first data set includes picture-text pairing data;

[0124] An image processing subunit, configured to perform data enhancement processing on at least one set of image data collected from the first data set based on a preset image processing process to obtain a third data set;

[0125] The model unimodal training subunit is used to perform image unimodal iterative training on the image classification model based on the first data set and the third data set until the loss value of the image classification model converges.

[0126] In one embodiment, the multimodal training unit includes:

[0127] A first feature extraction subunit is used to extract features from the picture data in the second data set based on the picture classification model to obtain a first feature vector;

[0128] A second feature extraction subunit is used to extract features from the text data in the second data set based on the first language model to obtain a second feature vector;

[0129] a loss calculation subunit, configured to calculate a contrast loss function of the image classification model and the first language model based on the first feature vector and the second feature vector;

[0130] The model multimodal training subunit is used to iteratively optimize model parameters of the picture classification model and the first language model based on the contrast loss function until the loss values ​​of the picture classification model and the first language model converge.

[0131] In one embodiment, the document generation device 400 based on the multimodal large model further includes a second model training module, and further includes:

[0132] An image encoding unit, used for encoding a preset image text based on an image encoder to generate a sparse spatial image;

[0133] The model training unit is used to perform model training on a preset diffusion model based on the sparse spatial image and a preset prompt text to obtain the document generation model.

[0134] In one embodiment, the input text includes one or more of picture text, text text and prompt text.

[0135] It should be noted that technical personnel in the relevant field can clearly understand that, for the convenience and conciseness of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned embodiment of the document generation method based on the multimodal large model, and will not be repeated here.

[0136] The apparatus provided in the above embodiment may be implemented in the form of a computer program. The computer program may be Figure 5 Runs on the computer device shown.

[0137] See also Figure 5 , Figure 5 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.

[0138] See also Figure 5 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0139] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any document generation method based on a multimodal large model.

[0140] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0141] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any document generation method based on the multimodal large model.

[0142] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0143] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0144] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0145] Based on the multimodal large model, the outline content of the input text is extracted, and the extracted outline content is structured to output structured text;

[0146] Based on a text parser, the structured text is parsed to obtain text attributes corresponding to the structured text;

[0147] Based on the document generation model, adaptive element matching is performed on the structured text and the text attributes corresponding to the structured text to generate a target document.

[0148] In one embodiment, when the processor extracts outline content from input text based on the multimodal macro model, structures the extracted outline content, and outputs structured text, it is used to implement:

[0149] Acquire the input text, wherein the input text includes image text and first text;

[0150] Based on the text recognition OCR model, the image text is identified and extracted to generate a second text;

[0151] Based on the multimodal macro model and a preset structured data format, key element identification and content matching are performed on the first text and the second text to generate the structured text.

[0152] In one embodiment, before implementing the steps of extracting outline content from input text based on the multimodal macro model, structuring the extracted outline content, and outputting structured text, the processor is further configured to implement:

[0153] Based on the first data set, performing image single-modal training on the image classification model until a loss value of the image classification model converges;

[0154] Based on the second data set, performing image-text multimodal training on the picture classification model and the pre-trained first language model until loss values ​​of the picture classification model and the first language model converge;

[0155] Based on the third data set, training the multilayer perceptron and the second language model until the loss values ​​of the multilayer perceptron and the second language model converge;

[0156] The multimodal large model is constructed based on the trained image classification model, the first language model, the multilayer perceptron and the second language model.

[0157] In one embodiment, when the processor performs image unimodal training on the image classification model based on the first data set until the loss value of the image classification model converges, it is used to implement:

[0158] Acquire a first data set, wherein the first data set includes picture-text pairing data;

[0159] Based on a preset image processing flow, performing data enhancement processing on at least one set of image data collected from the first data set to obtain a third data set;

[0160] Based on the first data set and the third data set, image single-modal iterative training is performed on the picture classification model until the loss value of the picture classification model converges.

[0161] In one embodiment, the processor, when implementing the image-text multimodal training of the picture classification model and the pre-trained first language model based on the second data set until the loss values ​​of the picture classification model and the first language model converge, is used to implement:

[0162] Based on the picture classification model, extract features from the picture data in the second data set to obtain a first feature vector;

[0163] Based on the first language model, extracting features from the text data in the second data set to obtain a second feature vector;

[0164] Calculating a contrast loss function of the image classification model and the first language model based on the first feature vector and the second feature vector;

[0165] Based on the contrast loss function, model parameters of the picture classification model and the first language model are iteratively optimized until loss values ​​of the picture classification model and the first language model converge.

[0166] In one embodiment, before implementing the adaptive element matching of the structured text and the text attributes corresponding to the structured text based on the document generation model to generate the target document, the processor is further configured to implement:

[0167] Based on the image encoder, the preset image text is encoded to generate a sparse spatial image;

[0168] Based on the sparse spatial image and the preset prompt text, a preset diffusion model is trained to obtain the document generation model.

[0169] In one embodiment, the input text includes one or more of picture text, text text and prompt text.

[0170] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement any one of the document generation methods based on a multimodal large model provided in the embodiments of the present application.

[0171] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0172] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A document generation method based on a multimodal large model, characterized in that: The method comprises: Based on the multimodal large model, the outline content of the input text is extracted, and the extracted outline content is structured to output structured text; Based on a text parser, the structured text is parsed to obtain text attributes corresponding to the structured text; Based on the document generation model, adaptive element matching is performed on the structured text and the text attributes corresponding to the structured text to generate a target document.

2. The document generation method based on a multimodal large model according to claim 1, characterized in that: The method of extracting outline content from input text based on the multimodal large model, structuring the extracted outline content, and outputting structured text includes: Acquire the input text, wherein the input text includes image text and first text; Based on the text recognition OCR model, the image text is identified and extracted to generate a second text; Based on the multimodal macro model and a preset structured data format, key element identification and content matching are performed on the first text and the second text to generate the structured text.

3. The document generation method based on a multimodal large model according to claim 1, characterized in that: The method further includes extracting outline content from the input text based on the multimodal large model, and structuring the extracted outline content before outputting the structured text: Based on the first data set, performing image single-modal training on the image classification model until a loss value of the image classification model converges; Based on the second data set, performing image-text multimodal training on the picture classification model and the pre-trained first language model until loss values ​​of the picture classification model and the first language model converge; Based on the third data set, training the multilayer perceptron and the second language model until the loss values ​​of the multilayer perceptron and the second language model converge; The multimodal large model is constructed based on the trained image classification model, the first language model, the multilayer perceptron and the second language model.

4. The document generation method based on a multimodal large model according to claim 3 is characterized in that: The performing image single-modality training on the image classification model based on the first data set until the loss value of the image classification model converges includes: Acquire a first data set, wherein the first data set includes picture-text pairing data; Based on a preset image processing flow, performing data enhancement processing on at least one set of image data collected from the first data set to obtain a third data set; Based on the first data set and the third data set, image single-modal iterative training is performed on the picture classification model until the loss value of the picture classification model converges.

5. The document generation method based on a multimodal large model according to claim 3 is characterized in that: The step of performing image-text multimodal training on the picture classification model and the pre-trained first language model based on the second data set until the loss values ​​of the picture classification model and the first language model converge, includes: Based on the picture classification model, extract features from the picture data in the second data set to obtain a first feature vector; Based on the first language model, extracting features from the text data in the second data set to obtain a second feature vector; Calculating a contrast loss function of the image classification model and the first language model based on the first feature vector and the second feature vector; Based on the contrast loss function, model parameters of the picture classification model and the first language model are iteratively optimized until loss values ​​of the picture classification model and the first language model converge.

6. The document generation method based on a multimodal large model according to claim 1 is characterized in that: The method further comprises: performing adaptive element matching on the structured text and the text attributes corresponding to the structured text based on the document generation model before generating the target document: Based on the image encoder, the preset image text is encoded to generate a sparse spatial image; Based on the sparse spatial image and the preset prompt text, a preset diffusion model is trained to obtain the document generation model.

7. The document generation method based on a multimodal large model according to claim 1 is characterized in that: The input text includes one or more of picture text, text text and prompt text.

8. A document generation device based on a multimodal large model, characterized in that: The document generation device based on the multimodal large model includes: The structured text output module extracts the outline content of the input text based on the multimodal large model, structures the extracted outline content, and outputs the structured text; A text attribute parsing module, used to perform content parsing on the structured text based on a text parser to obtain text attributes corresponding to the structured text; The presentation generation module is used to perform adaptive element matching on the structured text and the text attributes corresponding to the structured text based on the document generation model to generate a target document.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the document generation method based on a multimodal large model as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the document generation method based on a multimodal large model as described in any one of claims 1 to 7 are implemented.

Images