Demonstration document generation model training method and device and electronic equipment

Abstract

The invention relates to a demonstration document generation model training method and device and electronic equipment, and the method comprises the steps: extracting embedded images in a training demonstration document, carrying out the encoding of the embedded images through an encoder, and obtaining the encoding ID of each embedded image; in initial HTML data obtained by conversion of the training demonstration document, filling an image tag bit with the code ID to obtain target HTML data; the method comprises the steps of obtaining a description text of a training demonstration document, obtaining generated HTML data according to the description text through a large language model, taking an image tag bit in the generated HTML data as a target code ID, and constructing a text loss function according to the generated HTML data and the target HTML data; decoding the target coding ID through a decoder to obtain a decoded image, and constructing an image loss function according to the decoded image and the embedded image; and constructing a training loss function according to the text loss function and the image loss function, training the large language model and the decoder until the training loss function reaches a convergence condition, and obtaining a demonstration document generation model.

Description

Technical Field

[0001] The present application relates to the field of machine learning, and in particular to a method, device, and electronic device for training a presentation document generation model. Background Art

[0002] In the existing technology, during the training process of generating a PPT, the images involved are converted into SACII characters based on the Base64 encoding rules, and then the SACII characters are concatenated with the PPT text description. The concatenated training samples are processed through a transformer model to obtain a mapping between the text description and the HTML code.

[0003] The Base64 encoding rule encodes every 3 bytes of binary data into 4 characters of ASCII, which makes the image data bloated. times, making the transformer model training time-consuming. On the other hand, Base64 encoding destroys the direct spatial correlation between pixels, so the transformer model needs to reconstruct high-level semantic information from relatively low-level features (such as how a specific character sequence corresponds to a certain color gradient), and spends a lot of resources to learn how to map the character sequence in Base64 encoding back to the corresponding pixel value in the image, and how to understand the relationship between these pixel values. This causes a modality mismatch between the Base64-encoded signal representation and the transformer model architecture, resulting in the training focusing on image reconstruction during the PPT generation training process, and failing to generate PPT efficiently. Therefore, how to improve the efficiency of the PPT generation training process has become an urgent problem to be solved. Summary of the Invention

[0004] The embodiments of the present application provide a method, device and electronic device for training a presentation document generation model to at least solve the problem of low efficiency in the PPT training process in related technologies.

[0005] In a first aspect, an embodiment of the present application provides a method for training a presentation document generation model, the method being applied to a large language model and a VQVAE model, the VQVAE model including an encoder and a decoder, the method comprising:

[0006] Extracting embedded images from the training demonstration document, encoding the embedded images using the encoder, and obtaining an encoding ID for each embedded image;

[0007] In the initial HTML data obtained by converting the training demonstration document, the encoding ID is filled in the image tag position to obtain the target HTML data;

[0008] Obtaining a description text of the training demonstration document, generating HTML data based on the description text using a large language model, wherein the image label in the generated HTML data is a target encoding ID, and constructing a text loss function based on the generated HTML data and the target HTML data;

[0009] Decoding the target code ID by the decoder to obtain a decoded image, and constructing an image loss function based on the decoded image and the embedded image;

[0010] A training loss function is constructed according to the text loss function and the image loss function, and the large language model and the decoder are trained until the training loss function reaches a convergence condition, thereby obtaining a presentation document generation model.

[0011] In one embodiment, obtaining the description text based on the training demonstration document includes:

[0012] Converting each slide of the training presentation document into a page image;

[0013] Through the multimodal large model, a single page description text is generated according to each of the page images, and a continuous page description text is generated according to all the page images.

[0014] In one embodiment, constructing a training loss function based on the text loss function and the image loss function, training the large language model and the decoder until the training loss function reaches a convergence condition, and obtaining a presentation document generation model includes:

[0015] Constructing a first training loss function based on the single-page description text, and training the large language model and the decoder until the first training loss function converges, thereby obtaining a first model weight of the large language model and a second model weight of the encoder;

[0016] updating the large language model based on the first model weights, and updating the encoder based on the second model weights;

[0017] A second training loss function is constructed based on the continuous page text, and the updated decoder and the large language model are trained until the second training loss function converges to obtain the presentation document generation model.

[0018] In one embodiment, constructing a first training loss function based on the single-page description text, and training the large language model and the decoder until the first training loss function converges, thereby obtaining a first model weight of the large language model and a second model weight of the encoder, includes:

[0019] Based on the single-page description text, a single-page HTML data is obtained through the large language model, the image tag of the single-page HTML data is a first encoded ID, and a first text loss function is constructed according to the single-page HTML data and the target HTML data;

[0020] Decoding the first coding ID by the decoder to obtain a single-page decoded image, and constructing a first image loss function based on the single-page decoded image and the embedded image;

[0021] Constructing the first training loss function according to the first text loss function and the first image loss function;

[0022] The large language model and the decoder are trained until the first training loss function reaches a convergence condition, thereby obtaining a first model weight of the large language model and a second model weight of the encoder.

[0023] In one embodiment, constructing a second training loss function based on the continuous page text, and training the updated decoder and the large language model until the second training loss function converges to obtain the presentation document generation model, includes:

[0024] According to the continuous page description text, obtaining continuous page HTML data through the updated large language model, wherein the image tag of the continuous page HTML data is a second encoding ID, and constructing a second text loss function according to the continuous page HTML data and the initial HTML data;

[0025] Decoding the second encoding ID by the updated decoder to obtain continuous page decoded images, and constructing a second image loss function based on the continuous page decoded images and the embedded image;

[0026] Constructing the second training loss function according to the second text loss function and the second image loss function;

[0027] The updated decoder and the large language model are trained until the second training loss function reaches a convergence condition, thereby obtaining the presentation document generation model.

[0028] In one embodiment, the large language model includes an embedding layer and a Transformer decoding layer, and generating HTML data according to the description text using the large language model includes:

[0029] Segmenting the description text to obtain a plurality of tokens, and serializing all of the tokens to obtain an input sequence;

[0030] Inputting the input sequence into the embedding layer to obtain an embedding vector;

[0031] Processing the embedding vector through the Transformer decoding layer to update the hidden state layer by layer;

[0032] Obtaining a probability distribution of the token based on the hidden state of the last layer in the Transformer decoding layer, and using the token with the highest probability in the probability distribution as the generated token;

[0033] The generated tag is added to the embedding vector to update the embedding vector, and the embedding vector is continuously updated through the Transformer decoding layer until an end condition is met, and the finally updated embedding vector is used as the generated HTML data.

[0034] In one embodiment, extracting embedded images from the training demonstration document, encoding the embedded images using the encoder, and obtaining an encoding ID for each embedded image includes:

[0035] Extracting the embedded image in the training demonstration document, and standardizing and preprocessing the embedded image;

[0036] The processed embedded image is input into the encoder, and the encoder maps the continuous feature map of the processed embedded image to a preset code book to obtain the encoding ID of each embedded image.

[0037] In a second aspect, an embodiment of the present application provides a method for generating a presentation document, which is applied to the presentation document generation model of the first aspect, wherein the presentation document generation model includes a large language model and a decoder of a VQVAE model, including:

[0038] Inputting a document generation instruction into the large language model in the presentation document generation model to obtain HTML data, wherein the image tag in the HTML data is a coded ID;

[0039] Decoding the encoded ID by a decoder in the presentation document generation model to obtain a decoded image;

[0040] The HTML data is parsed to obtain page content of a presentation document, and a presentation document is generated according to the page content and the decoded image.

[0041] In a third aspect, an embodiment of the present application provides a presentation document generation training device, characterized by comprising:

[0042] An encoding module, configured to extract embedded images from a training demonstration document, encode the embedded images using the encoder, and obtain an encoding ID for each embedded image;

[0043] A first obtaining module is configured to fill the encoding ID in the image tag position of the initial HTML data obtained by converting the training demonstration document to obtain target HTML data;

[0044] Constructing a text loss function module, which is used to obtain the description text of the training demonstration document, generate HTML data based on the description text through a large language model, fill the image label bits in the generated HTML data with the target encoding ID, and construct a text loss function based on the target HTML data and the initial HTML data;

[0045] Constructing an image loss function module, configured to decode the target code ID through the decoder to obtain a generated image, and constructing an image loss function based on the generated image and the embedded image;

[0046] A demonstration document generation model module is obtained, which is used to construct a training loss function based on the text loss function and the image loss function, train the large language model and the decoder until the training loss function reaches a convergence condition, and obtain a demonstration document generation model.

[0047] In a fourth aspect, an embodiment of the present application provides a computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the method for training a presentation document generation model as described in the first aspect above is implemented.

[0048] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the presentation document generation model training method as described in the first aspect above.

[0049] The present invention provides a method, device and electronic device for training a presentation document generation model, which have at least the following technical effects.

[0050] By training the demonstration document, the initial HTML data and embedded image are obtained. The embedded image is encoded through the VQVAE encoder to obtain the encoding ID, which can retain key image features and remove pixel redundancy, thereby achieving efficient image compression. The encoding ID is used to fill the image tag bit in the initial HMTL data to obtain the target HTML data, so that the target HTML data only includes text data. This allows the large language model to focus on complex language structure and semantic information during training, which is beneficial to improving the quality of generated HTML data. By constructing a text loss function based on the target HTML data and the generated HTML data, and an image loss function based on the embedded image and the decoded image, the large language model and decoder are trained, so that the large language model focuses on training text data, while the VQVAE decoder focuses on training image data, achieving separate training of text data and images, thereby improving training efficiency.

[0051] The details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0053] Figure 1 is a flowchart of a method for training a presentation document generation model according to an exemplary embodiment;

[0054] Figure 2 is a flowchart of a method for training a presentation document generation model according to another exemplary embodiment;

[0055] Figure 3 is a block diagram of a presentation document generation training device according to an exemplary embodiment;

[0056] Figure 4 is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is described and illustrated below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely used to explain this application and are not intended to limit this application. Based on the embodiments provided in this application, all other embodiments obtained by those of ordinary skill in the art without making any creative efforts are within the scope of protection of this application.

[0058] Obviously, the drawings described below are merely examples or embodiments of the present application. Those skilled in the art can, without inventive effort, apply the present application to other similar scenarios based on these drawings. Furthermore, it is also understood that, although the effort involved in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, changes in design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as an insufficiency of the content disclosed in this application.

[0059] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it refer to independent or alternative embodiments that are mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments unless there is a conflict.

[0060] Unless otherwise defined, the technical or scientific terms used in this application should have the ordinary meaning understood by a person of ordinary skill in the technical field to which this application belongs. The words "one", "a", "the" and the like used in this application do not indicate a limit on quantity and may indicate the singular or plural. The terms "include", "comprise", "have" and any variations thereof used in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units that are inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The word "multiple" used in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific order for the objects.

[0061] In a first aspect, the present invention provides a method for training a presentation document generation model. Figure 1 is a flowchart of a method for training a presentation document generation model according to an exemplary embodiment. Figure 1As shown, the demonstration document generation model training method is applied to a large language model and a VQVAE model. The VQVAE model includes an encoder and a decoder. The demonstration document generation model training method should include:

[0062] Step S101: extract embedded images from the training demonstration document, encode the embedded images through an encoder, and obtain the encoding ID of each embedded image.

[0063] Each slide in the training presentation document contains several embedded images. All embedded images are extracted from the training presentation document. All embedded images are passed to the encoder to encode the embedded images and obtain the encoded ID of each embedded image.

[0064] The method for obtaining the encoded ID of each embedded image specifically includes:

[0065] Step S111: extract the embedded image in the training demonstration document, standardize and pre-process the embedded image, and obtain an input image.

[0066] Each embedded image is normalized. This means scaling each embedded image to a pre-set size, ensuring that all embedded images have the same size. The pixel values ​​in each embedded image are then normalized to the range [-1, 1] to obtain the input image.

[0067] Step S112: input the input image into the encoder, and map the processed continuous feature map of the embedded image to a preset codebook through the encoder to obtain the encoding ID of each embedded image.

[0068] The input image is fed into the encoder of the VQVAE model. Through the encoder's internal network, the input image is converted into a continuous feature map. A continuous feature map is a matrix-like feature map where each element is a continuous value.

[0069] For each vector of continuous eigenvalues, the closest embedding vector is found in the preset codebook. The index of the closest embedding vector is determined, where the index represents the position of the embedding vector in the preset codebook. The corresponding vector is replaced by the index to obtain a discrete vector. This discrete vector serves as the encoded ID for each embedded image after it is processed by the encoder.

[0070] Continuing with step S101, the VQVAE encoder encodes the image, converting the originally complex continuous feature map into a series of discrete index values. These indexes point to specific embedding vectors in a preset codebook, thereby achieving an efficient mapping from continuous space to discrete space. The encoded ID obtained by encoding the image through the VQVAE encoder not only retains the key semantic information of the embedded image, but also has higher information density and better generalization ability, thereby reducing the computational burden of the subsequent training process.

[0071] Step S102: Fill the encoding ID into the image tag position in the initial HTML data obtained by converting the training demonstration document to obtain the target HTML data.

[0072] The training demonstration document is converted into initial HTML data using a conversion tool. During this conversion, the image labels in the initial HTML data are used as placeholders. By replacing complex image encodings with placeholders, we avoid embedding lengthy image encodings directly in the initial HTML data, thus reducing the dataset size and training complexity.

[0073] The obtained encoding ID replaces the placeholder in the initial HTML data, so as to fill the encoding ID in the image tag position and obtain the target HTML data. That is, in the target HTML data, the image tag position of the target HTML data is the encoding ID.

[0074] Because the encoded ID is obtained by encoding with the VQVAE encoder, it not only contains the key semantic information of the embedded image but also has a higher information density, effectively compressing the embedded image. As a result, the target HTML data contains high-dimensional image information, which occupies very little storage space in the first HTML, avoiding the coexistence of large amounts of image and text information in the target HTML data.

[0075] Step S103: Obtain the description text of the training demonstration document, generate HTML data based on the description text through the large language model, use the image label bit in the generated HTML data as the target encoding ID, and construct a text loss function based on the generated HTML data and the target HTML data.

[0076] Obtain the description text of the training presentation document. This description text describes the content of the slides in the training presentation document, including the text content and the position of each element on the slide page. Use the tokenizer to tokenize and serialize the description text to obtain an input sequence. This input sequence is fed into the embedding layer of the large language model to obtain an embedding vector, which serves as the initial hidden state.

[0077] The embedding vector is input into the Transformer decoding layer of the large language model, and the hidden state of each token is updated in each Transformer decoding layer through multi-layer self-attention mechanism and feedforward neural network processing. Until the hidden state in the last Transformer decoding layer, the probability distribution of each token is obtained. The token with the highest probability in the probability distribution is used as the generated token, and the generated token is added to the embedding vector to update the embedding vector. The updated embedding vector is processed through the Transformer decoding layer again to continuously update the embedding vector until the end condition is met, and the final embedding vector is obtained. The embedding vector is then sent back to the word segmenter, and the final embedding vector is converted into HTML data to obtain generated HTML data. Optionally, the end condition includes reaching a preset maximum sequence length and encountering a preset end marker.

[0078] In the generated HTML data, the image label in the generated HTML data is the target encoding ID. The target encoding ID is the encoding ID corresponding to the VQVAE encoder generated by the large language model. A text loss function is constructed based on the generated HTML data and the target HTML data. The text loss function is specifically:

[0079]

[0080] L text is the text loss function, y t is the t-th text, y<t is the text sequence before the t-th text, P(y t |y ＜t ) is the probability of the model predicting the t-th text in the case of the text sequence before the t-th text, and T is the sequence length of the generated HTML data.

[0081] The large language model generates HTML data with the target encoding ID as the image label from the description text. By constructing a text loss function, the generated HTML data is made close to the target HTML data. Because the encoding ID is an efficient compression of the embedded image, when the large language model obtains the generated HTML data, it no longer uses a large amount of resources to reconstruct the image based on the complex image encoding. Instead, it generates a target encoding ID similar to the encoding ID. This allows the large language model to use more learning resources to capture the complex language structure and semantic information in the description text, thereby improving the accuracy and coherence of the text generation and obtaining high-quality HTML content.

[0082] Step S104: decode the target code ID through a decoder to obtain a decoded image, and construct an image loss function based on the decoded image and the embedded image.

[0083] The output of the large language model is connected to the VQVAE decoder. The target encoding ID in the generated HTML data generated by the large language model is input into the decoder. The decoder decodes the target encoding ID to obtain a decoded image. An image loss function is constructed based on the decoded image and the embedded image. The image loss function is specifically:

[0084]

[0085] Among them, L image is the image loss function, N is the total number of pixels in the embedded image and the decoded image, the total number of pixels in the embedded image and the decoded image are the same, I i is the i-th pixel value of the embedded image, is the i-th pixel value of the decoded image.

[0086] The VQVAE decoder can decode into high-quality images based on the target encoding ID. The VQVAE decoder only focuses on generating images, and the target encoding ID is close to the efficiently compressed encoding ID, thereby making full use of the characteristics of the VQVAE model, thereby ensuring high-fidelity image requirements while reducing the computational burden.

[0087] Step S105: construct a training loss function based on the text loss function and the image loss function, train the large language model and the decoder until the training loss function reaches the convergence condition, and obtain a presentation document generation model.

[0088] A training loss function is constructed based on the text loss function and the image loss function, and the large language model and decoder are trained. During the training process, the internal parameters of the large prediction model and the decoder are continuously updated so that the training loss function reaches the convergence condition and a demonstration document generation model is obtained, so that a demonstration document can be obtained based on the demonstration document generation model. The training loss function is specifically:

[0089] L total =L text +λ·L image

[0090] Among them, L total is the training loss function, L text is the text loss function, L image is the image loss function, and λ is a hyperparameter that balances the weights of text and image losses.

[0091] In one embodiment, training of the large language model and decoder is divided into two stages: a single-page training stage and a continuous-page training stage. The description texts used in the training process include single-page description texts and continuous-page description texts. The single-page description texts and continuous-page description texts are obtained as follows:

[0092] Convert each slide of the training presentation document into a page image. Input the page image into the multimodal large model, and use the multimodal large model's joint visual and language understanding capabilities to generate single-page description text based on each page image, and generate continuous page description text based on all page images. Among them, the single-page description text describes the content of a page of slides, including text content and the position of each element, and is used in the single-page training stage. The continuous page description text describes the content of slides in several consecutive pages and is used in the continuous page training stage. The difference between single-page description text and multi-page continuous description text is that the single-page description text only describes a single page of slides, while the continuous page description text describes all continuous slides in the presentation document.

[0093] Continuing with step S105, training the large language model and encoder is divided into two stages. Specifically, the two training stages are:

[0094] Step S511: construct a first training loss function based on the single-page description text, and train the large language model and the decoder until the first training loss function converges to obtain the first model weight of the large language model and the second model weight of the encoder.

[0095] The large language model obtains a single page of HTML data based on the single page of descriptive text, where the image tag in the single page of HTML data is a first encoded ID. A first text loss function is constructed based on the single page of HTML data and the target HTML data. The decoder decodes the first encoded ID to obtain a single page of decoded image, and a first image loss function is constructed based on the single page of decoded image and the embedded image. A first training loss function is constructed using the first text loss function and the first image loss function. The large language model and the decoder are trained until the first training loss function reaches a convergence condition, thereby obtaining the first model weights of the large language model and the second model weights of the encoder.

[0096] It should be noted that the first model weight of the large language model and the second model weight of the encoder represent the connection strength between nodes in different layers within the model, reflecting the influence of input features on the output. When the first training loss function reaches convergence, all relevant parameters within the large language model and decoder reach optimal state.

[0097] Step S512: Update the large language model based on the first model weights, and update the encoder based on the second model weights.

[0098] The first model weights obtained in the single-page training phase are used to update the large language model and the second model weights are used to update the encoder, so that the two models can obtain the training results of the single-page training phase and converge faster in the second training process.

[0099] Step S513: construct a second training loss function based on the continuous page text, and train the updated decoder and large language model until the second training loss function converges to obtain a presentation document generation model.

[0100] The updated large language model obtains continuous page HTML data based on the continuous page description text, where the image tag in the continuous page HTML data is the second encoded ID. A first text loss function is constructed based on the continuous page HTML data and the target HTML data. The decoder decodes the second encoded ID to obtain a continuous page decoded image, and a second image loss function is constructed based on the continuous page decoded image and the embedded image. A second training loss function is constructed using the second text loss function and the second image loss function. The large language model and decoder are trained until the second training loss function reaches a convergence condition, thereby obtaining a presentation document generation model.

[0101] The updated large language model and decoder have the results of the single-page training phase. When trained on this basis, the large language model and decoder will not only converge faster, but will also further refine the generated content based on the single-page training phase, making the generated continuous page HTML data and continuous page decoded images more accurate, thereby improving the quality of the generated HTML data and images.

[0102] Continuing to refer to steps S511 to S513, by training the large language model and the decoder in stages, the two models can progressively refine the generated content, making the generated content closer to the reference data in the training data, thereby ensuring the output of high-quality content.

[0103] In another embodiment, before extracting the embedded image in the training demonstration document in step S101, the training demonstration document for training the model needs to be screened. The method of screening the training demonstration document specifically includes:

[0104] Obtain several presentation documents and categorize them by keywords to ensure that the presentation documents used for training cover a variety of fields. For example, presentation document types include business reports, educational courseware, cultural promotions, product demonstrations, and event planning.

[0105] After the initial screening of presentations, we also need to filter them by complexity to ensure that they use relatively simple content, thereby reducing complexity during the training process. Optionally, we retain presentations where the area of ​​all elements in the slides exceeds 50% of the total page area to avoid adding too many elements to the training data, which would increase the time and complexity of training.

[0106] Some presentation documents are commercial documents. To avoid copyright disputes, copyright cleaning is required to remove commercial presentation documents. By extracting open source protocol identifiers, such as CC-BY and MIT, from presentation documents, we can determine that the presentation documents used for training are open source. Commercial presentation documents are then removed, ensuring that the presentation documents used for training are open source.

[0107] In another embodiment, Figure 2 is a flowchart of a method for training a presentation document generation model according to another exemplary embodiment. Figure 2 As shown, the original presentation document is processed by the preprocessing module to obtain a training presentation document, and the slides in the training presentation document are converted into page images in PNG format, and the page images are converted into description texts. The description texts include single-page description texts and continuous page description texts. The training presentation texts are converted into HTML data through a conversion tool, wherein the HTML data includes single-page HTML data and continuous page HTML data, and the image tag in each HTML data is vqid, that is, the encoding ID. The single-page HTML data and the single-page description text are constructed as a single-page training set, and the continuous page HTML data and the continuous page description text are constructed as a multi-page training set. Based on the two training sets, the large language model and the decoder perform two-stage training. When the two-stage training is completed, a presentation document generation model is obtained. In the presentation document generation model, the large language model generates HTML data containing vqid, the VQVAE decoder decodes the vqid to generate an image, parses the HTML data to obtain the page content, and combines the page content and the generated image to obtain a presentation document.

[0108] The above method is used to screen the presentation documents and determine the presentation documents used for training. Among these presentation documents, the diverse types of presentation documents provide rich training data for the training process. These training data do not include too many complex elements, which reduces the complexity of the training data and avoids copyright disputes.

[0109] In summary, the training method for a presentation document generation model provided by the embodiments of the present application, during the training process, encodes images using a VQVAE encoder to achieve efficient image compression, thereby reducing the computational burden and significantly reducing the computing resources required to process image data, thereby improving the overall system efficiency. The VQVAE decoder can directly convert the encoded ID in the generated HTML code into high-quality image data, avoiding the need to use a large language model to learn to generate images from lengthy encodings, thereby improving the model training efficiency from the perspective of network structure design. The large language model is used to generate HTML data containing the encoded ID based on the descriptive text, allowing the large language model to focus on more accurately capturing the complex language structure and semantic information in the text, thereby generating high-quality HTML content. By processing text and images separately and giving full play to the respective advantages of the two models, image quality and text coherence are guaranteed, thereby improving training efficiency. The large language model and decoder are trained in stages, improving the quality of the generated content. In addition, the separate processing of images and text gives the presentation document generation model greater flexibility and modularity, so that each component can be independently adjusted or upgraded according to specific needs, facilitating maintenance and expansion.

[0110] In a second aspect, an embodiment of the present application provides a method for generating a presentation document, the method comprising:

[0111] The document generation instruction is input into the large language model in the presentation document generation model to obtain HTML data, wherein the image tag in the HTML data is the encoded ID.

[0112] Decode the encoded ID using the decoder in the demo document generation model to obtain the decoded image.

[0113] Parse HTML data to obtain the page content of the presentation document, and generate the presentation document based on the page content and decoded images.

[0114] The document generation instruction represents the content of the presentation document that the user wishes to generate. Therefore, the user can input the document generation instruction into the presentation document generation model. The document generation instruction is then processed by the large language model to generate HTML data, where the image tag is an encoded ID. The decoder decodes the encoded ID to obtain a decoded image. After parsing the HTML data, the page content of each slide in the presentation document is obtained. Based on the page content and the decoded image, the presentation document is generated.

[0115] The presentation document generation model can be used to obtain the presentation document expected by the user, thereby improving the convenience of obtaining the presentation document.

[0116] In a third aspect, an embodiment of the present application provides a demonstration document generation training device. Figure 3 FIG. 1 is a block diagram of a training device for generating a presentation document according to an exemplary embodiment. Figure 3 As shown, the presentation document generation training device includes: an encoding module for extracting embedded images in the training presentation document, encoding the embedded images through the encoder, and obtaining an encoding ID of each embedded image;

[0117] A first obtaining module is configured to fill the encoding ID in the image tag position of the initial HTML data obtained by converting the training demonstration document to obtain target HTML data;

[0118] Constructing a text loss function module, for obtaining a description text of the training presentation document, generating HTML data based on the description text through a large language model, filling image label bits in the generated HTML data with a target encoding ID, and constructing a text loss function based on the target HTML data and the target HTML data;

[0119] Constructing an image loss function module, configured to decode the target code ID through the decoder to obtain a decoded image, and constructing an image loss function based on the generated image and the embedded image;

[0120] A demonstration document generation model module is obtained, which is used to construct a training loss function based on the text loss function and the image loss function, train the large language model and the decoder until the training loss function reaches a convergence condition, and obtain a demonstration document generation model.

[0121] In summary, the presentation document generation training device provided by this application obtains initial HTML data and embedded images by training presentation documents, and then encodes the embedded images through a VQVAE encoder to obtain an encoding ID. This can preserve key image features and remove pixel redundancy, thereby achieving efficient image compression. The encoding ID is used to fill the image tag bits in the initial HTML data to obtain target HTML data, so that the target HTML data only includes text data. This allows the large language model to focus on complex language structures and semantic information during training, which is beneficial for improving the quality of generated HTML data. By constructing a text loss function based on the target HTML data and the generated HTML data, and constructing an image loss function based on the embedded image and the decoded image, the large language model and decoder are trained, so that the large language model focuses on training text data, while the VQVAE decoder focuses on training image data, achieving separate training of text data and images, thereby improving training efficiency.

[0122] It should be noted that the presentation document generation training device provided in this embodiment is used to implement the above-mentioned embodiments, and the details that have been explained will not be repeated. As used above, the terms "module," "unit," "subunit," etc. can be a combination of software and / or hardware that implements the predetermined functions. Although the device described in the above embodiments is preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0123] In a fourth aspect, an embodiment of the present application provides an electronic device, Figure 4 FIG is a block diagram of an electronic device according to an exemplary embodiment. Figure 4 As shown, the electronic device may include a processor 81 and a memory 82 storing computer program instructions.

[0124] Specifically, the processor 81 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0125] Among them, the memory 82 may include a large-capacity memory for data or instructions. By way of example and not limitation, the memory 82 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 82 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 82 may be inside or outside the data processing device. In a specific embodiment, the memory 82 is a non-volatile memory. In a specific embodiment, the memory 82 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM) or a flash memory (FLASH), or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (SRAM) or a dynamic random access memory (DRAM), where the DRAM can be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0126] The memory 82 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 81 .

[0127] The processor 81 implements any one of the presentation document generation training methods in the above embodiments by reading and executing computer program instructions stored in the memory 82 .

[0128] In one embodiment, the presentation document generation training device may further include a communication interface 83 and a bus 80. Figure 4 As shown, the processor 81, the memory 82, and the communication interface 83 are connected via a bus 80 and communicate with each other.

[0129] The communication interface 83 is used to enable communication between the various modules, devices, units, and / or devices in the embodiments of the present application. The communication port 83 can also enable data communication with other components such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations.

[0130] Bus 80 includes hardware, software, or both, and couples the components of the presentation document generation training device to each other. Bus 80 includes, but is not limited to, at least one of the following: a data bus, an address bus, a control bus, an expansion bus, and a local bus. By way of example and not limitation, bus 80 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Bus 80 may include one or more buses, where appropriate. Although embodiments herein describe and illustrate a particular bus, this application contemplates any suitable bus or interconnect.

[0131] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the presentation document generation training method provided in the first aspect.

[0132] The readable storage medium may include, but is not limited to, a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0133] In a possible implementation, the present invention can also be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of the presentation document generation training method provided in the first aspect.

[0134] The program code for executing the present invention may be written in any combination of one or more programming languages, and may be executed entirely on the user device, partially on the user device, as an independent software package, partially on the user device and partially on a remote device, or entirely on the remote device.

[0135] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0136] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A method for training a presentation document generation model, characterized in that: The method is applied to a large language model and a VQVAE model, wherein the VQVAE model includes an encoder and a decoder, and the method includes: Extracting embedded images from the training demonstration document, encoding the embedded images using the encoder, and obtaining an encoding ID for each embedded image; In the initial HTML data obtained by converting the training demonstration document, the encoding ID is filled in the image tag position to obtain the target HTML data; Obtaining a description text of the training demonstration document, generating HTML data based on the description text using a large language model, wherein the image label in the generated HTML data is a target encoding ID, and constructing a text loss function based on the generated HTML data and the target HTML data; Decoding the target code ID by the decoder to obtain a decoded image, and constructing an image loss function based on the decoded image and the embedded image; A training loss function is constructed according to the text loss function and the image loss function, and the large language model and the decoder are trained until the training loss function reaches a convergence condition, thereby obtaining a presentation document generation model.

2. The method for training a presentation document generation model according to claim 1, wherein: The obtaining of a description text based on the training demonstration document includes: Converting each slide of the training presentation document into a page image; Through the multimodal large model, a single page description text is generated according to each of the page images, and a continuous page description text is generated according to all the page images.

3. The method for training a presentation document generation model according to claim 2, wherein: The step of constructing a training loss function based on the text loss function and the image loss function, training the large language model and the decoder until the training loss function reaches a convergence condition, and obtaining a presentation document generation model includes: Constructing a first training loss function based on the single-page description text, and training the large language model and the decoder until the first training loss function converges, thereby obtaining a first model weight of the large language model and a second model weight of the encoder; updating the large language model based on the first model weights, and updating the encoder based on the second model weights; A second training loss function is constructed based on the continuous page text, and the updated decoder and the large language model are trained until the second training loss function converges to obtain the presentation document generation model.

4. The method for training a presentation document generation model according to claim 3, wherein: The step of constructing a first training loss function based on the single-page description text, training the large language model and the decoder until the first training loss function converges, and obtaining a first model weight of the large language model and a second model weight of the encoder includes: Based on the single-page description text, obtaining single-page HTML data through the large language model, wherein the image tag of the single-page HTML data is a first encoding ID, and constructing a first text loss function according to the single-page HTML data and the target HTML data; Decoding the first coding ID by the decoder to obtain a single-page decoded image, and constructing a first image loss function based on the single-page decoded image and the embedded image; Constructing the first training loss function according to the first text loss function and the first image loss function; The large language model and the decoder are trained until the first training loss function reaches a convergence condition, thereby obtaining a first model weight of the large language model and a second model weight of the encoder.

5. The method for training a presentation document generation model according to claim 3, wherein: The step of constructing a second training loss function based on the continuous page text, and training the updated decoder and the large language model until the second training loss function converges, thereby obtaining the presentation document generation model, includes: According to the continuous page description text, obtaining continuous page HTML data through the updated large language model, wherein the image tag of the continuous page HTML data is a second encoding ID, and constructing a second text loss function according to the continuous page HTML data and the initial HTML data; Decoding the second encoding ID by the updated decoder to obtain continuous page decoded images, and constructing a second image loss function based on the continuous page decoded images and the embedded image; Constructing the second training loss function according to the second text loss function and the second image loss function; The updated decoder and the large language model are trained until the second training loss function reaches a convergence condition, thereby obtaining the presentation document generation model.

6. The method for training a presentation document generation model according to claim 1, wherein: The large language model includes an embedding layer and a Transformer decoding layer. The generating of HTML data according to the description text by the large language model includes: Segmenting the description text to obtain a plurality of tokens, and serializing all of the tokens to obtain an input sequence; Inputting the input sequence into the embedding layer to obtain an embedding vector; Processing the embedding vector through the Transformer decoding layer to update the hidden state layer by layer; Obtaining a probability distribution of the token based on the hidden state of the last layer in the Transformer decoding layer, and using the token with the highest probability in the probability distribution as the generated token; The generated markup is added to the embedding vector to update the embedding vector, the embedding vector is continuously updated through the Transformer decoding layer until an end condition is met, and the generated HTML data is obtained according to the finally updated embedding vector.

7. The method for training a presentation document generation model according to claim 1, wherein: The extracting embedded images from the training demonstration document, encoding the embedded images by the encoder, and obtaining the encoding ID of each embedded image includes: Extracting the embedded image from the training demonstration document, and standardizing and preprocessing the embedded image to obtain an input image; The input image is input into the encoder, and the encoder maps the processed continuous feature map of the embedded image to a preset code book to obtain the encoding ID of each embedded image.

8. A method for generating a presentation document, characterized in that: The method is applied to a presentation document generation model according to any one of claims 1 to 7, wherein the presentation document generation model includes a large language model and a decoder of a VQVAE model including: Inputting a document generation instruction into the large language model in the presentation document generation model to obtain HTML data, wherein the image tag in the HTML data is a coded ID; Decoding the encoded ID by a decoder in the presentation document generation model to obtain a decoded image; The HTML data is parsed to obtain page content of a presentation document, and a presentation document is generated according to the page content and the decoded image.

9. A demonstration document generation training device, characterized in that: include: An encoding module, configured to extract embedded images from a training demonstration document, encode the embedded images using the encoder, and obtain an encoding ID for each embedded image; A first obtaining module is configured to fill the encoding ID in the image tag position of the initial HTML data obtained by converting the training demonstration document to obtain target HTML data; Constructing a text loss function module, for obtaining a description text of the training presentation document, generating HTML data based on the description text through a large language model, filling image label bits in the generated HTML data with a target encoding ID, and constructing a text loss function based on the target HTML data and the target HTML data; Constructing an image loss function module, configured to decode the target code ID through the decoder to obtain a generated image, and constructing an image loss function based on the generated image and the embedded image; A demonstration document generation model module is obtained, which is used to construct a training loss function based on the text loss function and the image loss function, train the large language model and the decoder until the training loss function reaches a convergence condition, and obtain a demonstration document generation model.

10. An electronic device, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the presentation document generation model training method as described in any one of claims 1 to 7.

Images