A presentation document generation model training method, device and electronic equipment
By using the VQVAE model to encode images and training the large language model in stages, the problem of low training efficiency in PPT generation is solved. This achieves efficient separate training of text and images, improving the quality and efficiency of PPT generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 国科知机(杭州)智能科技有限公司
- Filing Date
- 2025-05-07
- Publication Date
- 2026-05-19
AI Technical Summary
Existing technologies for generating PPTs are inefficient during training. Base64 encoding increases the training time of the transformer model and causes modality mismatch, making it impossible to generate PPTs efficiently.
The encoder using the VQVAE model encodes the image to obtain the encoded ID, which is then filled into the image tag position of the HTML data. The HTML data is generated by combining the large language model. The model is trained using text and image loss functions, and the large language model and decoder are trained in stages.
It improves the efficiency of PPT generation training, ensures that the large language model focuses on learning complex language structures and semantic information, and the VQVAE decoder focuses on training image data, achieving high-quality separate training of text and images, thus improving generation efficiency and quality.
Smart Images
Figure CN120671641B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of machine learning, and in particular to a method, apparatus, and electronic device for training a demonstration document generation model. Background Technology
[0002] In existing technologies, during the training process for generating PPTs, the images involved are converted into ASCII characters based on Base64 encoding rules, and then the PPT text description is concatenated with the ASCII characters. The concatenated training samples are then processed by a transformer model to obtain the mapping relationship between the text description and the HTML code.
[0003] Base64 encoding encodes every 3 bytes of binary data into 4 ASCII characters, which increases the size of the image data. The Base64 encoding significantly increases the training time of the Transformer model. Furthermore, Base64 encoding disrupts the direct spatial relationships between pixels. Therefore, the Transformer model needs to reconstruct high-level semantic information from relatively low-level features (such as how a specific character sequence corresponds to a color gradient), expending considerable resources to learn how to map the Base64 encoded character sequences back to their corresponding pixel values in the image, and how to understand the relationships between these pixel values. This leads to a modal mismatch between the Base64 encoded signal representation and the Transformer model architecture, resulting in training focusing on image reconstruction rather than efficiently generating PPTs. Therefore, improving the efficiency of PPT generation training is a pressing issue. Summary of the Invention
[0004] This application provides a method, apparatus, and electronic device for training a presentation document generation model, in order to at least solve the problem of low efficiency in the PPT generation training process in related technologies.
[0005] In a first aspect, embodiments of this application provide a method for training a demonstration document generation model, the method being applied to a large language model and a VQVAE model, the VQVAE model including an encoder and a decoder, the method comprising:
[0006] Extract the embedded images from the training demonstration document, encode the embedded images using the encoder, and obtain the encoded ID of each embedded image;
[0007] In the initial HTML data obtained by converting the training demonstration document, the encoded ID is filled into the image tag position to obtain the target HTML data;
[0008] Obtain the descriptive text of the training demonstration document, generate HTML data based on the descriptive text using a large language model, wherein the image tag bits in the generated HTML data are the target encoding ID, and construct a text loss function based on the generated HTML data and the target HTML data;
[0009] The target encoded ID is decoded using the decoder to obtain a decoded image, and an image loss function is constructed based on the decoded image and the embedded image.
[0010] A training loss function is constructed based on the text loss function and the image loss function. The large language model and the decoder are trained until the training loss function reaches the convergence condition, thereby obtaining the demonstration document generation model.
[0011] In one embodiment, obtaining the descriptive text based on the training demonstration document includes:
[0012] Convert each slide of the training demonstration document into a page image;
[0013] Using a multimodal large model, single-page description text is generated based on each page image, and continuous page description text is generated based on all page images.
[0014] In one embodiment, the step of constructing a training loss function based on the text loss function and the image loss function, training the large language model and the decoder until the training loss function reaches the convergence condition, and obtaining the demonstration document generation model includes:
[0015] A first training loss function is constructed based on the single-page description text, and the large language model and the decoder are trained until the first training loss function converges, thereby obtaining the first model weights of the large language model and the second model weights of the encoder.
[0016] The large language model is updated based on the weights of the first model, and the encoder is updated based on the weights of the second model.
[0017] A second training loss function is constructed based on the continuous page text, and the updated decoder and the large language model are trained until the second training loss function converges, thus obtaining the demonstration document generation model.
[0018] In one embodiment, the step of constructing a first training loss function based on the single-page description text, training the large language model and the decoder until the first training loss function converges, and obtaining the first model weights of the large language model and the second model weights of the encoder includes:
[0019] Based on the single-page description text, single-page HTML data is obtained through the large language model. The image tag bit of the single-page HTML data is the first encoded ID. Based on the single-page HTML data and the target HTML data, a first text loss function is constructed.
[0020] The first encoded ID is decoded by the decoder to obtain a single-page decoded image, and a first image loss function is constructed based on the single-page decoded image and the embedded image.
[0021] The first training loss function is constructed based on the first text loss function and the first image loss function;
[0022] The large language model and the decoder are trained until the first training loss function reaches the convergence condition, thereby obtaining the first model weights of the large language model and the second model weights of the encoder.
[0023] In one embodiment, the step of constructing a second training loss function on the consecutive pages of text and training the updated decoder and the large language model until the second training loss function converges to obtain the demonstration document generation model includes:
[0024] Based on the continuous page description text, continuous page HTML data is obtained through the updated large language model. The image tag bits of the continuous page HTML data are the second encoding ID. A second text loss function is constructed based on the continuous page HTML data and the initial HTML data.
[0025] The second encoded ID is decoded by the updated decoder to obtain consecutive page decoded images, and a second image loss function is constructed based on the consecutive page decoded images and the embedded images;
[0026] The second training loss function is constructed based on the second text loss function and the second image loss function;
[0027] The updated decoder and the large language model are trained until the second training loss function reaches the convergence condition, thereby obtaining the demonstration document generation model.
[0028] In one embodiment, the large language model includes an embedding layer and a Transformer decoding layer, and the step of generating HTML data from the descriptive text using the large language model includes:
[0029] The descriptive text is segmented to obtain several tags, and all the tags are serialized to obtain the input sequence;
[0030] The input sequence is input into the embedding layer to obtain the embedding vector;
[0031] The embedding vector is processed by the Transformer decoding layer to update the hidden state layer by layer;
[0032] Based on the hidden state of the last layer in the Transformer decoding layer, the probability distribution of the tag is obtained, and the tag with the highest probability in the probability distribution is used as the generated tag;
[0033] The generated markup is added to the embedding vector to update the embedding vector. The embedding vector is continuously updated through the Transformer decoding layer until the termination condition is met. The finally updated embedding vector is used as the generated HTML data.
[0034] In one embodiment, the step of extracting embedded images from the training demonstration document and encoding the embedded images using the encoder to obtain the encoded ID of each embedded image includes:
[0035] Extract the embedded images from the training demonstration document and perform normalization preprocessing on the embedded images;
[0036] The processed embedded image is input into the encoder, which maps the continuous feature map of the processed embedded image to a preset codebook to obtain the encoded ID of each embedded image.
[0037] Secondly, embodiments of this application provide a method for generating a presentation document, the method being applied to a presentation document generation model of the first aspect, the presentation document generation model including a large language model and a VQVAE model decoder including:
[0038] The document generation instructions are input into the large language model in the demonstration document generation model to obtain HTML data, wherein the image tag bits in the HTML data are encoded IDs;
[0039] The decoder in the demonstration document generation model decodes the encoded ID to obtain the decoded image;
[0040] The HTML data is parsed to obtain the page content of the demo document, and the demo document is generated based on the page content and the decoded image.
[0041] Thirdly, embodiments of this application provide a demonstration document generation training apparatus, characterized in that it includes:
[0042] The encoding module is used to extract embedded images from the training demonstration document, and to encode the embedded images by the encoder to obtain the encoded ID of each embedded image;
[0043] The first obtaining module is used to fill the image tag position with the encoded ID in the initial HTML data obtained by converting the training demonstration document to obtain the target HTML data;
[0044] A text loss function module is constructed to obtain the descriptive text of the training demonstration document, generate HTML data based on the descriptive text using a large language model, fill the image tag bits in the generated HTML data with target encoding IDs, and construct a text loss function based on the target HTML data and the initial HTML data.
[0045] An image loss function construction module is used to decode the target encoding ID through the decoder to obtain the generated image, and construct an image loss function based on the generated image and the embedded image;
[0046] A demo document generation model module is obtained, which is used to construct a training loss function based on the text loss function and the image loss function, train the large language model and the decoder until the training loss function reaches the convergence condition, and obtain the demo document generation model.
[0047] Fourthly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the demonstration document generation model training method as described in the first aspect above.
[0048] Fifthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the demonstration document generation model training method as described in the first aspect above.
[0049] The demonstration document generation model training method, apparatus, and electronic device provided in this application embodiment have at least the following technical effects.
[0050] By training a demonstration document, initial HTML data and embedded images are obtained. The embedded images are then encoded using a VQVAE encoder to obtain encoded IDs, which preserves key image features and removes pixel redundancy, achieving efficient image compression. The encoded IDs are used to fill the image tag bits in the initial HTML data to obtain the target HTML data, ensuring that the target HTML data contains only text data. This allows the large language model to focus on complex language structures and semantic information during training, improving the quality of the generated HTML data. A text loss function is constructed based on the target HTML data and the generated HTML data, while an image loss function is constructed based on the embedded and decoded images. These are used to train the large language model and decoder, allowing the large language model to focus on text data training and the VQVAE decoder to focus on image data training. This separate training of text and images improves training efficiency.
[0051] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description
[0052] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0053] Figure 1 This is a flowchart illustrating a demonstration document generation model training method according to an exemplary embodiment;
[0054] Figure 2 This is a flowchart illustrating a method for generating a model training document according to another exemplary embodiment;
[0055] Figure 3 This is a block diagram illustrating a demonstration document generating training apparatus according to an exemplary embodiment;
[0056] Figure 4 This is a block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation
[0057] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.
[0058] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.
[0059] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.
[0060] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.
[0061] Firstly, embodiments of this application provide a method for training a demonstration document generation model. Figure 1 This is a flowchart illustrating a demonstration document generation model training method according to an exemplary embodiment, such as... Figure 1As shown, the demonstration document generation model training method is applied to a large language model and a VQVAE model. The VQVAE model includes an encoder and a decoder. The demonstration document generation model training method should include:
[0062] Step S101: Extract the embedded images from the training demonstration document, encode the embedded images using an encoder, and obtain the encoded ID of each embedded image.
[0063] Each slide in the training presentation document contains several embedded images. All embedded images are extracted from the training presentation document. All embedded images are then transmitted to the encoder to encode them, obtaining an encoded ID for each embedded image.
[0064] The methods for obtaining the encoded ID of each embedded image specifically include:
[0065] Step S111: Extract the embedded image from the training demonstration document, standardize and preprocess the embedded image to obtain the input image.
[0066] Each embedded image undergoes a standardization preprocessing step. This involves scaling each embedded image to a preset size to ensure all embedded images are the same size. Then, the pixel values in each embedded image are normalized to the range [-1, 1] to obtain the input image.
[0067] Step S112: Input the input image into the encoder, and use the encoder to map the continuous feature map of the processed embedded image to a preset codebook to obtain the encoding ID of each embedded image.
[0068] The input image is fed into the encoder of the VQVAE model. Through the encoder's internal network, the input image is transformed into a continuous feature map. This continuous feature map is a matrix-like feature map where each element is a continuous value.
[0069] For each vector in the continuous feature values, find the closest embedding vector in a pre-defined codebook and determine the index of the closest embedding vector, where the index represents the position of the embedding vector in the pre-defined codebook. Replace the corresponding vector with the index to obtain a discrete vector. The obtained discrete vector serves as the encoded ID obtained after each embedded image is processed by the encoder.
[0070] Continuing with step S101, the image encoding method described above using the VQVAE encoder transforms the originally complex continuous feature map into a series of discrete index values. These indices point to specific embedding vectors in a predefined codebook, thus achieving an effective mapping from continuous space to discrete space. The encoded ID obtained by encoding the image using the VQVAE encoder not only preserves the key semantic information of the embedded image but also possesses higher information density and better generalization ability, thereby reducing the computational burden of subsequent training processes.
[0071] Step S102: In the initial HTML data obtained by converting the training demonstration document, fill the image tag position with the encoded ID to obtain the target HTML data.
[0072] The training demonstration documents are converted into initial HTML data using a conversion tool. During this conversion, image tags in the initial HTML data are used as placeholders. By using placeholders instead of complex image encoding, lengthy image codes are avoided by directly embedding them into the initial HTML data, thus reducing the dataset size and training complexity.
[0073] The obtained encoded ID replaces the placeholders in the initial HTML data, thus filling the image tag positions with the encoded ID and obtaining the target HTML data. In other words, in the target HTML data, the image tag positions are the encoded IDs.
[0074] Since the encoded ID is obtained by the VQVAE encoder, it not only contains the key semantic information of the embedded image but also has a higher information density, resulting in efficient compression of the embedded image. Therefore, the target HTML data contains high-dimensional image information, and this high-dimensional image information occupies very little storage space in the first HTML, avoiding the simultaneous presence of large amounts of image and text information in the target HTML data.
[0075] Step S103: Obtain the description text of the training demonstration document, generate HTML data based on the description text using the large language model, use the image tag bits in the generated HTML data as the target encoding ID, and construct a text loss function based on the generated HTML data and the target HTML data.
[0076] Obtain the descriptive text of the training presentation document. This descriptive text describes the content of the slides in the training presentation document, including the text content on the slides and the position of each element. Segment and serialize the descriptive text using a tokenizer to obtain the input sequence. Input the input sequence into the embedding layer of the large language model to obtain embedding vectors, which are then used as the initial hidden states.
[0077] The embedding vector is input into the Transformer decoding layer of the large language model. Through multi-layered self-attention mechanisms and feedforward neural networks, the hidden state of each token is updated in each Transformer decoding layer. This process continues until the hidden state in the final Transformer decoding layer is reached, obtaining the probability distribution of each token. The token with the highest probability in the probability distribution is used as the generated token and added to the embedding vector to update it. The updated embedding vector is then processed again by the Transformer decoding layer to continuously update it until a termination condition is met, obtaining the final embedding vector. This final embedding vector is then fed back into the word segmenter to convert it into HTML data, resulting in the generated HTML data. Optionally, the termination condition includes reaching a preset maximum sequence length or encountering a preset end marker.
[0078] In the generated HTML data, the image tag bits are used as the target encoding ID. The target encoding ID is the encoding ID corresponding to the VQVAE encoder generated by the large language model. A text loss function is then constructed based on the generated HTML data and the target HTML data. The text loss function is as follows:
[0079]
[0080] L text It is a text loss function, y t Let y be the t-th text, and y < t be the sequence of texts preceding the t-th text. Let P(y) be the text preceding the t-th text. t |y ＜t Given a sequence of texts preceding the t-th text, the model predicts the probability of the t-th text, where T is the length of the sequence that generates the HTML data.
[0081] The Large Language Model (MLM) generates HTML data by using image labels as target encoding IDs from the description text. By constructing a text loss function, the generated HTML data closely approximates the target HTML data. Since the encoding ID is an efficient compression of the embedded image, the MLM, when generating HTML data, no longer devotes significant resources to reconstructing images from complex image encodings. Instead, it generates target encoding IDs similar to the encoding IDs, allowing the MLM to allocate more learning resources to capturing the complex linguistic structures and semantic information within the description text. This improves the accuracy and coherence of text generation, resulting in high-quality HTML content.
[0082] Step S104: Decode the target encoding ID using a decoder to obtain the decoded image, and construct an image loss function based on the decoded image and the embedded image.
[0083] The output of the large language model is connected to the decoder of VQVAE. The target encoding ID in the generated HTML data from the large language model is input into the decoder. The decoder decodes the target encoding ID to obtain the decoded image. An image loss function is constructed based on the decoded image and the embedded image. Specifically, the image loss function is as follows:
[0084]
[0085] Among them, L image It is the image loss function, where N is the total number of pixels in the embedded image and the decoded image, and the total number of pixels in the embedded image and the decoded image is the same. i It is the value of the i-th pixel in the embedded image. It is the value of the i-th pixel in the decoded image.
[0086] The VQVAE decoder can decode into a high-quality image based on the target encoded ID. The VQVAE decoder focuses only on generating the image, and the target encoded ID is close to the encoded ID of efficient compression. This fully utilizes the characteristics of the VQVAE model, thereby ensuring the high-fidelity requirement of the image and reducing the computational burden.
[0087] Step S105: Construct a training loss function based on the text loss function and the image loss function, train the large language model and decoder until the training loss function reaches the convergence condition, and obtain the demonstration document generation model.
[0088] A training loss function is constructed based on text and image loss functions. A large language model and decoder are trained, and their internal parameters are continuously updated during training until the training loss function converges, resulting in a presentation document generation model capable of generating presentation documents based on this model. The specific training loss function is as follows:
[0089] L total =L text +λ·L image
[0090] Among them, L total It is the training loss function, L text It is the text loss function, L image λ is the image loss function, and λ is a hyperparameter that balances the loss weights of text and image.
[0091] In one embodiment, training the large language model and decoder is divided into two phases: a single-page training phase and a continuous-page training phase. The descriptive text used in the training process includes both single-page and continuous-page descriptive text, which are obtained as follows:
[0092] Each slide of the training presentation document is converted into a page image. These page images are then input into a multimodal large-scale model. Leveraging the model's joint visual and language understanding capabilities, single-page descriptive text is generated for each page image, and continuous-page descriptive text is generated from all page images. Single-page descriptive text describes the content of a single slide, including text and the position of each element, and is used in the single-page training phase. Continuous-page descriptive text describes the content of several consecutive slides and is used in the continuous-page training phase. The difference between single-page and continuous-page descriptive text is that single-page descriptive text describes only a single slide, while continuous-page descriptive text describes all consecutive slides within the presentation document.
[0093] Continuing with step S105, training the large language model and encoder is divided into two stages, which are as follows:
[0094] Step S511: Construct a first training loss function based on the single-page description text, and train the large language model and decoder until the first training loss function converges, and obtain the first model weights of the large language model and the second model weights of the encoder.
[0095] The large language model obtains single-page HTML data from the single-page descriptive text, where the image tag bits in the single-page HTML data serve as the first encoded ID. A first text loss function is constructed based on the single-page HTML data and the target HTML data. The decoder decodes the first encoded ID to obtain the single-page decoded image, and a first image loss function is constructed based on the single-page decoded image and the embedded image. A first training loss function is constructed using the first text loss function and the first image loss function. The large language model and decoder are trained until the first training loss function converges, obtaining the first model weights of the large language model and the second model weights of the encoder.
[0096] It should be noted that the first model weights of the large language model and the second model weights of the encoder represent the connection strength between nodes in different layers within the model, reflecting the degree of influence of input features on the output. When the first training loss function reaches the convergence condition, all relevant parameters within the large language model and decoder reach their optimal state.
[0097] Step S512: Update the large language model based on the weights of the first model, and update the encoder based on the weights of the second model.
[0098] The weights of the first model obtained in the single-page training phase are used to update the weights of the large language model and the second model to update the encoder, so that the two models can obtain the training results of the single-page training phase and converge faster in the second training process.
[0099] Step S513: Construct a second training loss function based on continuous page text, and train the updated decoder and large language model until the second training loss function converges to obtain the demonstration document generation model.
[0100] The updated large language model obtains consecutive page HTML data based on the consecutive page description text, where the image tag bits in the consecutive page HTML data serve as the second encoded ID. A first text loss function is constructed based on the consecutive page HTML data and the target HTML data. The decoder decodes the second encoded ID to obtain the consecutive page decoded images, and a second image loss function is constructed based on the consecutive page decoded images and the embedded images. A second training loss function is constructed using the second text loss function and the second image loss function. The large language model and decoder are trained until the second training loss function reaches convergence, resulting in a demonstration document generation model.
[0101] The updated large language model and decoder have the results of the single-page training phase. Based on this, the large language model and decoder will not only converge faster, but will also further refine the generated content based on the single-page training phase. This will make the generated continuous page HTML data and continuous page decoded images more accurate, thereby improving the quality of the generated HTML data and images.
[0102] Continuing with steps S511 to S513, by training the large language model and decoder in stages, the two models can progressively refine the generated content, making the generated content closer to the reference data in the training data, thereby ensuring high-quality output content.
[0103] In another embodiment, before extracting the embedded images from the training demonstration documents in step S101, it is also necessary to screen the training demonstration documents used to train the model. The specific methods for screening the training demonstration documents include:
[0104] Acquire several demonstration documents and categorize them based on keywords to ensure that the demonstration documents used for training cover multiple fields. For example, the types of demonstration documents include business reports, educational courseware, cultural promotion, product demonstrations, and event planning.
[0105] After initial screening of the presentation documents, further complexity filtering is needed to ensure the use of relatively simple content, thereby reducing complexity during training. Optionally, presentation documents where all elements in the slides occupy more than 50% of the total page area should be retained to avoid adding too many elements to the training data, which would increase training time and complexity.
[0106] Some demonstration documents are commercial. To avoid copyright disputes, copyright cleanup is necessary to remove the commercial demonstration documents. By extracting open-source license identifiers such as CC-BY and MIT from the demonstration documents, it is determined that the demonstration documents used for training are open-source demonstration documents, thus removing the commercial demonstration documents and ensuring that the demonstration documents used for training are open-source demonstration documents.
[0107] In another embodiment, Figure 2 This is a flowchart illustrating a method for generating a model training document according to another exemplary embodiment, such as... Figure 2 As shown, the original presentation document is processed by a preprocessing module to obtain a training presentation document. The slides in the training presentation document are converted into PNG format page images, and the page images are then converted into descriptive text. This descriptive text includes single-page descriptive text and continuous-page descriptive text. A conversion tool is used to convert the training presentation text into HTML data. This HTML data includes single-page HTML data and continuous-page HTML data, with each HTML data entry containing an image tag bit named vqid (i.e., the encoded ID). Single-page HTML data and single-page descriptive text are used to construct a single-page training set, and continuous-page HTML data and continuous-page descriptive text are used to construct a multi-page training set. Based on these two training sets, the large language model and decoder undergo two stages of training. After both stages of training are completed, a presentation document generation model is obtained. In this model, the large language model generates HTML data containing vqid, the VQVAE decoder decodes the vqid to generate images, parses the HTML data to obtain page content, and combines the page content and generated images to obtain the presentation document.
[0108] The demonstration documents were selected using the methods described above to determine the final demonstration documents to be used for training. These demonstration documents, of various types, provided rich training data for the training process, and this training data did not include too many complex elements, reducing the complexity of the training data and avoiding copyright disputes.
[0109] In summary, the demonstration document generation model training method provided in this application, during the training process, encodes images using a VQVAE encoder to achieve efficient image compression, thereby reducing the computational burden and significantly decreasing the computational resources required to process image data, thus improving the overall system efficiency. The VQVAE decoder can directly convert the encoded IDs in the generated HTML code into high-quality image data, avoiding the need for a large language model to learn image generation from lengthy encodings. This improves the training efficiency of the model from the perspective of network structure design. The large language model generates HTML data containing encoded IDs based on the descriptive text, allowing it to focus on more accurately capturing complex language structures and semantic information in the text, thereby generating high-quality HTML content. By processing text and images separately and fully leveraging the advantages of both models, image quality and text coherence are ensured, thus improving training efficiency. Training the large language model and decoder in stages improves the quality of the generated content. Furthermore, processing images and text separately gives the demonstration document generation model greater flexibility and modularity, allowing for independent adjustment or upgrading of individual components according to specific needs, facilitating maintenance and expansion.
[0110] Secondly, embodiments of this application provide a method for generating a demonstration document, the method comprising:
[0111] Input the document generation instructions into the large language model in the demo document generation model to obtain HTML data, where the image tag bits in the HTML data are encoded IDs.
[0112] The decoder in the model is generated using the demo document to decode the encoded ID, thus obtaining the decoded image.
[0113] Parse HTML data to obtain the page content of the presentation document, and generate the presentation document based on the page content and decoded images.
[0114] The document generation instruction is the content that the user wants the generated presentation document to contain. Therefore, the user inputs the document generation instruction into the presentation document generation model. The instruction is processed by a large language model to generate HTML data, in which image tags serve as encoded IDs. The decoder decodes the encoded IDs to obtain the decoded images. After parsing the HTML data, the page content of each slide in the presentation document is obtained, and based on the page content and the decoded images, the presentation document is generated.
[0115] The demo document generation model can obtain the demo documents that users expect, improving the ease of obtaining demo documents.
[0116] Thirdly, embodiments of this application provide a demonstration document generation training device. Figure 3 This is a block diagram illustrating a demonstration document for generating a training apparatus, as shown in an exemplary embodiment. Figure 3 As shown, the demonstration document generation training device includes: an encoding module, used to extract embedded images from the training demonstration document, and to encode the embedded images by the encoder to obtain the encoded ID of each embedded image;
[0117] The first obtaining module is used to fill the image tag position with the encoded ID in the initial HTML data obtained by converting the training demonstration document to obtain the target HTML data;
[0118] A text loss function module is constructed to obtain the descriptive text of the training demonstration document, generate HTML data based on the descriptive text using a large language model, fill the image tag bits in the generated HTML data with target encoding IDs, and construct a text loss function based on the target HTML data.
[0119] An image loss function construction module is used to decode the target encoding ID through the decoder to obtain a decoded image, and to construct an image loss function based on the generated image and the embedded image;
[0120] A demo document generation model module is obtained, which is used to construct a training loss function based on the text loss function and the image loss function, train the large language model and the decoder until the training loss function reaches the convergence condition, and obtain the demo document generation model.
[0121] In summary, the demonstration document generation training apparatus provided in this application obtains initial HTML data and embedded images by training demonstration documents. The embedded images are encoded using a VQVAE encoder to obtain an encoding ID, which preserves key image features and removes pixel redundancy, achieving efficient image compression. The encoding ID is used to fill the image tag bits in the initial HTML data to obtain the target HTML data, ensuring that the target HTML data contains only text data. This allows the large language model to focus on complex language structures and semantic information during training, improving the quality of the generated HTML data. By constructing a text loss function based on the target HTML data and the generated HTML data, and an image loss function based on the embedded and decoded images, the large language model and decoder are trained. This allows the large language model to focus on text data training, while the VQVAE decoder focuses on image data training, achieving separate training for text and images, thereby improving training efficiency.
[0122] It should be noted that the demonstration document generation training device provided in this embodiment is used to implement the above-described embodiments, and details already described will not be repeated. As used above, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the above embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.
[0123] Fourthly, embodiments of this application provide an electronic device, Figure 4 This is a block diagram illustrating an electronic device according to an exemplary embodiment. (e.g.) Figure 4 As shown, the electronic device may include a processor 81 and a memory 82 storing computer program instructions.
[0124] Specifically, the processor 81 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.
[0125] The memory 82 may include a mass storage device for data or instructions. For example, and not limitingly, the memory 82 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 82 may include removable or non-removable (or fixed) media. Where appropriate, the memory 82 may be internal or external to a data processing device. In a particular embodiment, the memory 82 is non-volatile memory. In a particular embodiment, the memory 82 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.
[0126] The memory 82 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 81.
[0127] The processor 81 implements any of the demonstration document generation training methods in the above embodiments by reading and executing computer program instructions stored in the memory 82.
[0128] In one embodiment, the demonstration document generation training device may further include a communication interface 83 and a bus 80. Wherein, as... Figure 4 As shown, the processor 81, memory 82, and communication interface 83 are connected through bus 80 and complete communication with each other.
[0129] The communication interface 83 is used to enable communication between the various modules, devices, units, and / or equipment in the embodiments of this application. The communication port 83 can also enable data communication with other components such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations.
[0130] Bus 80 includes hardware, software, or both, that couples components of the demonstration document generation training device together. Bus 80 includes, but is not limited to, at least one of the following: data bus, address bus, control bus, expansion bus, and local bus. For example, and not as a limitation, bus 80 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 80 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.
[0131] Fifthly, embodiments of this application provide a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the demonstration document generation training method provided in the first aspect.
[0132] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.
[0133] In a possible implementation, the present invention can also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform steps implementing the demonstration document generation training method provided in the first aspect.
[0134] The program code for executing the present invention can be written in any combination of one or more programming languages. The program code can be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on a remote device.
[0135] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0136] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A method for training a demonstration document generation model, characterized in that, The method is applied to large language models and VQVAE models, wherein the VQVAE model includes an encoder and a decoder, and the method includes: Extract the embedded images from the training demonstration document, encode the embedded images using the encoder, and obtain the encoded ID of each embedded image; In the initial HTML data obtained by converting the training demonstration document, the encoded ID is filled into the image tag position to obtain the target HTML data; The descriptive text of the training demonstration document is obtained, and HTML data is generated based on the descriptive text using a large language model. The image tag bits in the generated HTML data are the target encoding IDs, and a text loss function is constructed based on the generated HTML data and the target HTML data. Obtaining the descriptive text based on the training demonstration document includes: converting each slide of the training demonstration document into a page image; generating single-page descriptive text based on each page image using a multimodal large model; and generating continuous page descriptive text based on all page images. The target encoded ID is decoded using the decoder to obtain a decoded image, and an image loss function is constructed based on the decoded image and the embedded image. The process involves constructing a training loss function based on the text loss function and the image loss function, training the large language model and the decoder until the training loss function converges, and obtaining a demonstration document generation model. This includes: constructing a first training loss function based on the single-page description text, and training the large language model and the decoder until the first training loss function converges; obtaining first model weights for the large language model and second model weights for the encoder; updating the large language model based on the first model weights, and updating the encoder based on the second model weights; constructing a second training loss function based on the continuous-page description text, and training the updated decoder and the large language model until the second training loss function converges, thus obtaining the demonstration document generation model.
2. The demonstration document generation model training method according to claim 1, characterized in that, The step of constructing a first training loss function based on the single-page description text, and training the large language model and the decoder until the first training loss function converges, to obtain the first model weights of the large language model and the second model weights of the encoder, includes: Based on the single-page descriptive text, single-page HTML data is obtained through the large language model. The image tag bits of the single-page HTML data are the first encoded ID. A first text loss function is constructed based on the single-page HTML data and the target HTML data. The first encoded ID is decoded by the decoder to obtain a single-page decoded image, and a first image loss function is constructed based on the single-page decoded image and the embedded image. The first training loss function is constructed based on the first text loss function and the first image loss function; The large language model and the decoder are trained until the first training loss function reaches the convergence condition, thereby obtaining the first model weights of the large language model and the second model weights of the encoder.
3. The demonstration document generation model training method according to claim 1, characterized in that, The process of constructing a second training loss function based on continuous page description text, and training the updated decoder and the large language model until the second training loss function converges to obtain the demonstration document generation model includes: Based on the continuous page description text, continuous page HTML data is obtained through the updated large language model. The image tag bits of the continuous page HTML data are the second encoding ID. A second text loss function is constructed based on the continuous page HTML data and the initial HTML data. The second encoded ID is decoded by the updated decoder to obtain consecutive page decoded images, and a second image loss function is constructed based on the consecutive page decoded images and the embedded images; The second training loss function is constructed based on the second text loss function and the second image loss function; The updated decoder and the large language model are trained until the second training loss function reaches the convergence condition, thus obtaining the demonstration document generation model.
4. The demonstration document generation model training method according to claim 1, characterized in that, The large language model includes an embedding layer and a Transformer decoding layer. The step of generating HTML data from the descriptive text using the large language model includes: The descriptive text is segmented to obtain several tags, and all the tags are serialized to obtain the input sequence; The input sequence is input into the embedding layer to obtain the embedding vector; The embedding vector is processed by the Transformer decoding layer to update the hidden state layer by layer; Based on the hidden state of the last layer in the Transformer decoding layer, the probability distribution of the tag is obtained, and the tag with the highest probability in the probability distribution is used as the generated tag; The generated tags are added to the embedding vector to update the embedding vector. The embedding vector is continuously updated through the Transformer decoding layer until the termination condition is met, and the generated HTML data is obtained based on the finally updated embedding vector.
5. The demonstration document generation model training method according to claim 1, characterized in that, The process of extracting embedded images from the training demonstration document and encoding these embedded images using the encoder to obtain the encoded ID of each embedded image includes: Extract the embedded image from the training demonstration document, standardize and preprocess the embedded image to obtain the input image; The input image is input into the encoder, and the encoder maps the continuous feature map of the processed embedded image to a preset codebook to obtain the encoded ID of each embedded image.
6. A method for generating a presentation document, characterized in that, The method is applied to a presentation document generation model obtained by the presentation document generation model training method of any one of claims 1 to 5, wherein the presentation document generation model includes decoders for a large language model and a VQVAE model, comprising: The document generation instructions are input into the large language model in the demonstration document generation model to obtain HTML data, wherein the image tag bits in the HTML data are encoded IDs; The decoder in the demonstration document generation model decodes the encoded ID to obtain the decoded image; The HTML data is parsed to obtain the page content of the demo document, and the demo document is generated based on the page content and the decoded image.
7. A demonstration document generation training device, characterized in that, include: The encoding module is used to extract embedded images from the training demonstration document, and to encode the embedded images by an encoder to obtain the encoded ID of each embedded image; The first obtaining module is used to fill the image tag position with the encoded ID in the initial HTML data obtained by converting the training demonstration document to obtain the target HTML data; A text loss function module is constructed to obtain the descriptive text of the training demonstration document, generate HTML data based on the descriptive text using a large language model, wherein the image tag bits in the generated HTML data are the target encoding IDs, and a text loss function is constructed based on the generated HTML data and the target HTML data; wherein, based on the training demonstration document, the descriptive text is obtained for: converting each slide of the training demonstration document into a page image; generating single-page descriptive text based on each page image using a multimodal large model, and generating continuous page descriptive text based on all page images; An image loss function construction module is used to decode the target encoding ID through a decoder to obtain a generated image, and to construct an image loss function based on the generated image and the embedded image; A demo document generation model module is used to construct a training loss function based on the text loss function and the image loss function, train the large language model and the decoder until the training loss function reaches the convergence condition, and obtain the demo document generation model. Specifically, it is used to: construct a first training loss function based on the single-page description text, and train the large language model and the decoder until the first training loss function converges, and obtain the first model weights of the large language model and the second model weights of the encoder; update the large language model based on the first model weights, and update the encoder based on the second model weights; construct a second training loss function based on the continuous-page description text, and train the updated decoder and the large language model until the second training loss function converges, and obtain the demo document generation model.
8. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor, when executing the computer program, implements the demonstration document generation model training method as described in any one of claims 1 to 5.