Model training method and device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a model training method and device, equipment and a medium, and the method comprises the steps: generating a plurality of groups of first positive and negative sample pairs based on a plurality of first image data and a plurality of second image data, and generating a plurality of groups of second positive and negative sample pairs based on a plurality of first image data and a plurality of first text data; training the first image encoder based on the plurality of groups of first positive sample pairs and the plurality of groups of first negative sample pairs to obtain a second image encoder; based on the multiple groups of second positive sample pairs and the multiple groups of second negative sample pairs, training the second image encoder and the first large language model to obtain a third image encoder and a second large language model; and based on the multiple groups of second positive sample pairs, the multiple groups of second negative sample pairs, a third image encoder and a third large language model, training the second large language model to obtain a speech generation model. Therefore, the finally generated lecture generation model can unify the format of the slides and understand the formats in the form of pictures.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and natural language processing technology, and in particular to a model training method, device, equipment and medium. Background Art

[0002] Generating a speech from slide content is of great significance for personal speeches in formal occasions. The existing speech generation model extracts and recognizes the text content of the slide, and then generates relevant speech content from the read content text. However, due to the layout of the slide, the speech generation model has relatively poor understanding of some special layout texts, and may not be able to accurately recognize and extract text content, resulting in the omission of important content in the generated speech, affecting the accuracy and completeness of the speech. Summary of the invention

[0003] The present invention provides a model training method, device, computer equipment and medium to solve the technical problem that the speech generation model has relatively poor understanding of some specially formatted texts and may not be able to accurately identify and extract text content, resulting in the omission of important content in the generated speech, affecting the accuracy and completeness of the speech.

[0004] In a first aspect, a model training method is provided, comprising:

[0005] Acquire a plurality of first image data and a plurality of first text data and a plurality of second image data corresponding thereto;

[0006] Based on the plurality of first image data and the plurality of second image data, a plurality of groups of first positive sample pairs and a plurality of groups of first negative sample pairs are generated, and based on the plurality of first image data and the plurality of first text data, a plurality of groups of second positive sample pairs and a plurality of groups of second negative sample pairs are generated;

[0007] Based on the multiple groups of first positive sample pairs and the multiple groups of first negative sample pairs, the first image encoder is trained to obtain a second image encoder;

[0008] Based on the multiple groups of second positive sample pairs and the multiple groups of second negative sample pairs, the second image encoder and the first large language model are trained to obtain a third image encoder and a second large language model;

[0009] Copy the second largest language model to get the third largest language model;

[0010] Based on multiple groups of second positive sample pairs, multiple groups of second negative sample pairs, a third image encoder and a third language model, the second language model is trained to obtain a speech generation model.

[0011] In a second aspect, a model training device is provided, comprising:

[0012] An acquisition module, used for acquiring a plurality of first image data and a plurality of first text data and a plurality of second image data corresponding thereto;

[0013] A first generating module, configured to generate a plurality of first positive sample pairs and a plurality of first negative sample pairs based on a plurality of first image data and a plurality of second image data, and to generate a plurality of second positive sample pairs and a plurality of second negative sample pairs based on a plurality of first image data and a plurality of first text data;

[0014] A training module, configured to train the first image encoder based on the plurality of groups of first positive sample pairs and the plurality of groups of first negative sample pairs to obtain a second image encoder;

[0015] The training module is also used to train the second image encoder and the first large language model based on multiple groups of second positive sample pairs and multiple groups of second negative sample pairs to obtain a third image encoder and a second large language model;

[0016] A second generation module is used to copy the second largest language model to obtain a third largest language model;

[0017] The training module is also used to train the second largest language model based on multiple groups of second positive sample pairs, multiple groups of second negative sample pairs, a third image encoder and a third largest language model to obtain a speech generation model.

[0018] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned model training method when executing the computer program.

[0019] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned model training method are implemented.

[0020] In the scheme implemented by the above-mentioned model training method, device, computer equipment and storage medium, the speech generation model for explaining slides is trained through three stages. In the first stage, when training the image encoder, the focus is on improving the visual features, avoiding the interference of the language model when processing image information, so that the model can learn more attentively. In the second stage, the text features and image features are aligned and mapped to the same representation space, so that the model can understand the association between images and texts. Joint training can enhance the synergy between the image encoder and the language model, and promote the understanding of visual features and language context. In the third stage, the performance of the language model in the speech generation task is further refined, so that it can generate more natural and logically clear text based on the processed visual information, so that the model can better adapt to different topics and improve flexibility and accuracy. Through three stages of gradual training, the final speech generation model can unify the format of the slides in the form of pictures. The picture format retains the visual content in the slides (such as icons, illustrations, and layouts), eliminates garbled characters and typographical errors that may be caused by different text formats, and ensures the consistency of information transmission. The encoding model is then used to understand the pictures and user intentions, extract key information and features, and enhance the model's ability to understand visual information. Finally, by understanding the combination of picture content and user intentions, the generative model can automatically generate more targeted speech content, making the speech more professional and organized. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0022] Figure 1 is a schematic diagram of an application environment of a model training method in one embodiment of the present invention;

[0023] Figure 2 is a flow chart of a model training method in one embodiment of the present invention;

[0024] Figure 3 yes Figure 2 A schematic flow chart of a specific implementation of step S30;

[0025] Figure 4 yes Figure 2 A schematic flow chart of a specific implementation of step S60;

[0026] Figure 5 is a structural schematic diagram of a model training device in one embodiment of the present invention;

[0027] Figure 6 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0028] Figure 7 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] The model training method provided by the embodiment of the present invention can be applied in Figure 1 In an application environment, the client communicates with the server through a network. The server can receive multiple first image data and multiple first text data corresponding to them through the client; based on the multiple first image data, generate multiple second image data and multiple third image data; based on the multiple second image data and multiple third image data, train the first image encoder to obtain the second image encoder; based on the multiple first image data and multiple first text data, train the second image encoder and the first large language model to obtain the third image encoder and the second large language model; copy the second large language model to obtain the third large language model; based on the multiple first image data, the multiple first text data, the third image encoder and the second large language model, train the second large language model to obtain the speech generation model. In the present invention, the model framework for generating slide speeches can be trained through three stages. The first stage is to strengthen the image encoder, that is, the ability to understand picture features; the second stage is to align the text features and picture features and map them to the same representation space, so that the model can understand the association between pictures and texts; the third stage is to use picture features and artificial intelligence to understand and finally generate a speech generation model, so that the speech generation model can understand the format of the slide in a unified picture form, and extract information by understanding the picture and user intention, so as to generate a slide speech that better meets the needs. Among them, the client can be but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0031] See also Figure 2 As shown, Figure 2A flow chart of a model training method provided in an embodiment of the present invention includes the following steps:

[0032] S10: Acquire a plurality of first image data and a plurality of first text data and a plurality of second image data corresponding thereto.

[0033] The model training method provided by the present invention can be applied to the scenario of training a speech generation model for generating a speech from slide content, and the speech generation model is intended to automatically generate a speech corresponding to the slide (PPT) content. The first image data is the collected slide images, the first text data is the speech text written by the relevant personnel based on each slide image, and the second image data is the image data after data enhancement processing (such as image flipping, rotation or noise addition) is performed on the first image data.

[0034] For example, the first image data can be advertising planning PPT, internal corporate training PPT, annual report PPT, various industry conferences, exhibitions or press conferences PPT, etc. By collecting PPT images of different themes, styles and contents involved in different business scenarios for learning, the diversity and comprehensive understanding ability of the model are improved, so that the model can better understand and generate speeches for different business scenarios, thereby improving the quality of text generation.

[0035] In one embodiment of the present application, step S10, that is, obtaining a plurality of first image data and a plurality of first text data and a plurality of second image data corresponding thereto, includes the following steps:

[0036] S11: Acquire a plurality of first image data and a plurality of first text data;

[0037] S12: performing data enhancement processing on each first image data to generate second image data corresponding to each first image data;

[0038] The data enhancement processing includes at least one of the following: image flipping, image rotation and image noise addition.

[0039] For steps S11-S12, after receiving a large number of slide images (i.e., first image data) and manually written speech texts corresponding to a large number of slide images, the large number of slide images are subjected to image enhancement processing, such as flipping the image up and down or horizontally, rotating the image, or performing Gaussian noise addition on the image, to obtain second image data corresponding to each first image data. By data enhancement, the diversity of training data is increased, so that the model can better cope with noise and erroneous information that occurs in practical applications, and improve its adaptability to new data.

[0040] For example, we receive a large number of PPT images of advertising planning activities, as well as the text of speeches that are manually annotated for each PPT image. Afterwards, we perform data enhancement processing on each PPT image.

[0041] S20: Generate multiple groups of first positive sample pairs and multiple groups of first negative sample pairs based on multiple first image data and multiple second image data, and generate multiple groups of second positive sample pairs and multiple groups of second negative sample pairs based on multiple first image data and multiple first text data.

[0042] After data enhancement is performed on a large number of original slide images, multiple groups of first positive and negative sample pairs are formed from multiple first image data and multiple second image data. By constructing the first positive and negative sample pairs, the model can better understand the similarities between images of the same type and promote the model to recognize the differences between images of different types, thereby improving the generalization ability and the classification accuracy.

[0043] Furthermore, multiple groups of second positive and negative sample pairs are formed by multiple first image data and multiple first text data. By constructing the second positive and negative sample pairs, the model is helped to learn the direct relationship between the image and the text, while at the same time, the model is prompted to understand what is irrelevant content, so as to better distinguish valid information.

[0044] In one embodiment of the present application, a specific scheme for constructing positive and negative sample pairs is provided. In S20, multiple groups of first positive sample pairs and multiple groups of first negative sample pairs are generated based on multiple first image data and multiple second image data, and multiple groups of second positive sample pairs and multiple groups of second negative sample pairs are generated based on multiple first image data and multiple first text data, which specifically includes the following steps S21-S26:

[0045] S21: forming a set of first positive sample pairs through each first image data and its corresponding second image data;

[0046] S22: forming a set of first negative sample pairs by using each first image data and any other first image data or any other second image data;

[0047] The other second image data is image data other than the image data corresponding to the first image data.

[0048] For steps S21-S22, each first image data and its corresponding second image data form a set of first positive sample pairs, and each original slide image and its related enhanced image form a positive sample pair, so that the model can learn deeper features. At the same time, each first image data and second image data unrelated to the first image data or other first image data other than the first image data form a set of first negative sample pairs, and each original slide image and unrelated enhanced image form a negative sample pair, so that the model can recognize the differences between different types of images, so that the model can better learn the ability to distinguish different types and different features.

[0049] By effectively constructing the first positive and negative sample pairs, when faced with uncertain or noisy data, the model can improve its ability to recognize and process adversarial samples through training on these positive and negative sample pairs, thereby improving classification accuracy.

[0050] S23: forming a set of second positive sample pairs through each first image data and its corresponding first text data;

[0051] S24: forming a set of second negative sample pairs through each first image data and any other first text data;

[0052] The other first text data is text data other than the text data corresponding to the first image data.

[0053] For steps S23-S24, each first image data and its corresponding first text data form a set of second positive sample pairs, and each slide image and its corresponding manually written speech text form a positive sample pair, which helps the model learn the direct relationship between the image and the text, and helps the model form a clearer feature representation during the training process. At the same time, each first image data and other first text data except the corresponding text form a set of second negative sample pairs, and each slide image and irrelevant speech text form a negative sample pair, which prompts the model to understand what is irrelevant content, so as to better distinguish effective information.

[0054] By constructing the second positive and negative sample pairs, the model can deeply understand the semantics conveyed by the content of different slides, so that the model can more accurately generate corresponding speech scripts based on the content of the slides.

[0055] S25 . Preprocess the image data included in the multiple groups of first positive sample pairs and the multiple groups of first negative sample pairs based on a preset splicing rule to form image block data corresponding to each image data.

[0056] Before model training, all images need to be formatted and preprocessed to form image block data corresponding to each image data through resizing, segmentation and splicing. Ensure that all image data have the same input dimension to facilitate model processing.

[0057] In one embodiment of the present application, in step S25, that is, based on a preset splicing rule, preprocessing the image data contained in the plurality of groups of first positive sample pairs and the plurality of groups of first negative sample pairs to form image block data corresponding to each image data includes the following steps:

[0058] S251: For any image data, resize the image data according to a first preset size;

[0059] S252: Cutting the image data after the size adjustment according to preset cutting parameters to obtain multiple image blocks;

[0060] S253: resizing the resized image data again according to a second preset size;

[0061] S254: splicing the image data after resizing with a plurality of image blocks to form image block data.

[0062] For steps S251-S254, for any image data, the size of the image is uniformly adjusted to a first preset size. Thereafter, the image of the first preset size is divided into a plurality of image blocks of uniform size according to preset cutting parameters without overlapping pixels. The resized image is downsampled to a second preset size, the cut image blocks are tiled, and the resized image is placed at the front of the tiled image blocks to form image block data.

[0063] Optionally, the first preset size may be 64*64; the preset cutting parameters include the number of cutting blocks (such as 16 blocks) and the size of each image block (such as 4*4); and the second preset size may be 4*4.

[0064] S26: labeling the sample data included in the multiple groups of first positive sample pairs and the multiple groups of second positive sample pairs with a first binary label, and labeling the sample data included in the multiple groups of first negative samples and the multiple groups of second negative sample pairs with a second binary label.

[0065] The positive sample pairs are marked with the first binary label, and the negative sample pairs are marked with the second binary label. By using binary labels, the model can clearly distinguish between positive samples (relevant) and negative samples (irrelevant), so as to better identify and distinguish the two types of samples, which helps to form a clear learning goal during the training process. At the same time, it is also convenient to calculate the loss function.

[0066] For example, the label of a positive sample pair is marked as 1, and the label of a negative sample pair is marked as 0.

[0067] S30: Based on the multiple groups of first positive sample pairs and the multiple groups of first negative sample pairs, training the first image encoder to obtain a second image encoder;

[0068] The image encoder is used to provide an efficient representation for image processing and recognition tasks. By training positive and negative sample pairs, the trained second image encoder can more effectively learn important features in the image, thereby improving its performance in image recognition and classification tasks.

[0069] Optionally, the image encoder may be a ViT (Vision Transformer) model.

[0070] In one embodiment of the present application, Figure 3 As shown, a specific method of using a self-supervised image encoder to train an image feature extraction scheme is provided. In S30, the first image encoder is trained based on multiple groups of first positive sample pairs and multiple groups of first negative sample pairs to obtain a second image encoder, which specifically includes the following steps S31-S35:

[0071] S31: inputting a plurality of groups of first positive sample pairs and a plurality of groups of first negative sample pairs as training data into a first image encoder to obtain a plurality of first output vectors output by the first image encoder;

[0072] S32: determining a plurality of target feature vectors from among the plurality of first output vectors, wherein the target feature vector refers to a feature vector corresponding to the first image block of each image block data in each group of sample pairs;

[0073] S33: inputting multiple target feature vectors into a mapping layer to obtain a similarity value of each group of sample pairs;

[0074] S34: Calculate the cross entropy loss value based on the similarity value and binary label of each group of sample pairs;

[0075] S35: Perform back propagation and gradient calculation on the cross entropy loss value to train the parameters in the first image encoder until the loss converges, and output the trained second image encoder.

[0076] For steps S31-S35, during the training process of the first image encoder, a group of positive sample pairs or a group of negative sample pairs are input, and the image block data of the two images in the sample pair are input into the first image encoder. The first image encoder will output two groups of vectors (i.e., first output vectors). After the vector corresponding to the first image block output by the first image encoder (i.e., the target feature vector), the vector of the first image block is input again into the mapping layer of the two added MLP (Multilayer Perceptron) layers. The mapping layer is finally a sigmoid activation function, which outputs a decimal in the range of 0-1, representing the similarity between the images of this group of sample pairs. Therefore, when the spliced ​​image blocks of the two images of any group of sample pairs are input into the same first image encoder, there will be two target feature vectors. After splicing these two target feature vectors, they are input into the two layers of MLP to output a similarity value.

[0077] Subsequently, the cross entropy loss value is calculated using the similarity value output by the first image encoder and the binary label corresponding to the input sample pair, and the gradient of the loss value calculation is back-propagated to train the parameters in the first image encoder until the loss value converges, and the trained second image encoder is output.

[0078] In actual application scenarios, the self-supervised first image encoder is used to train the image feature extraction process. First, the massive slide images are subjected to image data enhancement (such as flipping, rotating, and Gaussian noise addition). The original image and the data-enhanced image are used as a set of positive sample pairs.<x,x_p> , the original image and any unrelated image after data augmentation are taken as a set of negative sample pairs<x,x_n> . The label of the positive sample pair is marked as 1, and the label of the negative sample pair is marked as 0. From the dataset of positive sample pairs and negative sample pairs, 32 are randomly sampled as a training batch, and the training batch is input into the first image encoder model as a batch. Before input, all pictures need to be resized, and the input size is uniformly resized to a size of 64*64. Then the 64*64 image is divided into 16 4*4 image blocks (patches) without overlapping pixels. After dividing, the patches are tiled, that is, tiled into a 1*16 patch. The difference from the related art is that the present application will downsample the resized original image to 4*4 and place it at the front of the 1*16 patches to form a 1*17 patch. When training the first image encoder, because a set of positive sample pairs / negative sample pairs will be input, the present application will input the patches of the two pictures into a ViT model with shared weights. At this time, taking the positive sample pair as an example, the input style is: [x (downsampling)], [x_1], [x_2]... [x_16]. Another input is: [x_p (downsampling)], [x_p_1], [x_p_2], ... [x_p_16]. For any set of sample pairs as input, the 1*17 patches are input into an existing ViT model, and the ViT model will output 17 vectors of dimension 1024. A set of sample pairs will have two sets of outputs. After the first 1*1024 vector output by the ViT model, a two-layer MLP mapping layer is added, that is, only the first 1024 vector of the ViT model will be input into the mapping layer of this two-layer MLP layer. The last of this mapping layer is a sigmoid activation function, which outputs a decimal in the range of 0-1, representing the similarity between this set of sample images. When two images are input into the same ViT, there will be two first 1*1024 vectors. These two vectors are concatenated and input into two layers of MLP to output a similarity value. If the input is a positive sample pair, the label is 1, and the cross entropy loss value is calculated using the similarity value output by the model and label 1; if the input is a negative sample pair, the label is 0, and the cross entropy loss value is also calculated using the similarity value output by the model and label 0. The gradient of the loss value is calculated and back-propagated to train the parameters in the first image encoder until the loss value converges, and the second image encoder is output.

[0079] It can be seen from the above scheme that by specially training the image encoder, it is possible to fully extract the visual information (such as charts, pictures, layouts, etc.) in the slides, thereby providing a reliable basis for the subsequent generation of speech scripts.

[0080] S40: Based on the multiple groups of second positive sample pairs and the multiple groups of second negative sample pairs, the second image encoder and the first large language model are trained to obtain a third image encoder and a second large language model.

[0081] The large language model is used to process and generate text. By constructing positive and negative sample pairs of images and text, the text features and image features are aligned and mapped to the same representation space. The parameters of the image encoder and the large language model are then trained, allowing the model to understand the association between images and text, thereby improving its ability to understand complex scenes.

[0082] In one embodiment of the present application, a specific solution for training the second image encoder and the first large language model is provided. In S40, the second image encoder and the first large language model are trained based on multiple groups of second positive sample pairs and multiple groups of second negative sample pairs to obtain a third image encoder and a second large language model, which specifically includes the following steps S41-S45:

[0083] S41: inputting a plurality of image block data corresponding to the plurality of first image data as training data into a second image editor to obtain a second output vector;

[0084] S42: performing word segmentation processing on each first text data to obtain vector data corresponding to each first text data;

[0085] S43: inputting the plurality of vector data as training data into the first language model to obtain a third output vector;

[0086] S44: Calculate a loss value and a relative entropy based on the second output vector and the third output vector;

[0087] S45: Perform back propagation and gradient calculation on the sum of the loss value and the relative entropy to train the parameters in the second image encoder and the first large language model until the loss converges, and output the trained third image encoder and the second large language model.

[0088] For steps S41-S45, the first image data is input into the trained second image encoder to obtain the second output vector. The sentences in the first text data corresponding to each first image data are segmented using the word segmentation model to obtain the corresponding vector data, and the vector data is input into the first large language model to obtain the last hidden state vector (i.e., the third output vector) output by the first large language model. The loss value of the information noise comparison estimation loss function (InfoNCE) of the second output vector and the third output vector is calculated. At the same time, the relative entropy (KL divergence) of the second output vector and the third output vector needs to be calculated. Thereafter, after adding the loss values ​​of the relative entropy and InfoNCE, the parameters of the second image encoder and the first large language model are trained by back-propagating the loss value gradient until the loss value converges, and the third image encoder and the second large language model are output.

[0089] Optionally, the large language model may be a ChatGLM model.

[0090] In actual application scenarios, a set of positive sample pairs is generated by each slide image x1 and the speech text t_x1 corresponding to the slide image<x1,t_x1> ; Generate a set of negative sample pairs from each slide image x1 and the speech text of other unrelated slide images (such as t_x2, t_x3, etc.). Afterwards, input the image block data of the slide image into the trained ViT model to obtain a 1*17 vector output, and the dimension of each vector is 1024, denoted as V. Use the hanlp word segmentation model to segment the description text sentence corresponding to the slide image to obtain vector data (token). Input the token into the ChatGLM model to obtain the last hidden state vector output by the ChatGLM model, which is also a vector with a dimension of 1024, denoted as T. If the current input data is a positive sample pair, then the label is 1. If the current input data is a negative sample pair, then the label is 0. Calculate the loss value of the two vectors InfoNCE of V and T. At the same time, it is also necessary to calculate the KL divergence (relative entropy) of the two vectors V and T. Similarly, after adding the KL divergence to the loss value of InfoNCE, the parameters of the ViT and ChatGLM models are trained by back-propagating the loss value gradient until the loss value converges, and the trained ViT model and ChatGLM model are output.

[0091] It can be seen from the above scheme that joint training can effectively enhance the synergy between the image encoder and the language model and promote the understanding of visual features and language context.

[0092] S50: copy the second largest language model to obtain the third largest language model;

[0093] S60: Based on multiple groups of second positive sample pairs, multiple groups of second negative sample pairs, a third image encoder and a third language model, the second language model is trained to obtain a speech generation model.

[0094] For steps S50-S60, after the first language model is trained to obtain the second language model, the trained second language model is copied as the final speech generation model. Thereafter, based on the second positive and negative sample pairs, the copied third language model is trained in combination with the trained third image encoder and the second language model to generate the final speech generation model.

[0095] In one embodiment of the present application, Figure 4 As shown, a specific training speech generation scheme is provided. In S60, the second image encoder and the first large language model are trained based on multiple groups of second positive sample pairs and multiple groups of second negative sample pairs to obtain a third image encoder and a second large language model, which specifically includes the following steps S61-S68:

[0096] S61: inputting a plurality of image block data corresponding to a plurality of first image data as training data into a third image encoder to obtain a key matrix and a value matrix;

[0097] S62: inputting a plurality of vector data corresponding to the plurality of first text data as training data into a second language model to obtain a query matrix;

[0098] S63: Generate a first matrix vector based on the key matrix, the value matrix and the query matrix;

[0099] S64: Input the first matrix vector into a normalized exponential function to obtain an attention weight vector;

[0100] S65: multiply each vector value in the attention weight vector by each vector value in the value matrix to obtain a second matrix vector;

[0101] S66: Input the second matrix vector into the third language model to obtain a plurality of second text data;

[0102] S67: Calculate a cross entropy loss value based on the plurality of second text data and the plurality of first text data;

[0103] S68: Perform back propagation and gradient calculation on the cross entropy loss value to train the parameters in the first matrix, the second matrix and the third language model until the loss converges, and output the trained speech generation model.

[0104] For steps S61-S68, the second positive and negative sample pairs are used as training data and input into the third image encoder and the second largest language model, wherein the image block data corresponding to the first image data in the sample pair is input into the third image encoder, and a key matrix and a value matrix are generated based on the vector output by the third image encoder; at the same time, the vector data after word segmentation of the first text data in the sample pair is input into the second largest language model, and a query matrix is ​​generated based on the output vector of the second largest language model.

[0105] After that, after matrix multiplication of the transposed matrices of the query matrix and the key matrix, the multiplied matrices are added according to the dimension of the number of image blocks to obtain a first matrix vector. This vector is input into the normalized exponential function (softmax function) to obtain an attention weight vector, and each value in this attention weight vector is multiplied by the value matrix to finally obtain a second matrix vector after weighted cross attention. The second matrix vector is used as the input data of the third language model, and the third language model outputs the explanation manuscript text (i.e., the second text data). At this time, the cross entropy loss value is calculated using the second text data output by the model and the first text data. The cross entropy loss value is gradient calculated and back-propagated to update the parameters in the key matrix, value matrix, and query matrix, as well as the parameters in the second language model, until the loss converges, and the trained speech generation model is output.

[0106] In actual application scenarios, the second positive and negative sample pairs are used as training data, and the image block data of the first image data of any sample pair is input into the trained ViT model (third image encoder). The ViT model outputs a 1*17 feature vector, each of which has a dimension of 1024, that is, a 17*1024 matrix. At the same time, the vector data of the first text data corresponding to the image data after word segmentation is input into the ChatGLM model, and the ChatGLM model outputs a 1*n feature vector, each of which has a dimension of 1024, where n is the number of tokens in the prompt text (non-slide speech text), that is, an n*1024 matrix. The prompt text can be: Summarize the content of this ppt. Perform cross attention on these two matrices, that is, use 17*1024 as K (key matrix) and V (value matrix), and use n*1024 as Q (query matrix). Among them, Q, K, and V are all parameter matrices. Perform matrix multiplication of the transposed matrices of matrix Q and matrix K to obtain an n*17 matrix, then add the multiplied matrices in 17 dimensions to obtain a 1*17 vector, input this vector into the softmax function to obtain an attention weight vector, and multiply each value in this attention weight vector with matrix V, that is, the first value in the vector will be multiplied with each value in the first row of 1*1024 in matrix V, and the second value will be multiplied with each value in the second row of 1*1024 in matrix V. Finally, a 1*7 vector after weighted cross attention is obtained, and each vector is still of 1024 dimensions. Afterwards, this 1*17 vector is used as the input of the ChatGLM-copy model (i.e., the third language model), and the ChatGLM-copy model outputs the explanation manuscript text (i.e., the second text data). At this time, the explanation manuscript text output by the model and the manually annotated manuscript text (i.e., the first text data) are used to calculate the cross entropy loss value. The gradient of the cross entropy loss is back-propagated, and only the parameters in the Q, K, V matrices in the cross attention and the parameters of ChatGLM are updated. The ViT and ChatGLM-copy models are not updated until the loss converges.

[0107] It can be seen from the above scheme that after jointly training the synergistic effect between the image encoder and the language model, its performance on the speech generation task is further refined, enabling it to generate more natural and logically clear text based on the processed visual information.

[0108] It can be seen that in the above scheme, the speech generation model for explaining slides is trained through three stages. In the first stage, when training the image encoder, the focus is on improving the visual features, avoiding the interference of the language model when processing image information, so that the model can learn more attentively. In the second stage, the text features and image features are aligned and mapped to the same representation space, so that the model can understand the association between images and texts. Joint training can enhance the synergy between the image encoder and the language model, and promote the understanding of visual features and language context. In the third stage, the performance of the language model in the speech generation task is further refined, so that it can generate more natural and logically clear text based on the processed visual information, so that the model can better adapt to different topics and improve flexibility and accuracy. Through three stages of gradual training, the final speech generation model can unify the format of the slides in the form of pictures. The picture format retains the visual content in the slides (such as icons, illustrations, and layouts), eliminates garbled characters and typographical errors that may be caused by different text formats, and ensures the consistency of information transmission. The encoding model is then used to understand the pictures and user intentions, extract key information and features, and enhance the model's ability to understand visual information. Finally, by understanding the combination of picture content and user intentions, the generative model can automatically generate more targeted speech content, making the speech more professional and organized.

[0109] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0110] In one embodiment, a model training device is provided, which corresponds one-to-one to the model training method in the above embodiment. Figure 5 As shown, the model training device includes: an acquisition module 101, a first generation module 102, a training module 103, and a second generation module 104. The functional modules are described in detail as follows:

[0111] An acquisition module 101 is used to acquire a plurality of first image data and a plurality of first text data and a plurality of second image data corresponding thereto;

[0112] A first generating module 102, configured to generate a plurality of first positive sample pairs and a plurality of first negative sample pairs based on the plurality of first image data and the plurality of second image data, and to generate a plurality of second positive sample pairs and a plurality of second negative sample pairs based on the plurality of first image data and the plurality of first text data;

[0113] A training module 103 is used to train the first image encoder based on multiple groups of first positive sample pairs and multiple groups of first negative sample pairs to obtain a second image encoder;

[0114] The training module 103 is further configured to train the second image encoder and the first large language model based on the plurality of groups of second positive sample pairs and the plurality of groups of second negative sample pairs to obtain a third image encoder and a second large language model;

[0115] The second generation module 104 is used to copy the second largest language model to obtain a third largest language model;

[0116] The training module 103 is also used to train the second largest language model based on multiple groups of second positive sample pairs, multiple groups of second negative sample pairs, the third image encoder and the third largest language model to obtain a speech generation model.

[0117] In one embodiment, the acquisition module 101 is specifically configured to:

[0118] Acquire a plurality of first image data and a plurality of first text data;

[0119] Performing data enhancement processing on each first image data to generate second image data corresponding to each first image data;

[0120] The data enhancement processing includes at least one of the following: image flipping, image rotation and image noise addition.

[0121] In one embodiment, the first generating module 102 is specifically configured to:

[0122] A set of first positive sample pairs is formed by each first image data and its corresponding second image data;

[0123] A set of first negative sample pairs is formed by each first image data and any other first image data or any other second image data, wherein the other second image data is image data other than the image data corresponding to the first image data;

[0124] A set of second positive sample pairs is formed by each first image data and its corresponding first text data;

[0125] A set of second negative sample pairs is formed by each first image data and any other first text data, wherein the other first text data is text data other than the text data corresponding to the first image data;

[0126] Based on a preset splicing rule, preprocess the image data contained in the plurality of groups of first positive sample pairs and the plurality of groups of first negative sample pairs to form image block data corresponding to each image data;

[0127] The sample data included in the multiple groups of first positive sample pairs and the multiple groups of second positive sample pairs are labeled with a first binary label, and the sample data included in the multiple groups of first negative samples and the multiple groups of second negative sample pairs are labeled with a second binary label.

[0128] In one embodiment, the first generating module 102 is further configured to:

[0129] For any image data, resize the image data according to a first preset size;

[0130] According to preset cutting parameters, the image data after the size adjustment is cut to obtain multiple image blocks;

[0131] resizing the resized image data again according to a second preset size;

[0132] The resized image data is concatenated with a plurality of image blocks to form image block data.

[0133] In one embodiment, the training module 103 is specifically configured to:

[0134] Inputting a plurality of groups of first positive sample pairs and a plurality of groups of first negative sample pairs as training data into a first image encoder to obtain a plurality of first output vectors output by the first image encoder;

[0135] Determine a plurality of target feature vectors from the plurality of first output vectors, wherein the target feature vector refers to a feature vector corresponding to the first image block of each image block data in each group of sample pairs;

[0136] Input multiple target feature vectors into the mapping layer to obtain the similarity value of each group of sample pairs;

[0137] Calculate the cross entropy loss value based on the similarity value and binary label of each set of sample pairs;

[0138] The cross entropy loss value is back-propagated and gradient calculated to train the parameters in the first image encoder until the loss converges, and the trained second image encoder is output.

[0139] In one embodiment, the training module 103 is further configured to:

[0140] Inputting a plurality of image block data corresponding to the plurality of first image data as training data into a second image editor to obtain a second output vector;

[0141] Perform word segmentation processing on each first text data to obtain vector data corresponding to each first text data;

[0142] Inputting the plurality of vector data as training data into the first language model to obtain a third output vector;

[0143] Calculate the loss value and the relative entropy based on the second output vector and the third output vector;

[0144] The loss value and the sum of the relative entropy are back-propagated and gradient calculated to train the parameters in the second image encoder and the first large language model until the loss converges, and the trained third image encoder and the second large language model are output.

[0145] In one embodiment, the training module 103 is further configured to:

[0146] Inputting a plurality of image block data corresponding to the plurality of first image data as training data into a third image encoder to obtain a key matrix and a value matrix;

[0147] Inputting a plurality of vector data corresponding to the plurality of first text data as training data into the second largest language model to obtain a query matrix;

[0148] Generate a first matrix vector based on the key matrix, the value matrix and the query matrix;

[0149] Input the first matrix vector into the normalized exponential function to obtain the attention weight vector;

[0150] Multiply each vector value in the attention weight vector by each vector value in the value matrix to obtain a second matrix vector;

[0151] Inputting the second matrix vector into the third language model to obtain a plurality of second text data;

[0152] Calculating a cross entropy loss value based on the plurality of second text data and the plurality of first text data;

[0153] Backpropagate and calculate the gradient of the cross entropy loss value to train the key matrix, value matrix, query matrix, and parameters in the second language model until the loss converges, and output the trained speech generation model.

[0154] The present invention provides a model training device, which trains a speech generation model for explaining slides through three stages. In the first stage, when training the image encoder, the focus is on improving the visual features, avoiding the interference of the language model when processing image information, so that the model can learn more attentively. In the second stage, the text features and the picture features are aligned and mapped to the same representation space, so that the model can understand the association between the picture and the text. Through joint training, the synergy between the image encoder and the language model can be enhanced, and the understanding of visual features and language context can be promoted. In the third stage, the performance of the language model in the speech generation task is further refined, so that it can generate more natural and logically clear text based on the processed visual information, so that the model can better adapt to different topics and improve flexibility and accuracy. Through three stages of gradual training, the final speech generation model can unify the format of the slides in the form of pictures. The picture format retains the visual content in the slides (such as icons, illustrations, and layouts), eliminates garbled characters and typographical errors that may be caused by different text formats, and ensures the consistency of information transmission. The encoding model is then used to understand the pictures and user intentions, extract key information and features, and enhance the model's ability to understand visual information. Finally, by understanding the combination of picture content and user intentions, the generative model can automatically generate more targeted speech content, making the speech more professional and organized.

[0155] For the specific definition of the model training device, please refer to the definition of the model training method above, which will not be repeated here. Each module in the above-mentioned model training device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0156] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, it implements a function or step on the server side of a model training method.

[0157] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of the client side of a model training method.

[0158] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0159] Acquire a plurality of first image data and a plurality of first text data and a plurality of second image data corresponding thereto;

[0160] Based on the plurality of first image data and the plurality of second image data, a plurality of groups of first positive sample pairs and a plurality of groups of first negative sample pairs are generated, and based on the plurality of first image data and the plurality of first text data, a plurality of groups of second positive sample pairs and a plurality of groups of second negative sample pairs are generated;

[0161] Based on the multiple groups of first positive sample pairs and the multiple groups of first negative sample pairs, the first image encoder is trained to obtain a second image encoder;

[0162] Based on the multiple groups of second positive sample pairs and the multiple groups of second negative sample pairs, the second image encoder and the first large language model are trained to obtain a third image encoder and a second large language model;

[0163] Copy the second largest language model to get the third largest language model;

[0164] Based on multiple groups of second positive sample pairs, multiple groups of second negative sample pairs, a third image encoder and a third language model, the second language model is trained to obtain a speech generation model.

[0165] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0166] Acquire a plurality of first image data and a plurality of first text data and a plurality of second image data corresponding thereto;

[0167] Based on the plurality of first image data and the plurality of second image data, a plurality of groups of first positive sample pairs and a plurality of groups of first negative sample pairs are generated, and based on the plurality of first image data and the plurality of first text data, a plurality of groups of second positive sample pairs and a plurality of groups of second negative sample pairs are generated;

[0168] Based on the multiple groups of first positive sample pairs and the multiple groups of first negative sample pairs, the first image encoder is trained to obtain a second image encoder;

[0169] Based on the multiple groups of second positive sample pairs and the multiple groups of second negative sample pairs, the second image encoder and the first large language model are trained to obtain a third image encoder and a second large language model;

[0170] Copy the second largest language model to get the third largest language model;

[0171] Based on multiple groups of second positive sample pairs, multiple groups of second negative sample pairs, a third image encoder and a third language model, the second language model is trained to obtain a speech generation model.

[0172] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0173] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0174] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0175] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A model training method, characterized in that: include: Acquire a plurality of first image data and a plurality of first text data and a plurality of second image data corresponding thereto; Based on the plurality of first image data and the plurality of second image data, generating a plurality of groups of first positive sample pairs and a plurality of groups of first negative sample pairs, and based on the plurality of first image data and the plurality of first text data, generating a plurality of groups of second positive sample pairs and a plurality of groups of second negative sample pairs; Based on the multiple groups of first positive sample pairs and the multiple groups of first negative sample pairs, training the first image encoder to obtain a second image encoder; Based on the multiple groups of second positive sample pairs and the multiple groups of second negative sample pairs, the second image encoder and the first large language model are trained to obtain a third image encoder and a second large language model; Copy the second largest language model to obtain the third largest language model; Based on the multiple groups of second positive sample pairs, the multiple groups of second negative sample pairs, the third image encoder and the third largest language model, the second largest language model is trained to obtain a speech generation model.

2. The method according to claim 1, characterized in that The step of obtaining a plurality of first image data and a plurality of first text data and a plurality of second image data corresponding thereto specifically comprises: Acquire the plurality of first image data and the plurality of first text data; Performing data enhancement processing on each first image data to generate second image data corresponding to each first image data; The data enhancement processing includes at least one of the following: image flipping, image rotation and image noise addition.

3. The method according to claim 1, characterized in that The step of generating a plurality of groups of first positive sample pairs and a plurality of groups of first negative sample pairs based on the plurality of first image data and the plurality of second image data, and generating a plurality of groups of second positive sample pairs and a plurality of groups of second negative sample pairs based on the plurality of first image data and the plurality of first text data, specifically comprises: A set of first positive sample pairs is formed by each first image data and its corresponding second image data; A set of first negative sample pairs is formed by each of the first image data and any other first image data or any other second image data, wherein the other second image data is image data other than the image data corresponding to the first image data; By using each first image data and its corresponding first text data, a group of second positive sample pairs is formed; A set of second negative sample pairs is formed by each of the first image data and any other first text data, wherein the other first text data is text data other than the text data corresponding to the first image data; Based on a preset splicing rule, preprocess the image data contained in the plurality of groups of first positive sample pairs and the plurality of groups of first negative sample pairs to form image block data corresponding to each image data; The sample data included in the multiple groups of first positive sample pairs and the multiple groups of second positive sample pairs are labeled with a first binary label, and the sample data included in the multiple groups of first negative samples and the multiple groups of second negative sample pairs are labeled with a second binary label.

4. The method according to claim 3, characterized in that The step of preprocessing the image data contained in the plurality of groups of first positive sample pairs and the plurality of groups of first negative sample pairs based on the preset splicing rule to form image block data corresponding to each image data specifically includes: For any image data, resize the image data according to a first preset size; According to preset cutting parameters, the resized image data is cut to obtain a plurality of image blocks; resizing the resized image data again according to a second preset size; The image data after resizing is spliced ​​with the plurality of image blocks to form the image block data.

5. The method according to claim 3, characterized in that: The step of training the first image encoder based on the multiple groups of first positive sample pairs and the multiple groups of first negative sample pairs to obtain the second image encoder specifically includes: Inputting a plurality of groups of first positive sample pairs and a plurality of groups of first negative sample pairs as training data into a first image encoder to obtain a plurality of first output vectors output by the first image encoder; Determine a plurality of target feature vectors from the plurality of first output vectors, wherein the target feature vector refers to a feature vector corresponding to the first image block of each image block data in each group of sample pairs; Input the multiple target feature vectors into the mapping layer to obtain a similarity value for each group of sample pairs; Calculate the cross entropy loss value based on the similarity value and binary label of each set of sample pairs; The cross entropy loss value is back-propagated and gradient calculated to train the parameters in the first image encoder until the loss converges, and the trained second image encoder is output.

6. The method according to claim 3, characterized in that The step of training the second image encoder and the first large language model based on the multiple groups of second positive sample pairs and the multiple groups of second negative sample pairs to obtain a third image encoder and a second large language model specifically includes: Inputting a plurality of image block data corresponding to the plurality of first image data as training data into the second image editor to obtain a second output vector; Perform word segmentation processing on each first text data to obtain vector data corresponding to each first text data; Inputting the plurality of vector data as training data into the first language model to obtain a third output vector; Calculate a loss value and a relative entropy based on the second output vector and the third output vector; Back-propagation and gradient calculation are performed on the sum of the loss value and the relative entropy to train the parameters in the second image encoder and the first large language model until the loss converges, and the trained third image encoder and the second large language model are output.

7. The method according to claim 6, characterized in that The step of training the second largest language model based on the multiple groups of second positive sample pairs, the multiple groups of second negative sample pairs, the third image encoder and the third largest language model to obtain a speech generation model specifically includes: Inputting a plurality of image block data corresponding to a plurality of first image data as training data into the third image encoder to obtain a key matrix and a value matrix; Inputting a plurality of vector data corresponding to a plurality of first text data as training data into the second largest language model to obtain a query matrix; generating a first matrix vector based on the key matrix, the value matrix, and the query matrix; Input the first matrix vector into a normalized exponential function to obtain an attention weight vector; Multiply each vector value in the attention weight vector by each vector value in the value matrix to obtain a second matrix vector; Inputting the second matrix vector into the third language model to obtain a plurality of second text data; Calculating a cross entropy loss value based on the plurality of second text data and the plurality of first text data; The cross entropy loss value is back-propagated and gradient calculated to train the key matrix, the value matrix, the query matrix and the parameters in the second largest language model until the loss converges, and the trained speech generation model is output.

8. A model training device, characterized in that: include: An acquisition module, used for acquiring a plurality of first image data and a plurality of first text data and a plurality of second image data corresponding thereto; A first generating module, configured to generate a plurality of first positive sample pairs and a plurality of first negative sample pairs based on the plurality of first image data and the plurality of second image data, and to generate a plurality of second positive sample pairs and a plurality of second negative sample pairs based on the plurality of first image data and the plurality of first text data; A training module, configured to train the first image encoder based on the multiple groups of first positive sample pairs and the multiple groups of first negative sample pairs to obtain a second image encoder; The training module is further used to train the second image encoder and the first large language model based on the multiple groups of second positive sample pairs and the multiple groups of second negative sample pairs to obtain a third image encoder and a second large language model; A second generation module, used for copying the second largest language model to obtain a third largest language model; The training module is also used to train the second largest language model based on the multiple groups of second positive sample pairs, the multiple groups of second negative sample pairs, the third image encoder and the third largest language model to obtain a speech generation model.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the model training method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the model training method as described in any one of claims 1 to 7 are implemented.

Images