Similar sentence generation model training method and related equipment based on UniLM and contrastive learning

Abstract

The embodiment of the present application belongs to the field of artificial intelligence, and relates to a similar sentence generation model training method based on UniLM and contrastive learning, including inputting a sample sentence into a similar sentence generation model, wherein a sentence encoding layer is used to encode the sample sentence to obtain a dense vector, the multi-head self-attention structure is used to extract key information in the dense vector, and the key information and the dense vector are combined into a positive sample to calculate a contrast loss function; the dense vector is input into a UniLM model to obtain an output sentence, and a text alignment loss function between the sample sentence and the output sentence is calculated; after adding the contrast loss function and the text alignment loss function, a total loss function is obtained, and the final value of the total loss function is calculated based on the gradient descent method to obtain a trained similar sentence generation model. The use of this method improves the accuracy of the trained model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a method, apparatus, computer equipment and storage medium for training a similar sentence generation model based on UniLM and contrastive learning. Background Art

[0002] With the development of technology, natural language processing has been increasingly widely used in finance, the Internet, education, and medical fields, especially in the field of FAQ question-answering. Through the questions given by users, the answers are retrieved in the intelligent question-answering system. The technology involved is to match the user's questions with the FAQ standard question set, and retrieve the questions with the highest matching degree according to the matching degree, and return the corresponding answers. Traditional FAQ standard question sets are often expanded through manual annotation, which is time-consuming and labor-intensive, and increases a lot of labor costs. In recent years, models based on generative adversarial networks (GAN) or variational autoencoders (VAE) have been used to generate similar sentences, but GAN faces the problem of uncontrollable generation results and sentences that do not conform to grammatical specifications, while VAE depends on the distribution and number of samples and faces the problem of posterior collapse. In addition, with the development of pre-trained language models, some people have studied the use of the Bert model for similar sentence generation. The study found that the sentences generated by randomly dropping the vectors generated by the self-supervised model have a higher similarity with the original sentences, which is better than the conventional method of expanding the sentences by trimming and deleting words. Although this method can generate similar sentences to a certain extent, it may also discard more valuable information through random dropout, so there is a possibility of losing important information. Summary of the invention

[0003] Based on this, in response to the above technical problems, the present application provides a similar sentence generation model training method, device, computer equipment and storage medium based on UniLM and contrastive learning to solve the technical problem of losing important information caused by random dropout in the prior art.

[0004] A similar sentence generation model training method based on UniLM and contrastive learning, the method comprising:

[0005] Inputting the sample sentence into a similar sentence generation model, wherein the similar sentence generation model includes a sentence encoding layer and a multi-head self-attention structure including a mask matrix, wherein the sentence encoding layer is used to encode the sample sentence to obtain a dense vector, and the multi-head self-attention structure is used to extract key information from the dense vector, and the key information and the dense vector are combined into a positive sample to calculate a contrast loss function;

[0006] Input the dense vector into the UniLM model to obtain an output sentence, and calculate a text alignment loss function between the sample sentence and the output sentence;

[0007] After adding the contrast loss function and the text alignment loss function, a total loss function is obtained, and a final value of the total loss function is calculated based on a gradient descent method to obtain a trained similar sentence generation model.

[0008] A similar sentence generation model training device based on UniLM and contrastive learning, the device comprising:

[0009] A first loss module is used to input the sample sentence into a similar sentence generation model, wherein the similar sentence generation model includes a sentence encoding layer and a multi-head self-attention structure including a mask matrix, the sentence encoding layer is used to encode the sample sentence to obtain a dense vector, and the multi-head self-attention structure is used to extract key information from the dense vector, and the key information and the dense vector are combined into a positive sample to calculate a contrast loss function;

[0010] A second loss module, used for inputting the dense vector into a UniLM model to obtain an output sentence, and calculating a text alignment loss function between the sample sentence and the output sentence;

[0011] The total loss module is used to obtain a total loss function by adding the contrast loss function and the text alignment loss function, and calculate the final value of the total loss function based on the gradient descent method to obtain a trained similar sentence generation model.

[0012] A computer device comprises a memory and a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein when the processor executes the computer-readable instructions, the steps of the similar sentence generation model training method based on UniLM and contrastive learning are implemented.

[0013] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the similar sentence generation model training method based on UniLM and contrastive learning.

[0014] The above-mentioned similar sentence generation model training method, device, computer equipment and storage medium based on UniLM and contrastive learning can effectively screen out information with great value for the generation task through the proposed key information extractor. This application designs a key point extractor for screening information with great value for the generation task, and also proposes a method for joint training of improved small batch contrastive learning and UniLM native unidirectional language model (Left-to-Right LM) to further improve the quality of generated samples. In addition, the model obtained by the training method of this application can generate multiple similar sentences at a time, so as to achieve the purpose of rapidly expanding the data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0016] Figure 1 A schematic diagram of the application environment of the similar sentence generation model training method based on UniLM and contrastive learning;

[0017] Figure 2 A flowchart of the similar sentence generation model training method based on UniLM and contrastive learning;

[0018] Figure 3 Schematic diagram of the mask corresponding to the language model;

[0019] Figure 4 For training network structure diagram;

[0020] Figure 5 is the loss function graph;

[0021] Figure 6 A schematic diagram of a similar sentence generation model training device based on UniLM and contrastive learning;

[0022] Figure 7 FIG. 1 is a schematic diagram of a computer device in one embodiment. DETAILED DESCRIPTION

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0024] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0025] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0026] The similar sentence generation method based on UniLM and contrastive learning provided by the embodiment of the present invention can be applied to Figure 1 In the application environment shown, the application environment may include a terminal 102, a network and a server 104, the network is used to provide a communication link medium between the terminal 102 and the server 104, and the network may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0027] The user can use the terminal 102 to interact with the server 104 through the network to receive or send messages, etc. Various communication client applications can be installed on the terminal 102, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0028] Terminal 102 can be various electronic devices with a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV), laptop computers and desktop computers, etc.

[0029] The server 104 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal 102 .

[0030] It should be noted that the similar sentence generation method based on UniLM and contrastive learning provided in the embodiment of the present application is generally executed by the server / terminal, and accordingly, the similar sentence generation device based on UniLM and contrastive learning is generally arranged in the server / terminal device.

[0031] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0032] It should be understood that Figure 1 The number of terminals, networks and servers in the embodiment is only for illustration. Any number of terminals, networks and servers may be provided according to the implementation requirements.

[0033] The terminal 102 communicates with the server 104 through a network. The terminal 102 and the server 104 are connected through a network, which can be a wired network or a wireless network. The terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server 104 can be implemented with an independent server or a server cluster composed of multiple servers.

[0034] In one embodiment, Figure 2 As shown in the figure, a similar sentence generation model training method based on UniLM and contrastive learning is provided. Figure 1 The server in the example is used as an example to illustrate, including the following steps:

[0035] Step 202, inputting the sample sentence into a similar sentence generation model, wherein the similar sentence generation model includes a sentence encoding layer and a multi-head self-attention structure including a mask matrix, the sentence encoding layer is used to encode the sample sentence to obtain a dense vector, the multi-head self-attention structure is used to extract key information from the dense vector, and the key information and the dense vector are combined into a positive sample to calculate a contrast loss function;

[0036] This application proposes a similar sentence generation method based on UniLM and contrastive learning, and designs a key point extractor to filter information that is of great value to the generation task to solve the problem that the dropout method may lose important information. In addition, a method of jointly training an improved small-batch contrastive learning and UniLM's native single language model (Left-t-RightLM) is proposed to further improve the quality of generated samples.

[0037] Among them, UniLM is a unified pre-trained language model in this field. It jointly optimizes a Transformer network through three language models with different objective functions. These three language models include a bidirectional language model (Bidrectional LM), a unidirectional language model (Left-to-Right LM) and a sequence-to-sequence language model (Seq-to-Seq LM). This application adopts the pre-trained UniLM language model and only uses the unidirectional language model part for improvement. UniLM includes a sentence encoding module, several Transformer Blocks, and an optimization function layer (i.e., the optimization objective function of the three language models).

[0038] Let the input sentence be X. ,in represents the i-th word of the sentence, input X into the sentence encoding layer, and record the obtained sentence encoding as X_vec (such as Figure 2 As shown, X is used in calculating the cross entropy loss function, and X_vec is used in calculating the contrastive learning loss function).

[0039] The dense vector in this embodiment belongs to the conventional operation of NLP, which maps words into vectors.

[0040] In one embodiment, the structure of the transformer block of UniLM is basically the same as that of Bert (a pre-trained language model), which adopts a multi-head self-attention structure. The difference is that a mask matrix is ​​added. The calculation formula (1) is as follows: (1)

[0041] Among them, QKV is the result of multiplying the input vector by three transformation matrices. is the output of the corresponding block. is a mask, indicating whether the word can be seen by other words. For example, the mask corresponding to the one-way language model is Figure 3 shown.

[0042] For unidirectional LM (left-to-right or right-to-left), we can only attend to the context of one direction for each word when doing attention. For example, for left-to-right LM, we can only attend to the information on the left side of the word and need to mask the words on the right side; the same is true for right-to-left LM. In this way, the mask matrix we get is an upper triangle or a lower triangle.

[0043] In this embodiment, in order to solve the technical problem, the existing UniLM model is improved:

[0044] Design key information extractor: The method of using dropout to generate similar sentences is to Perform random dropout to obtain new , this method will discard important information. This method is designed as follows (2) (3) (4) formula:

[0045] (2);

[0046] (3);

[0047] (4);

[0048] Among them, formula (2) Calculate the similarity between the i-th component of the Q vector and all K-th component vectors, is the dimension of the vector, such as (64, 256, 512, etc.), and lk is the length of the input sentence. Formula (3) means that all Q components are calculated according to formula (2), and the obtained values ​​are sorted from large to small, and the first m components are selected. Here, m can be customized, indicating that m important vectors are retained. Formula (4) refers to selecting important sentence vectors through methods (2) and (3) (formula (3) can achieve this purpose), adding them to the mask M, and multiplying them by V to obtain a new value , that is, a new sentence encoding is obtained. Formula (2) can calculate the connection between the two vectors qk (which can be understood as similarity). If qk is closely related, the obtained value will be relatively large. If the connection is not so close, the obtained value will be relatively small. q is calculated with all k to obtain a series of values, which are then sorted by formula (3). The first m largest values ​​are taken and the vectors corresponding to these values ​​are retained to achieve the purpose of key information extraction. Among them, each layer of transformer block corresponds to a key information extractor. Through L layers of transformer blocks and corresponding key information extractors, L corresponding outputs can be obtained. These L outputs and the input sentence encoding X_vec form a positive sample pair to calculate the contrast loss function, such as Figure 4 shown.

[0049] In this embodiment, contrastive learning is a type of self-supervised learning. It trains a representation learning model by constructing similar instances and dissimilar examples. Through this model, similar instances are closer in the projection space, while dissimilar instances are farther away in the projection space. That is, the purpose of contrastive learning is to shorten the distance between similar sentences and increase the distance between unrelated sentences, so that the sentences generated by the trained model will be more similar to the original sentences.

[0050] From the above formulas (2), (3) and (4), we can get L vectors obtained by the key information extractor, which are recorded as , (that is, the above steps obtained ). During the training phase, the input sentence encoding X_vec and They form L groups of positive sample pairs, namely , and X_vec and other samples in the same batch of training samples form negative sample pairs. The following formulas (5), (6), and (7) are designed to calculate the contrast loss function:

[0051] (5);

[0052] (6);

[0053] (7);

[0054] In formula (5), the numerator calculates the similarity of the i-th pair of positive samples, and the denominator calculates the similarity of all negative sample pairs, and the cumulative sum is calculated. represents the input sample, namely X_vec, Represents the vector obtained by the key point extractor, that is,

[0055] In , is the hyperparameter to be trained. Formula (6) is the total contrastive learning loss function, which consists of the losses of L positive samples and gives each loss a weight , Satisfies the relationship of equation (7). Parameters It can be trained together with the network (such as gradient descent algorithm) to determine the final value through training. This improved contrast loss function can comprehensively consider the impact of the vector selected by the extractor on the quality of the final generated sample. The existence of will give important vectors a larger weight and secondary vectors a smaller weight, making the generated result more reasonable; the generated result refers to the sentence decoded by the Left-to-Right LM module, that is, the process of decoding from a bunch of vectors into text.

[0056] Step 204, inputting the dense vector into the UniLM model to obtain an output sentence, and calculating a text alignment loss function between the sample sentence and the output sentence;

[0057] like Figure 5 As shown, the text alignment loss function adopts the cross entropy loss function commonly used in the generative model, denoted by J, formula (8). The input sentence is the input sentence X in step 202, and the generated sentence is the sentence generated by X after UniLM. The cross entropy loss function is obtained by calculating the input sentence and the generated sentence.

[0058] (8);

[0059] in, The dense vector is mapped into a probability value through the softmax function, that is, the normalized exponential function, and the probability of the current dense vector being decoded into a certain word is obtained. The loss function of each step is summed up to obtain the total cross entropy loss function J.

[0060] Step 206, after adding the contrast loss function and the text alignment loss function, a total loss function is obtained, and a final value of the total loss function is calculated based on a gradient descent method to obtain a trained similar sentence generation model.

[0061] The similar sentence generation model training of this application adopts a multi-task joint training method. The total loss function includes a contrastive learning loss function and a cross entropy loss function, namely:

[0062] ;

[0063] In another embodiment, generated sentences can be obtained based on the trained model, and the training samples can be input into the model for training. After the model is trained, L+1 generated sentences can be obtained, where L generated sentences are vectors obtained after conversion by the key information extractor and then input into the decoding module (i.e., Left-to-Right LM) to generate sentences, and the other is a sentence generated without being decoded by the key information extractor.

[0064] Example:

[0065] In the field of intelligent customer service data expansion or other intelligent question-answering fields, it is often necessary to accurately match user questions with questions in the standard question library in order to quickly return the user's request. However, in the past, the expansion of the standard question library was manually entered, resulting in a large amount of labor costs. This method can automatically generate similar sentences, quickly expand the standard question library, and reduce labor costs.

[0066] For example, there are the following training samples:

[0067] What are the benefits of eating more leeks

[0068] How to make money in the cloud

[0069] What does long fresh mean?

[0070] Can we get married?

[0071] The above data is used as a training batch input model for training, and will be reconstructed. For example, the sentence "What are the benefits of eating more leeks?" will get L feature vectors after the feature extractor. The feature vector of this sentence and these L feature vectors form a positive sample pair, and form a negative sample pair with other 4 sentences such as "How to make money in the cloud", and finally train. After the training, if the user inputs a sentence "What are the benefits of eating more leeks?", L+1 sentences similar to "What are the benefits of eating more leeks" can be obtained, such as "What are the benefits of eating more leeks?", "What are the benefits of eating more leeks?", and so on.

[0072] The key information extractor proposed by the method of this application can effectively filter out information that is of great value to the generation task. This application designs a key point extractor for filtering information that is of great value to the generation task, and also proposes a method for joint training of improved small batch contrastive learning and UniLM's native unidirectional language model (Left-to-Right LM) to further improve the quality of generated samples. In addition, the model obtained by the training method of this application can generate multiple similar sentences at a time, achieving the purpose of rapidly expanding the data set.

[0073] It should be understood that although Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages do not necessarily have to be executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages does not have to be sequentially, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0074] In one embodiment, Figure 6 As shown, a similar sentence generation model training device based on UniLM and contrastive learning is provided, and the similar sentence generation model training device based on UniLM and contrastive learning corresponds one-to-one to the similar sentence generation model training method based on UniLM and contrastive learning in the above embodiment. The similar sentence generation model training device based on UniLM and contrastive learning includes:

[0075] A first loss module 602 is used to input the sample sentence into a similar sentence generation model, wherein the similar sentence generation model includes a sentence encoding layer and a multi-head self-attention structure including a mask matrix, the sentence encoding layer is used to encode the sample sentence to obtain a dense vector, and the multi-head self-attention structure is used to extract key information from the dense vector, and the key information and the dense vector are combined into a positive sample to calculate a contrast loss function;

[0076] A second loss module 604 is used to input the dense vector into the UniLM model to obtain an output sentence, and calculate a text alignment loss function between the sample sentence and the output sentence;

[0077] The total loss module 606 is used to obtain a total loss function by adding the contrast loss function and the text alignment loss function, and calculate the final value of the total loss function based on the gradient descent method to obtain a trained similar sentence generation model.

[0078] The key information extractor proposed by the device of this application can effectively screen out information that is of great value to the generation task. This application designs a key point extractor for screening information that is of great value to the generation task, and also proposes a method of joint training of improved small batch contrastive learning and UniLM's native unidirectional language model (Left-to-Right LM) to further improve the quality of generated samples. In addition, the model obtained by the training method of this application can generate multiple similar sentences at a time, so as to achieve the purpose of rapidly expanding the data set.

[0079] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer-readable instructions and a database. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the non-volatile storage medium. The database of the computer device is used to store sample data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer-readable instructions are executed by the processor, a similar sentence generation model training method based on UniLM and contrastive learning is implemented.

[0080] Among them, technicians in this technical field can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0081] In one embodiment, a computer-readable storage medium is provided on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the steps of the similar sentence generation model training method based on UniLM and contrastive learning in the above embodiment are implemented, for example: Figure 2Steps 202 to 206 shown, or, when the processor executes the computer-readable instructions, the functions of each module / unit of the similar sentence generation model training device based on UniLM and contrastive learning in the above embodiment are realized, for example Figure 6 The functions of modules 602 to 606 are shown.

[0082] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through computer-readable instructions, and the computer-readable instructions can be stored in a non-volatile computer-readable storage medium. When the computer-readable instructions are executed, they may include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0083] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0084] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0085] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for those of ordinary skill in the art, without departing from the concept of the present application, several modifications, improvements or equivalent replacements of some technical features can be made, and these modifications or replacements do not make the essence of the same technical solution depart from the spirit and scope of the technical solution of each embodiment of the present invention, and all belong to the protection scope of the present application. Therefore, the protection scope of the patent of this application shall be subject to the attached claims.

Claims

1. A similar sentence generation model training method based on UniLM and contrastive learning, characterized in that: The method comprises: Inputting the sample sentence into a similar sentence generation model, wherein the similar sentence generation model includes a sentence encoding layer and a multi-head self-attention structure including a mask matrix, wherein the sentence encoding layer is used to encode the sample sentence to obtain a dense vector, and the multi-head self-attention structure is used to extract key information from the dense vector, and the key information and the dense vector are combined into a positive sample to calculate a contrast loss function; Input the dense vector into the UniLM model to obtain an output sentence, and calculate a text alignment loss function between the sample sentence and the output sentence; After adding the contrast loss function and the text alignment loss function, a total loss function is obtained, and a final value of the total loss function is calculated based on a gradient descent method to obtain a trained similar sentence generation model; Among them, the key information extractor is designed, and the following (2) (3) (4) formulas are designed: （2）； （3）； （4）； Among them, formula (2) Calculate the similarity between the i-th component of the Q vector and all K-th component vectors, is the dimension of the vector, lk is the length of the input sentence, formula (3) means calculating formula (2) for all Q components, sorting them from large to small according to the obtained values, and selecting the first m components, where m can be customized, indicating that m important vectors are retained. Formula (4) means selecting important sentence vectors through methods (2) and (3), adding them to the mask M, and multiplying them by V to obtain a new value , that is, a new sentence encoding is obtained; each layer of multi-head self-attention structure corresponds to a key information extractor.

2. The method according to claim 1, characterized in that The text alignment loss function adopts a cross entropy loss function, denoted as J, and the cross entropy loss function J is: ; in, is the input vector After being mapped into a probability value by a normalized exponential function, the probability of the current dense vector being decoded into a certain word is obtained.

3. The method according to claim 1, characterized in that The total loss function Loss is: ; Among them, Lcont is the contrast loss function and J is the cross entropy loss function.

4. A similar sentence generation model training device based on UniLM and contrastive learning, characterized in that: include: A first loss module is used to input the sample sentence into a similar sentence generation model, wherein the similar sentence generation model includes a sentence encoding layer and a multi-head self-attention structure including a mask matrix, the sentence encoding layer is used to encode the sample sentence to obtain a dense vector, and the multi-head self-attention structure is used to extract key information from the dense vector, and the key information and the dense vector are combined into a positive sample to calculate a contrast loss function; A second loss module, used for inputting the dense vector into a UniLM model to obtain an output sentence, and calculating a text alignment loss function between the sample sentence and the output sentence; A total loss module, used for adding the contrast loss function and the text alignment loss function to obtain a total loss function, and calculating a final value of the total loss function based on a gradient descent method to obtain a trained similar sentence generation model; Among them, the key information extractor is designed, and the following (2) (3) (4) formulas are designed: （2）； （3）； （4）； Among them, formula (2) Calculate the similarity between the i-th component of the Q vector and all K-th component vectors, is the dimension of the vector, lk is the length of the input sentence, formula (3) means calculating formula (2) for all Q components, sorting them from large to small according to the obtained values, and selecting the first m components, where m can be customized, indicating that m important vectors are retained. Formula (4) means selecting important sentence vectors through methods (2) and (3), adding them to the mask M, and multiplying them by V to obtain a new value , that is, a new sentence encoding is obtained; each layer of multi-head self-attention structure corresponds to a key information extractor.

5. A computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, characterized in that: When the processor executes the computer-readable instructions, the steps of the method according to any one of claims 1 to 3 are implemented.

6. A computer-readable storage medium having computer-readable instructions stored thereon, characterized in that: When the computer readable instructions are executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

Images