Training method, device, equipment and storage medium of autoregressive language model

Abstract

The present application relates to the field of artificial intelligence technology and discloses a training method, apparatus, medium and equipment for an autoregressive language model, wherein the method comprises: obtaining a training sentence and extracting a training text vector of the training sentence; obtaining a configuration instruction and an expected training result of the training sentence, and configuring a sentence template according to identifier information and configuration rules in the configuration instruction; inputting the training text vector and the sentence template into a word vector model, performing mapping calculation on the training text vector and the sentence template through the word vector model, and obtaining content mapping information of the training text vector under the sentence template; based on an attention model, iteratively training the word vector model according to the content mapping information and the expected training result to obtain an autoregressive language model based on the sentence template, thereby ensuring generation efficiency while generating text according to the format.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method, apparatus, medium, and equipment for an autoregressive language model. Background Art

[0002] With the development of artificial intelligence (AI) technology, neural network-based text generation has made significant progress in various tasks. For tasks like dialogue generation and story generation, text can be generated freely without being restricted by specific formats. However, certain specific texts, such as insurance contracts, bank asset certificates, and even legal documents, have strict formatting requirements. Current technologies clearly fall short of meeting these requirements. These tasks often have specific requirements for word count, formatting, and symbols. Even the slightest omission or error in formatting can lead to significant losses and immeasurable consequences.

[0003] However, it is difficult for existing technologies to quickly generate text in corresponding formats under a large number of text generation tasks, and it is difficult to achieve a text format that meets the text generation requirements while ensuring the generation rate. Summary of the Invention

[0004] The main purpose of this application is to provide a training method, device, medium and equipment for an autoregressive language model, aiming to solve the technical problem in the prior art that it is difficult to meet the generated text format while ensuring generation efficiency.

[0005] In order to achieve the above-mentioned object of the invention, the present application proposes a method for training an autoregressive language model, the method comprising:

[0006] Obtaining a training sentence and extracting a training text vector of the training sentence;

[0007] Obtaining a configuration instruction and an expected training result of the training sentence, and configuring a sentence template according to identifier information in the configuration instruction and a configuration rule, wherein the identifier information includes one or more of ordinary text characters, punctuation characters, rhyme characters, separators, and terminators;

[0008] Inputting the training text vector and the sentence template into a word vector model, performing a mapping calculation on the training text vector and the sentence template through the word vector model to obtain content mapping information of the training text vector under the sentence template;

[0009] Based on the attention model, the word vector model is iteratively trained according to the content mapping information and the expected training result to obtain an autoregressive language model based on the sentence template.

[0010] Furthermore, before iteratively training the word vector model based on the attention model according to the content mapping information and the expected training result, the method further includes:

[0011] Inputting the sentence template separately into a word vector model, performing mapping calculation on the sentence template through the word vector model, and obtaining format mapping information of the sentence template;

[0012] The iterative training of the word vector model based on the attention model according to the content mapping information and the expected training result includes:

[0013] Based on the attention model, the word vector model is iteratively trained according to the content mapping information, the expected training result and the format mapping information.

[0014] Furthermore, the attention model includes a hidden multi-head self-attention model and a global multi-head attention model;

[0015] The iterative training of the word vector model based on the attention model according to the content mapping information and the expected training result includes:

[0016] Inputting the content mapping information into the hidden multi-head self-attention model to obtain a context vector;

[0017] Inputting the context vector and the format mapping information into the global multi-head attention model to obtain a feature learning result of the context vector under the format mapping information;

[0018] Calculate the error value between the feature learning result and the expected training result, adjust the model parameters of the word vector model according to the error value, and use the adjusted word vector model to iteratively calculate the content mapping information until the error value meets the preset error range.

[0019] Furthermore, the content mapping information is input into a hidden multi-head self-attention model to obtain a context vector, including:

[0020] Performing mask calculation on the content mapping information to obtain hidden mapping information;

[0021] The hidden mapping information is input into the transformer layer of the hidden multi-head self-attention model to obtain the context vector.

[0022] Furthermore, inputting the context vector and the format mapping information into a global multi-head attention model to obtain a feature learning result of the context vector under the format mapping information includes:

[0023] The global multi-head attention model is used to perform self-attention model calculation on the context vector and the format mapping information to obtain the feature learning result after the context vector and the format mapping information are fused.

[0024] Furthermore, the identifier information also includes a location identifier;

[0025] The configuring statement template according to the identifier information and configuration rules in the configuration instruction includes:

[0026] Identifying the total number of characters in the training sentence, and generating position identifiers having the same number as the total number of characters, wherein each of the position identifiers includes a first numerical identifier;

[0027] According to the configuration rule, the position identifiers are sorted according to the size of the first digital identifiers;

[0028] A one-to-one correspondence is established between each of the position identifiers and each of the characters in sorted order to obtain the sentence template corresponding to the training sentence.

[0029] Furthermore, the identifier information also includes a sentence separator;

[0030] The configuring statement template according to the identifier information and configuration rules in the configuration instruction includes:

[0031] Identifying the punctuation marks of the training sentence, and taking the text content between two consecutive occurrences of the punctuation marks as segmented content;

[0032] generating sentence separators whose number is the same as the number of the segmented contents, wherein each of the sentence separators includes a second digital identifier;

[0033] According to the configuration rule, the statement separators are sorted according to the size of the second digital identifier;

[0034] A one-to-one correspondence is established between each of the sentence separators and each of the segmented contents in a sorted order, so as to obtain the sentence template corresponding to the training sentence.

[0035] This application also proposes a training device for an autoregressive language model, comprising:

[0036] A vector extraction module, configured to obtain a training sentence and extract a training text vector of the training sentence;

[0037] a sentence template configuration module, configured to obtain a configuration instruction and an expected training result of the training sentence, and configure a sentence template according to identifier information in the configuration instruction and configuration rules, wherein the identifier information includes one or more of ordinary text characters, punctuation characters, rhyme characters, separators, and terminators;

[0038] A mapping calculation module, configured to input the training text vector and the sentence template into a word vector model, perform mapping calculation on the training text vector and the sentence template through the word vector model, and obtain content mapping information of the training text vector under the sentence template;

[0039] A model generation training module is used to iteratively train the word vector model based on the attention model according to the content mapping information and the expected training results to obtain an autoregressive language model based on the sentence template.

[0040] The present application also proposes a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above methods when executing the computer program.

[0041] The present application also proposes a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above-mentioned methods are implemented.

[0042] The training method, device, medium and equipment of the autoregressive language model of the present application designs the text generation format through configuration instructions to generate a sentence template corresponding to the format, thereby ensuring the format diversity and flexibility of text generation; the training text vector and the sentence template are mapped and calculated by the word vector model to obtain the content mapping information of the training text vector, so that the above-mentioned word vector model can generate text in the corresponding format based on the text format of the sentence template, without the need for additional typesetting of the text, thereby improving the text generation efficiency; the word vector model is iteratively calculated by the attention model to obtain an autoregressive language model based on the above-mentioned sentence template, and since the sentence template can be set according to the configuration instructions, the coverage of the sentence template can be flexibly changed by setting the identifier information and configuration instructions with different configuration rules, so that it can be applied not only to finance, but also to a series of scenarios with fixed format text generation requirements such as poetry, lyrics, songs, etc., thereby improving the versatility of the autoregressive language model and the flexibility of format setting. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Schematic diagram of a flow chart of a method for training an autoregressive language model according to an embodiment of the present application;

[0044] Figure 2A flowchart of a method for training an autoregressive language model according to a specific embodiment of the present application is shown;

[0045] Figure 3 This is a schematic block diagram of the structure of a training device for an autoregressive language model according to an embodiment of the present application;

[0046] Figure 4 This is a schematic block diagram of the structure of a computer device according to an embodiment of the present application.

[0047] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0049] Reference Figure 1 In order to achieve the above-mentioned purpose, the present application provides a method for training an autoregressive language model, which includes:

[0050] S1: Obtain a training sentence and extract a training text vector of the training sentence;

[0051] S2: Obtaining a configuration instruction and an expected training result of the training sentence, and configuring a sentence template according to the identifier information in the configuration instruction and the configuration rule, wherein the identifier information includes one or more of ordinary text characters, punctuation characters, rhyme characters, separators, and terminators;

[0052] S3: Inputting the training text vector and the sentence template into a word vector model, performing a mapping calculation on the training text vector and the sentence template through the word vector model to obtain content mapping information of the training text vector under the sentence template;

[0053] S4: Based on the attention model, iteratively train the word vector model according to the content mapping information and the expected training result to obtain an autoregressive language model based on the sentence template.

[0054] This embodiment designs the text generation format through configuration instructions to generate a sentence template corresponding to the format, thereby ensuring the format diversity and flexibility of text generation; the training text vector and the sentence template are mapped and calculated through the word vector model to obtain the content mapping information of the training text vector, so that the above-mentioned word vector model can generate text in the corresponding format based on the text format of the sentence template, without the need for additional typesetting of the text, thereby improving the text generation efficiency; the word vector model is iteratively calculated through the attention model to obtain an autoregressive language model based on the above-mentioned sentence template, and since the sentence template can be set according to the configuration instructions, the coverage of the sentence template can be flexibly changed by setting the identifier information and configuration instructions with different configuration rules, so that it can be applied not only to finance, but also to a series of scenarios with fixed format text generation requirements such as poetry, lyrics, songs, etc., thereby improving the versatility of the autoregressive language model and the flexibility of format setting.

[0055] For step S1, this embodiment is generally used in the field of intelligent text generation, and can train the autoregressive language model based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Specifically, the above-mentioned training sentences refer to example sentences used for model training, and the above-mentioned training text vector is a digital expression of the training sentence text, so as to facilitate the computer to understand the text, that is, whether the training text vector can accurately express the text determines whether the model can accurately understand the meaning of the text; this embodiment uses NLP (Neuro-Linguistic Programming, natural language processing) model performs text preprocessing on the training sentences, that is, part-of-speech tagging, syntactic analysis and semantic analysis; the above-mentioned part-of-speech tagging usually uses morphology, or the relationship between words, to identify the purpose of each word in the training sentence, such as verbs, nouns, adjectives, etc. by using established rules in the text corpus; the above-mentioned syntactic analysis usually uses syntax or analyzes how words and sentences are combined to understand the structure of the sentence, thereby breaking down the training sentences into phrases; the above-mentioned semantic analysis usually combines other words that appear before and after the word to better understand the meaning of the target word and reduce the occurrence of ambiguity; after text preprocessing, the text is converted It is a digital form that can be understood by the model, including generating a bag-of-words model, N-gram model or word embedding corresponding to each word in the training sentence, wherein the bag-of-words model is a representation of the text that describes the occurrence of words in the document, which represents the text by counting the number of times each word in the input document appears compared with the vocabulary of known words. The output result of the bag-of-words model is a set of vectors corresponding to the training sentence, which contains numbers describing the number of times each word appears. These vectors are "bags of words"; however, the output result of the bag-of-words model does not contain any information about the structure of the input document. At this time, the N-gram model can be used to make the bag-of-words model reduce context loss, which is achieved by creating a vocabulary of combined characters instead of single characters.These grouped characters are called "n-grams", where "n" is the group size. The resulting method is called "N-gram model". It is understandable that each n-gram can capture more context than a single character. Since machine learning methods often cannot process text data directly, text data needs to be converted into numerical data, that is, it needs to be converted through the word embedding method of Word Embedding. It maps each character to a separate training text vector, making the training text vector tend to be "dense" rather than "sparse", and in the mapping process, each character and its surrounding characters are considered. The resulting denser training text vector can help better analyze and compare characters and context. Specifically, algorithms such as Word2Vec, GloVe and FastText can be used to extract the training text vectors of the above training sentences.

[0056] In step S2, after obtaining the configuration instruction, the identifier information and configuration rules are parsed from the configuration instruction. In order to ensure the rhythm and sentence integrity of the sentence template, this embodiment defines several symbols such as ordinary text symbols, punctuation symbols, rhyme symbols, separators and end symbols to model these special information; the configuration rules can be: using marker c0 to represent ordinary text, marker c1 to represent punctuation marks, marker c2 to represent rhyme, and marker c3 to represent separators. <eos>Indicates the end character. Here, taking "love is not love, bends with the remove to remove" as an example, the language template C obtained by configuring according to the configuration rules is as follows:

[0057] C={c0,c0,c0,c2,c1,c0,c0,c0,c0,c0,c2,c1, <eos>}

[0058] Among them, since the rhymes of the above English poem are "love" and "remove", the corresponding ones are c2, the label symbol is c1, and the rest are c0. In a specific implementation, a language template can be generated according to the actual format requirements to improve the versatility and flexibility of the model.

[0059] The expected training result may be a standard result artificially generated based on the training sentence, that is, the result that the user expects the autoregressive language model to obtain based on the training sentence; the expected training result is the result after the training sentence is arranged in the format of the sentence template.

[0060] For step S3, the word vector model can be an initial word-level encoder (Bidirectional Encoder Representations from Transformers, BERT). Specifically, the content mapping information is obtained by summing all input training text vectors and tokens of sentence templates. The calculation method is as follows:

[0061]

[0062] Where 0 represents the 0th layer of the iteration, w refers to the training text vector, c is the above-mentioned language template, g represents the global position embedding vector, and this embodiment uses absolute position encoding. t represents different positions in the language template, such as ct: c0, c1, c2, etc.; H t 0 That is, the above-mentioned content mapping information of layer 0; the above-mentioned absolute position encoding is usually to directly randomly initialize a global position vector global position embedding for different positions, add it to word embedding, and train it as a parameter of the word vector model.

[0063] For example, for C = {c0,c0,c0,c2,c1,c0,c0,c0,c0,c0,c2,c1, <eos>}, and its corresponding content mapping information is

[0064] For step S4, after obtaining the above-mentioned content mapping information, the attention model is used to select the important parts of the content mapping information as the input for encoding; specifically, the weight value of the corresponding part of the content mapping information is obtained through softmax, and a weight is added to each content mapping information in the encoding stage, thereby obtaining the importance of each part in the content mapping information sequence, improving the attention of the autoregressive language model, and then accurately obtaining the gap between the content mapping information and the expected training result, so that the attention model iteratively trains the model parameters of the word vector model to reduce the gap between the content mapping information and the expected training result.

[0065] In one embodiment, before iteratively training the word vector model based on the attention model according to the content mapping information and the expected training result S4, the method further includes:

[0066] S401: Inputting the sentence template into a word vector model separately, performing mapping calculation on the sentence template through the word vector model, and obtaining format mapping information of the sentence template;

[0067] The iterative training S4 of the word vector model based on the attention model according to the content mapping information and the expected training result includes:

[0068] S41: Based on the attention model, the word vector model is iteratively trained according to the content mapping information, the expected training result and the format mapping information.

[0069] This embodiment iteratively trains the word vector model through content mapping information and format mapping information at the same time, so that the word vector model can see subsequent information of the sentence template during the dynamic calculation process, thereby improving the accuracy of the model.

[0070] For step S401, in order to allow the dynamic operation process of the word vector model to see the subsequent information of the sentence template in addition to the current information, and thus obtain the global dynamic information of the sequence, this embodiment introduces format mapping information F; specifically, the word vector model may want to know whether the decoding process should be ended by generating the last word and a punctuation mark to end the sentence. In order to represent the global dynamic information, this embodiment uses the format mapping information F obtained by inputting the sentence template into the word vector model separately. This variable only integrates the symbol information in the previously defined sentence template, and does not have the input training sentence to avoid information leakage on the right side. Its specific calculation method is as follows:

[0071]

[0072] Specifically, iterative calculations are performed simultaneously based on the format mapping information F and the content mapping information H, so that when the word vector model calculates the gap between the content mapping information and the expected training result, it can see the subsequent information of the sentence template during the dynamic calculation process, so that the generated autoregressive language model can obtain global dynamic information, thereby improving the accuracy of the model.

[0073] In one embodiment, referring to Figure 2 , the attention model includes a hidden multi-head self-attention model and a global multi-head attention model;

[0074] The iterative training of the word vector model based on the attention model according to the content mapping information, the expected training result and the format mapping information S41 includes:

[0075] S411: Input the content mapping information into the hidden multi-head self-attention model to obtain a context vector;

[0076] S412: Inputting the context vector and the format mapping information into the global multi-head attention model to obtain a feature learning result of the context vector under the format mapping information;

[0077] S413: Calculate the error value between the feature learning result and the expected training result, adjust the model parameters of the word vector model according to the error value, and use the adjusted word vector model to iteratively calculate the content mapping information until the error value meets the preset error range.

[0078] This embodiment uses an attention model to perform feature learning on content mapping information and format mapping information, and then iteratively calculates the model parameters according to the error value of the learning result, and finally obtains an autoregressive language model with smaller error, thereby reducing the model error and further improving the accuracy of the model.

[0079] For step S411, the hidden multi-head self-attention model is Masking Multi-Head Self-Attention, which is used to perform mask calculation on the content mapping information to obtain a context vector.

[0080] For step S412, the global multi-head attention model is Global Multi-Head Attention. At each time step, the autoregressive language model should know the overall format of the text so that it can better output text that conforms to the format. Therefore, the format information should not be masked, that is, attention without mask is Global Multi-Head Attention. In addition, the Global Multi-Head Attention model also integrates content mapping information and format mapping information through self-attention.

[0081] For step S413, after obtaining the feature learning result, the feature learning result is compared with the above-mentioned expected training result, and the gap between the feature learning result and the above-mentioned expected training result, that is, the error value, is calculated. If the error value is too large, the model parameters of the word vector model are adjusted, and iterative calculations are performed through S411 and S412 until the above-mentioned error value meets the preset error range under a certain model parameter.

[0082] In one embodiment, inputting the content mapping information into the hidden multi-head self-attention model to obtain a context vector S411 includes:

[0083] S421: performing mask calculation on the content mapping information to obtain hidden mapping information;

[0084] S422: Input the hidden mapping information into the transformer layer of the hidden multi-head self-attention model to obtain the context vector.

[0085] This embodiment performs mask calculation on the content mapping information to obtain hidden mapping information, thereby realizing the conversion of the word vector model to the autoregressive language model.

[0086] Regarding step S421, specifically, in actual application, the autoregressive language model completes the generation of the entire text by iteratively generating tokens (words). For example, assuming the training sentence "The sun is really nice today", these words are sequentially generated by the model as "Today, the sun is really, not wrong". However, the above-mentioned word vector model BERT is a bidirectional transformer model, which knows the content of the following training sentence at each time step, and thus cannot be used directly as a generation model. This embodiment adds a mask matrix to the word vector model BERT to turn BERT into an autoregressive language model. The mask matrix is ​​added through the above-mentioned hidden multi-head self-attention model Masking Multi-Head Self-Attention.

[0087] For step S422, the content mapping information is masked to obtain the hidden mapping information maskembedding, and then the hidden mapping information mask embedding is input into the transformer layer of the hidden multi-head self-attention model MaskingMulti-Head Self-Attention to obtain the context vector context vector, thereby realizing the autoregressive language model conversion.

[0088] In one embodiment, inputting the context vector and the format mapping information into the global multi-head attention model to obtain a feature learning result of the context vector under the format mapping information (S413) includes:

[0089] S423: Perform self-attention model calculation on the context vector and the format mapping information through the global multi-head attention model to obtain a feature learning result after the context vector and the format mapping information are fused.

[0090] This embodiment performs self-attention model calculation on the context vector and format mapping information through a global multi-head attention model to obtain a feature learning result after the context vector and format mapping information are fused, thereby realizing the fusion of the format information corresponding to the training sentence and the sentence template.

[0091] Before step S423, taking the content mapping information H as an example, the content embedding information After inputting the Masking Multi-Head Self-Attention model, we get the context vector content vector

[0092] For step S423, the content vector and format mapping information Input them into GlobalMulti-Head Attention together to get the output feature learning results Complete one iteration; when iterative calculation is required, As a new Enter the above hidden multi-head self-attention model Masking Multi-Head Self-Attention and perform the above steps again to obtain After the next iteration is completed, the subsequent iteration steps are the same as above, so they will not be repeated here.

[0093] In one embodiment, the identifier information further includes a location identifier;

[0094] The configuration statement template S2 according to the identifier information and configuration rules in the configuration instruction includes:

[0095] S21: Identify the total number of characters in the training sentence, and generate position identifiers whose number is the same as the total number of characters, wherein each of the position identifiers includes a first digital identifier;

[0096] S22: sorting the location identifiers according to the size of the first digital identifiers in accordance with the configuration rule;

[0097] S23: Establishing a one-to-one correspondence between each of the position identifiers and each of the characters in sorted order, to obtain the sentence template corresponding to the training sentence.

[0098] This embodiment configures position identifiers to mark the local positions of different characters in the training sentences, so that the autoregressive language model accurately captures the end position of the sentence, thereby improving the accuracy of the format of the generated text.

[0099] In step S21, to indicate the length of a sentence and accurately capture the sentence end position in the model so that the model can recognize that the sentence generation has reached its end, this embodiment uses the position identifier pi to represent the local position of the character in the sentence, where i is the first numerical identifier. Taking "love is not love, bends with the remove to remove" as an example, the language template P including the position identifier is shown as follows:

[0100] P={p4,p3,p2,p1,p0,p6,p5,p4,p3,p2,p1,p0, <eos>}

[0101] For step S22, if the configuration rule is from large to small, then based on the above-mentioned language template P, the first digital identifier i in the position identifier pi in a sentence is set from large to small. If the sentence contains 5 characters, the first digital identifier i marks the 5 characters in sequence from 4 to 0, so that the position of each character in the sentence can be distinguished according to the numerical value of the first digital identifier i in the position identifier pi. When i=0, it can be considered that the sentence has ended.

[0102] In one embodiment, the identifier information further includes a sentence separator;

[0103] The configuration statement template S2 according to the identifier information and configuration rules in the configuration instruction includes:

[0104] S24: Identify the punctuation marks of the training sentence, and use the text content between two consecutive punctuation marks as segmented content;

[0105] S25: generating sentence separators whose number is the same as the number of the segmented contents, wherein each of the sentence separators includes a second digital identifier;

[0106] S26: sorting the statement separators according to the size of the second digital identifiers according to the configuration rule;

[0107] S27: Establishing a one-to-one correspondence between each of the sentence separators and each of the segmented contents in sorted order, and obtaining the sentence template corresponding to the training sentence.

[0108] This embodiment configures sentence segmentation identifiers for marking training sentences to which different characters belong, so that the autoregressive language model accurately identifies the sentence segmentation, thereby improving the accuracy of generating a format of text with echoing and symmetrical relationships.

[0109] For step S24, still taking "love is not love, bends with the remove to remove." as an example, there are two punctuation marks "", and ".", and the text content of "bends with the remove toremove" between the two symbols is used as a segmentation content of the training sentence. In a specific implementation, since the text at the beginning of a paragraph is not between two punctuation marks, for the text at the beginning of the paragraph, the first character of the training sentence is selected as the starting point, and the text content between the starting point and the next punctuation mark is also obtained as the segmentation content. Therefore, "love is not love" between the starting point and "", is also used as a segmentation content; that is, for the training sentence "love is not love, bends with theremove to remove.", the above two segmentation contents exist.

[0110] For step S25, whether it is poetry or couplets, sentences often need to correspond to each other and rhyme, which involves segmenting the sentence level. This embodiment uses the sentence separator sj to indicate which segmented content the character belongs to, and j is the second digital identifier.

[0111] For step S26, a configuration rule of the second digital identifiers from small to large may be adopted.

[0112] For step S27, still taking "love is not love, bends with the remove to remove." as an example, the language template S including the segmentation identifier is as follows:

[0113] S={s0,s0,s0,s0,s0,s1,s1,s1,s1,s1,s1,s1, <eos>}

[0114] During sentence generation, we learn rhyming sentence pairs. For example, the rhyme pattern of Shakespeare's sonnets is "ABAB CDCD EFEF GG." Of the 14 lines, lines 1 and 3 rhyme, and lines 2 and 4 rhyme. To generate language templates with rhyming patterns, the corresponding sentence segmentation representations must also be the same. For example:

[0115] S'={s0,s0,s0,s0,s0,s1,s1,s1,s1,s1,s1,s1,,

[0116] s0,s0,s0,s0,s0,s1,s1,s1,s1,s1,s1,s1, <eos>}

[0117] The format of ABAB is represented. After the autoregressive language model is trained according to the above language template, the target sentence in the text format to be generated can be input into the autoregressive language model, so that the model adapts the target sentence to the above format and obtains the output text in the above ABAB format.

[0118] Reference Figure 3 , this application also proposes a training device for an autoregressive language model, comprising:

[0119] The vector extraction module 100 is used to obtain a training sentence and extract a training text vector of the training sentence;

[0120] The sentence template configuration module 200 is used to obtain a configuration instruction and the expected training result of the training sentence, and configure the sentence template according to the identifier information in the configuration instruction and the configuration rules, wherein the identifier information includes one or more of ordinary text characters, punctuation characters, rhyme characters, separators, and terminators;

[0121] A mapping calculation module 300 is configured to input the training text vector and the sentence template into a word vector model, perform a mapping calculation on the training text vector and the sentence template using the word vector model, and obtain content mapping information of the training text vector under the sentence template;

[0122] The model generation training module 400 is used to iteratively train the word vector model based on the attention model according to the content mapping information and the expected training results to obtain an autoregressive language model based on the sentence template.

[0123] This embodiment designs the text generation format through configuration instructions to generate a sentence template corresponding to the format, thereby ensuring the format diversity and flexibility of text generation; the training text vector and the sentence template are mapped and calculated by the word vector model to obtain the content mapping information of the training text vector, so that the above-mentioned word vector model can generate a text of the corresponding format based on the text format of the sentence template, without the need for additional typesetting of the text, thereby improving the text generation efficiency; the word vector model is iteratively calculated by the attention model to obtain an autoregressive language model based on the above-mentioned sentence template, and since the sentence template can be set according to the configuration instructions, the coverage of the sentence template can be flexibly changed by setting the identifier information and configuration instructions with different configuration rules, so that it can be applied not only to finance, but also to a series of scenarios with fixed format text generation requirements such as poetry, lyrics, songs, etc., thereby improving the versatility of the autoregressive language model and the flexibility of format setting.

[0124] In one embodiment, a format mapping module 500 is further included, configured to:

[0125] Inputting the sentence template separately into a word vector model, performing mapping calculation on the sentence template through the word vector model, and obtaining format mapping information of the sentence template;

[0126] The model generation training module 400 is further used to:

[0127] Based on the attention model, the word vector model is iteratively trained according to the content mapping information, the expected training result and the format mapping information.

[0128] In one embodiment, the model generation training module 400 is further configured to:

[0129] Inputting the content mapping information into the hidden multi-head self-attention model to obtain a context vector;

[0130] Inputting the context vector and the format mapping information into the global multi-head attention model to obtain a feature learning result of the context vector under the format mapping information;

[0131] Calculate the error value between the feature learning result and the expected training result, adjust the model parameters of the word vector model according to the error value, and use the adjusted word vector model to iteratively calculate the content mapping information until the error value meets the preset error range.

[0132] In one embodiment, the model generation training module 400 is further configured to:

[0133] Performing mask calculation on the content mapping information to obtain hidden mapping information;

[0134] The hidden mapping information is input into the transformer layer of the hidden multi-head self-attention model to obtain the context vector.

[0135] In one embodiment, the model generation training module 400 is further configured to:

[0136] The global multi-head attention model is used to perform self-attention model calculation on the context vector and the format mapping information to obtain a feature learning result after the context vector and the format mapping information are fused.

[0137] In one embodiment, the statement template configuration module 200 is further configured to:

[0138] Identifying the total number of characters in the training sentence, and generating position identifiers having the same number as the total number of characters, wherein each of the position identifiers includes a first numerical identifier;

[0139] According to the configuration rule, the position identifiers are sorted according to the size of the first digital identifiers;

[0140] A one-to-one correspondence is established between each of the position identifiers and each of the characters in sorted order to obtain the sentence template corresponding to the training sentence.

[0141] In one embodiment, the statement template configuration module 200 is further configured to:

[0142] The configuring statement template according to the identifier information and configuration rules in the configuration instruction includes:

[0143] Identifying the punctuation marks of the training sentence, and taking the text content between two consecutive occurrences of the punctuation marks as segmented content;

[0144] generating sentence separators whose number is the same as the number of the segmented contents, wherein each of the sentence separators includes a second digital identifier;

[0145] According to the configuration rule, the statement separators are sorted according to the size of the second digital identifier;

[0146] A one-to-one correspondence is established between each of the sentence separators and each of the segmented contents in a sorted order, so as to obtain the sentence template corresponding to the training sentence.

[0147] Reference Figure 4 In the embodiment of the present application, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer design is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as a training method for an autoregressive language model. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a training method for an autoregressive language model is implemented. The training method of the autoregressive language model includes: configuring a sentence template, wherein the sentence template includes one or more of ordinary text, punctuation marks, rhymes, separators and terminators; obtaining a training sentence and extracting a training text vector of the training sentence; inputting the training text vector and the sentence template into a word vector model, performing a mapping calculation on the training text vector and the sentence template through the word vector model to obtain content mapping information of the training text vector under the sentence template; based on an attention model, iteratively training the word vector model according to the content mapping information and the expected training result to obtain an autoregressive language model based on the sentence template.

[0148] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, a method for training an autoregressive language model is implemented, comprising the steps of: configuring a sentence template, wherein the sentence template includes one or more of ordinary text, punctuation marks, rhymes, separators, and terminators; obtaining a training sentence, and extracting a training text vector of the training sentence; inputting the training text vector and the sentence template into a word vector model, and performing a mapping calculation on the training text vector and the sentence template through the word vector model to obtain content mapping information of the training text vector under the sentence template; based on an attention model, iteratively training the word vector model according to the content mapping information and the expected training result to obtain an autoregressive language model based on the sentence template.

[0149] The training method of the autoregressive language model executed above, this embodiment designs the text generation format through configuration instructions to generate a sentence template corresponding to the format, thereby ensuring the format diversity and flexibility of text generation; the training text vector and the sentence template are mapped and calculated by the word vector model to obtain the content mapping information of the training text vector, so that the above word vector model can generate a text of the corresponding format based on the text format of the sentence template, without the need for additional typesetting of the text, thereby improving the text generation efficiency; the word vector model is iteratively calculated by the attention model to obtain an autoregressive language model based on the above sentence template, and since the sentence template can be set according to the configuration instructions, the coverage of the sentence template can be flexibly changed by setting the identifier information and configuration instructions with different configuration rules, so that it can be applied not only to finance, but also to a series of scenarios with fixed format text generation requirements such as poetry, lyrics, songs, etc., thereby improving the versatility of the autoregressive language model and the flexibility of format setting.

[0150] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and RAM bus dynamic RAM (RDRAM), etc.

[0151] It should be noted that, in this document, the terms "comprise," "include," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, apparatus, article, or method comprising a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, apparatus, article, or method. In the absence of further limitations, an element defined by the phrase "comprising a..." does not preclude the presence of other identical elements in the process, apparatus, article, or method comprising the element.

[0152] The above description is only a preferred embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.< / eos> < / eos> < / eos> < / eos> < / eos> < / eos>

Claims

1. A method for training an autoregressive language model, characterized in that: The method comprises: Obtaining a training sentence and extracting a training text vector of the training sentence; Obtaining a configuration instruction and an expected training result of the training sentence, and configuring a sentence template according to identifier information in the configuration instruction and configuration rules, wherein the identifier information includes one or more of ordinary text characters, punctuation characters, rhyme characters, separators, and terminators; the expected training result is a standard result manually generated according to the training sentence; Inputting the training text vector and the sentence template into a word vector model, performing a mapping calculation on the training text vector and the sentence template through the word vector model to obtain content mapping information of the training text vector under the sentence template; Based on the attention model, iteratively train the word vector model according to the content mapping information and the expected training result to obtain an autoregressive language model based on the sentence template; Before the step of iteratively training the word vector model according to the content mapping information and the expected training result to obtain the autoregressive language model based on the sentence template, the following steps are included: The weight value of the corresponding part of the content mapping information is obtained through softmax, and a weight is added to each content mapping information in the encoding stage, thereby obtaining the importance of each part in the content mapping information sequence; Through the attention model, important parts of the content mapping information are selected as the input for encoding.

2. The method for training an autoregressive language model according to claim 1, wherein: Before iteratively training the word vector model based on the attention model according to the content mapping information and the expected training result, the method further includes: Inputting the sentence template separately into a word vector model, performing mapping calculation on the sentence template through the word vector model, and obtaining format mapping information of the sentence template; The iterative training of the word vector model based on the attention model according to the content mapping information and the expected training result includes: Based on the attention model, the word vector model is iteratively trained according to the content mapping information, the expected training result and the format mapping information.

3. The method for training an autoregressive language model according to claim 2, wherein: The attention model includes a hidden multi-head self-attention model and a global multi-head attention model; The iterative training of the word vector model based on the attention model according to the content mapping information and the expected training result includes: Inputting the content mapping information into the hidden multi-head self-attention model to obtain a context vector; Inputting the context vector and the format mapping information into the global multi-head attention model to obtain a feature learning result of the context vector under the format mapping information; Calculate the error value between the feature learning result and the expected training result, adjust the model parameters of the word vector model according to the error value, and use the adjusted word vector model to iteratively calculate the content mapping information until the error value meets the preset error range.

4. The method for training an autoregressive language model according to claim 3, wherein: Inputting the content mapping information into the hidden multi-head self-attention model to obtain a context vector includes: Performing mask calculation on the content mapping information to obtain hidden mapping information; The hidden mapping information is input into the transformer layer of the hidden multi-head self-attention model to obtain the context vector.

5. The method for training an autoregressive language model according to claim 3, wherein: The inputting the context vector and the format mapping information into the global multi-head attention model to obtain a feature learning result of the context vector under the format mapping information includes: The global multi-head attention model is used to perform self-attention model calculation on the context vector and the format mapping information to obtain the feature learning result after the context vector and the format mapping information are fused.

6. The method for training an autoregressive language model according to claim 1, wherein: The identifier information also includes a location identifier; The configuring statement template according to the identifier information and configuration rules in the configuration instruction includes: Identifying the total number of characters in the training sentence, and generating position identifiers having the same number as the total number of characters, wherein each of the position identifiers includes a first numerical identifier; According to the configuration rule, the position identifiers are sorted according to the size of the first digital identifiers; A one-to-one correspondence is established between each of the position identifiers and each of the characters in sorted order to obtain the sentence template corresponding to the training sentence.

7. The method for training an autoregressive language model according to claim 1, wherein: The identifier information also includes a sentence separator; The configuring statement template according to the identifier information and configuration rules in the configuration instruction includes: Identifying the punctuation marks of the training sentence, and using the text content between two consecutive occurrences of the punctuation marks as segmented content; generating sentence separators whose number is the same as the number of the segmented contents, wherein each of the sentence separators includes a second digital identifier; According to the configuration rule, the statement separators are sorted according to the size of the second digital identifier; A one-to-one correspondence is established between each of the sentence separators and each of the segmented contents in a sorted order, so as to obtain the sentence template corresponding to the training sentence.

8. A training device for an autoregressive language model, characterized in that: include: A vector extraction module, configured to obtain a training sentence and extract a training text vector of the training sentence; a sentence template configuration module, configured to obtain a configuration instruction and an expected training result of the training sentence, and configure a sentence template according to identifier information in the configuration instruction and configuration rules, wherein the identifier information includes one or more of ordinary text characters, punctuation characters, rhyme characters, separators, and terminators; and the expected training result is a standard result manually generated based on the training sentence; A mapping calculation module, configured to input the training text vector and the sentence template into a word vector model, perform mapping calculation on the training text vector and the sentence template through the word vector model, and obtain content mapping information of the training text vector under the sentence template; A model generation training module is used to iteratively train the word vector model based on the attention model according to the content mapping information and the expected training results to obtain an autoregressive language model based on the sentence template; The iterative training of the word vector model according to the content mapping information and the expected training result to obtain the autoregressive language model based on the sentence template includes: The weight value of the corresponding part of the content mapping information is obtained through softmax, and a weight is added to each content mapping information in the encoding stage, thereby obtaining the importance of each part in the content mapping information sequence; Through the attention model, important parts of the content mapping information are selected as the input for encoding.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images