Language model training method and natural language processing method and device

Abstract

The invention provides a language model training method and a natural language processing method and device. The method comprises the following steps: performing natural language processing on an input text by utilizing a language model to obtain an output text; scoring the output text by using a reward model to obtain an original score; calculating the repetition degree of the output text; determining a target score according to the original score and the repetition degree; and adjusting parameters of the language model according to the target score.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for training a language model, and a method and device for natural language processing. Background Art

[0002] ChatGPT is an intelligent dialogue system based on natural language processing and machine learning technology. It can simulate human language communication and realize intelligent dialogue with users. It has a great impact and has been widely used in intelligent customer service, chatbots, voice assistants and other fields, providing people with more convenient, fast and intelligent services. Since its launch last year, it has triggered a wave of large model research.

[0003] However, the use of large models often encounters the problem of repeated generation, which greatly affects the generation quality of large models. Repeated generation can be repeated at the Token (character) / Phrase (word) level, that is, the large model repeatedly generates a character or a word. For example, in the e-commerce translation scenario, "steckdose steckdose steckdose steckdose steckdose steckdose steckdose..." will appear; it can also be repeated at the Sentence level, that is, the large model repeatedly generates a sentence. For example, in the multimodal large model image understanding, the generated results may repeat part of the content of the image, such as "This is a cup, this is a cup..." Therefore, how to avoid the repetition of large models is a problem that needs to be solved. Summary of the invention

[0004] In order to overcome the problems existing in the related art, this specification provides a method for training a language model, a method and a device for natural language processing.

[0005] According to a first aspect of an embodiment of the present specification, a method for training a language model is provided, the method comprising: performing natural language processing on an input text using a language model to obtain an output text; scoring the output text using a reward model to obtain an original score; calculating the repetition of the output text; determining a target score based on the original score and the repetition; and adjusting the parameters of the language model based on the target score.

[0006] According to the second aspect of the embodiments of this specification, a natural language processing method is provided, the method comprising: inputting a question text into a language model to obtain answer data, the language model being trained according to the method for training a language model according to the first aspect or any corresponding implementation manner thereof.

[0007] According to a third aspect of an embodiment of this specification, a device for training a language model is provided, the device comprising: a processing module, used to perform natural language processing on an input text using a language model to obtain an output text; a scoring module, used to score the output text using a reward model to obtain an original score; a repetition calculation module, used to calculate the repetition of the output text; a target score determination module, used to determine a target score based on the original score and the repetition; and an adjustment module, used to adjust the parameters of the language model based on the target score.

[0008] According to the fourth aspect of the embodiments of this specification, a natural language processing device is provided, the device comprising: an input module for inputting a question text into a language model to obtain answer data, and the language model is trained according to the method for training a language model of the embodiments of this specification.

[0009] According to a fifth aspect of the embodiments of this specification, an electronic device is provided, including:

[0010] processor;

[0011] a memory for storing processor-executable instructions;

[0012] The processor is configured to execute the method for training a language model and / or the method for natural language processing according to the embodiments of this specification.

[0013] According to the sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which computer instructions are stored, and the computer instructions are used to enable a computer to execute the method for training a language model and / or the natural language processing method according to the embodiments of this specification.

[0014] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0015] In the embodiment of this specification, the target score is determined according to the score of the reward model and the repetition degree, and the language model adjusts the model parameters according to the target score to change its generation strategy. As a result, the language model can gradually learn how to avoid repetition, thereby reducing the phenomenon of model repetition generation and generating higher quality text that is more in line with human preferences.

[0016] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.

[0018] Figure 1 It is a schematic diagram of a system architecture shown in this specification according to an exemplary embodiment.

[0019] Figure 2 It is a flowchart of a method for training a language model according to an exemplary embodiment of the present specification.

[0020] Figure 3 It is a hardware structure diagram of the computer device where the language model training device or the natural language processing device of the embodiment of this specification is located.

[0021] Figure 4 It is a block diagram of a device for training a language model according to an exemplary embodiment of the present specification.

[0022] Figure 5 It is a block diagram of a natural language processing device shown in this specification according to an exemplary embodiment. DETAILED DESCRIPTION

[0023] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this specification. Instead, they are merely examples of devices and methods consistent with some aspects of this specification as detailed in the appended claims.

[0024] The terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit this specification. The singular forms "a", "the" and "the" used in this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0025] It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0026] Next, the embodiments of this specification are described in detail.

[0027] The following combination Figure 1The system architecture of the method and device for training a language model in the embodiments of this specification is described. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of this specification can be applied, to help those skilled in the art understand the technical content of this specification, but it does not mean that the embodiments of this specification cannot be used in other devices, systems, environments or scenarios.

[0028] Figure 1 It is a schematic diagram of a system architecture shown in this specification according to an exemplary embodiment.

[0029] like Figure 1 As shown, the system architecture may include, for example, a terminal device, a network and a server. The network is used to provide a medium for a communication link between the terminal device and the server. The network may include various connection types, such as wired and / or wireless communication links, etc.

[0030] Users can use terminal devices to interact with servers through the network to receive or send messages, etc. Various communication client applications can be installed on the terminal devices, such as service access clients, web browser applications, search applications, instant messaging tools, email clients and / or social platform software.

[0031] The terminal device may be any electronic device having a display screen and supporting web browsing, including but not limited to a smart phone, a tablet computer, a laptop computer, a desktop computer, and the like.

[0032] The server may be a server that provides various services, such as a background management server that provides support for the content browsed by the user using the terminal device. The background management server may analyze and process the received data such as the user request, and feed back the processing results (such as the web page, information, or data obtained or generated according to the user request) to the terminal device.

[0033] The following is a detailed description of the method for training a language model provided in the embodiments of this specification. Figure 2 As shown, Figure 2 This is a flowchart of a method for training a language model according to an exemplary embodiment of this specification. The method can be applied to the server shown above, for example. The method for training a language model provided in the embodiment of this specification may include the following steps.

[0034] In step 210, the input text is subjected to natural language processing using a language model to obtain an output text.

[0035] According to the embodiments of this specification, the language model can be used to perform natural language processing on the input text, such as machine translation, public opinion monitoring, automatic summarization, opinion extraction, text classification, question answering, text semantic comparison, etc. The language model can be based on a large model.

[0036] In step 220, the output text is scored using the reward model to obtain an original score.

[0037] According to an embodiment of the present specification, the reward model is a machine learning model used to evaluate the quality of generated text and provide feedback to the generation model. It determines whether the generated text meets the requirements based on human preferences and evaluation criteria, and gives corresponding reward or penalty signals. The reward model can receive the generated text as input and output a reward score. This reward score reflects the degree of match between the quality of the generated text and human preferences.

[0038] In step 230, the repetition degree of the output text is calculated.

[0039] According to an embodiment of the present specification, the repetition degree may indicate the repetition degree of characters in the output text.

[0040] In step 240, a target score is determined based on the original score and the repetition rate.

[0041] According to the embodiments of this specification, for example, a repetition penalty score may be determined based on the repetition degree, and then the repetition penalty score is subtracted from the original score to obtain a target score.

[0042] If you want the reward model to have the ability to evaluate repeated features, you need to construct a large amount of repeated data and score the repeated data for training the reward model.

[0043] According to the embodiments of the present specification, by setting a duplicate penalty score based on the reward model score, there is no need for duplicate data to train the reward model, thereby avoiding the workload of constructing duplicate data and scoring duplicate data.

[0044] In step 250, the parameters of the language model are adjusted according to the target score.

[0045] According to an embodiment of the present specification, if the target score is lower than the preset score, it means that the language model does not meet the standard, and the parameters of the language model can be adjusted, and then return to step 210. If the target score is higher than or equal to the preset score, it means that the language model has met the standard and the training can be ended. The preset score can be set according to actual needs.

[0046] During the training process, the language model can adjust the model parameters according to the target score to change its generation strategy. If the output text of the language model obtains a higher target score, the generation model will tend to generate similar text. If the generated text obtains a lower target score, the language model will adjust the model parameters accordingly to change its generation strategy to avoid generating similar text. In this way, the language model can gradually learn how to avoid repetition, thereby reducing the phenomenon of repeated generation of the model and generating higher quality text that is more in line with human preferences.

[0047] According to the embodiments of this specification, after the language model is trained, the language model can be applied to intelligent dialogue scenarios. For example, a question text can be input into the language model, and the language model can perform natural language processing on the question text to obtain answer data. Since the language model has been trained as described above, there are fewer repetitive contents in the answer data, and the answer quality is higher.

[0048] According to an embodiment of the present specification, for example, repeated text segments in the output text may be determined, and then the ratio between the total number of words in the repeated text segments and the total number of words in the output text is calculated as the repetition degree of the output text.

[0049] Optionally, multiple first text segments can be extracted from the output text, each of which includes n consecutively appearing words, where n is a positive integer. Then, a first text segment whose number of repeated appearances is greater than or equal to a preset number of times among the multiple first text segments is determined as a repeated text segment. The preset number of times can be set according to actual needs.

[0050] Exemplarily, in this embodiment, an N-gram algorithm may be used to extract the first text segment. For example, when n is 6, multiple 6-grams may be extracted from the output text, and each 6-gram is a first text segment.

[0051] The above method can identify text segments containing the same text in the output text. However, sometimes the repeated information in the output text is not exactly the same, such as "In autumn, the Fragrant Hills in Beijing are so beautiful that people can't help but linger."; "In autumn, the Fragrant Hills in Beijing are so beautiful that people can't help but linger."; "In autumn, the Fragrant Hills in Beijing are so beautiful that people can't help but linger."; "In autumn, the Fragrant Hills in Beijing are so beautiful that people can't help but linger.". These longer sentences are different from each other, but very similar. If two or three of them appear at the same time in the language model output, they are also repeated.

[0052] Based on this, multiple second text segments can be extracted from the output text, each second text segment includes m words that appear continuously, and m is a positive integer. The distances between the multiple second text segments are calculated. Then, the second text segments whose distances between the multiple second text segments are less than or equal to the preset distance are determined as repeated text segments. The preset distance can be set according to actual needs.

[0053] For example, in this embodiment, an N-gram algorithm may be used to extract the second text segment. For example, when m is 22, multiple 22-grams may be extracted from the output text, and each 22-gram is a second text segment.

[0054] Exemplarily, in this embodiment, the distance may include, for example, the Levenshtein distance. The Levenshtein distance is a type of edit distance and is a standard metric for measuring the difference between two strings. Specifically, it is the minimum number of edit operations required to convert one string into another, i.e., the number of times characters are inserted, deleted, or replaced.

[0055] According to another embodiment of the present disclosure, n can be 6, and the preset number of times can be 3 times. m can be 22, and the preset distance can be 1. 6-gram (a text segment consisting of 6 consecutive characters, i.e., a first text segment) and 22-gram (a text segment consisting of 22 consecutive characters, i.e., a second text segment) can be extracted based on the output text of the language model. The number of times that all 6-grams appear the same is calculated, and the Levenshtein distance between all 22-grams is calculated. If a 6-gram appears the same number of times greater than or equal to 3, the 6-gram is regarded as a repeated text segment. If the Levenshtein distance between two 22-grams is less than or equal to 1, the two 22-grams are regarded as repeated text segments.

[0056] For example, if we input “what hairstyle is suitable for an oval face” into the language model, the output text output by the language model is: “the hairstyle for an oval face should be to comb the hair into a shallow shawl hairstyle, which can make the oval face more spacious without making the face too narrow and long. In addition, the hairstyle for an oval face can also choose some shawl hairstyles, such as shawl comb, shawl comb, shawl comb comb, etc. These hairstyles can make the oval face more spacious without making the face too narrow and long. In short, the hairstyle for an oval face should be to comb the hair into a shallow shawl hairstyle, which can make the oval face more spacious without making the face too narrow and long".

[0057] The output text contains "hairstyles for oval faces" and "can make the oval" which appear 3 times each. In addition, the Levenshtein distance between "hairstyles for oval faces should be combed into a light shawl hairstyle" and "hairstyles for oval faces should be combed into a light shawl hairstyle" is 1, and the Levenshtein distance between "this can make the oval face wider without making the face shape become" and "this can make the oval face wider without making the face shape become" is 0.

[0058] To further measure the severity of repetition, we can mark all 6-grams with a repetition frequency greater than or equal to 3 and 22-grams with a Levenshtein distance less than or equal to 1, count their total number of words, and calculate the ratio of their total number of words to the total number of words in the output text. This ratio can be used to measure the severity of repetition. The closer the ratio is to 1, the more serious the repetition phenomenon is.

[0059] For example, in this embodiment, the repeated text may include: "The hairstyle for an oval face should be a light shawl hairstyle, which can make the oval face wider without making the face too narrow and long.", "Hairstyle for an oval face", "It can make the oval face wider without making the face too narrow and long.", "The hairstyle for an oval face should be a light shawl hairstyle, which can make the oval face wider without making the face too narrow and long.", a total of 133 words. The total number of words in the output text is 174, so the ratio of repeated generation is 133 / 174. This ratio is used as the repetition degree.

[0060] According to an embodiment of the present specification, for example, the product of a preset reference penalty score and the degree of repetition may be calculated to obtain a repetition penalty score, and then the difference between the original score and the repetition penalty score may be calculated to obtain a target score.

[0061] For example, the target score can be calculated according to the following formula:

[0062] RM_Score'=RM_Score–A*repetition_ratio

[0063] Among them, RM Score is the original score, RM_Score' is the target score, A is the baseline penalty score, and repetition_ratio is the repetition rate. A*repetition_ratio is the repetition penalty score, which can be adjusted by repetition_ratio, that is, the greater the repetition rate, the greater the penalty on the score.

[0064] Optionally, combined with the scoring mechanism of the above target score, the language model can be reinforced trained by the PPO (Proximal Policy Optimization) method so that the language model actively avoids repeated generation. PPO includes four models: the actor model (generative model, also known as the policy model, responsible for generating responses), the reference model (used to calculate the KL penalty using fixed parameters), the reward model (provides a reward score for the large model response) and the critic model (used to learn the value of each token). When the language model is reinforced trained by the PPO method, the language model acts as an actor model, and both the actor model and the critic model are updated by the standard PPO algorithm.

[0065] Corresponding to the embodiments of the aforementioned method, this specification also provides embodiments of a device for training a language model, a natural language processing device, and a terminal used therein.

[0066] The embodiments of the apparatus for training a language model or a natural language processing apparatus of this specification can be applied to a computer device, such as a server or a terminal device. The apparatus embodiments can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a logically sensed apparatus, it is formed by the processor in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory and running them. From the hardware level, such as Figure 3 As shown, it is a hardware structure diagram of the computer device where the language model training device or natural language processing device of the embodiment of this specification is located, except Figure 3 In addition to the processor 310, memory 330, network interface 320, and non-volatile memory 340 shown, the server or electronic device where the device 331 is located in the embodiment may also include other hardware according to the actual function of the computer device, which will not be described in detail.

[0067] like Figure 4 As shown, Figure 4 is a block diagram of a device for training a language model according to an exemplary embodiment of the present specification, the device comprising:

[0068] The processing module 410 is used to perform natural language processing on the input text using the language model to obtain an output text;

[0069] A scoring module 420 is used to score the output text using the reward model to obtain an original score;

[0070] A repetition calculation module 430, used to calculate the repetition of the output text;

[0071] A target score determination module 440 is used to determine a target score according to the original score and the repetition degree;

[0072] The adjustment module 450 is used to adjust the parameters of the language model according to the target score.

[0073] Optionally, the repetition calculation module may include:

[0074] A repeated text segment determination submodule, used for determining repeated text segments in the output text;

[0075] The ratio calculation submodule is used to calculate the ratio between the total number of words in the repeated text segment and the total number of words in the output text as the repetition degree of the output text.

[0076] Optionally, the repeated text segment determination submodule may include:

[0077] A first extraction unit, used to extract a plurality of first text segments from the output text, each first text segment includes n consecutively appearing words, where n is a positive integer;

[0078] The first determining unit is used to determine, as a repeated text segment, a first text segment whose number of repeated appearances is greater than or equal to a preset number of times among the plurality of first text segments.

[0079] Optionally, the repeated text segment determination submodule may include:

[0080] A second extraction unit is used to extract a plurality of second text segments from the output text, each second text segment includes m consecutively appearing characters, where m is a positive integer;

[0081] A distance calculation unit, used for calculating the distance between the plurality of second text segments;

[0082] The second determining unit is configured to determine, as repeated text segments, a plurality of second text segments whose distances between the second text segments are less than or equal to a preset distance.

[0083] Optionally, the target score determination module may include:

[0084] The first calculation submodule is used to calculate the product of a preset benchmark penalty score and a repetition degree to obtain a repetition penalty score;

[0085] The second calculation submodule is used to calculate the difference between the original score and the repetition penalty score to obtain the target score.

[0086] According to the embodiments of this specification, the target score is determined according to the score and repetition of the reward model, and the language model adjusts the model parameters according to the target score to change its generation strategy. As a result, the language model can gradually learn how to avoid repetition, thereby reducing the phenomenon of model repetition generation and generating higher quality text that is more in line with human preferences.

[0087] like Figure 5 As shown, Figure 5 1 is a block diagram of a natural language processing device according to an exemplary embodiment of the present specification, the device comprising:

[0088] The input module 510 is used to input the question text into the language model to obtain answer data. The language model is trained according to the method for training a language model in the embodiment of this specification.

[0089] Accordingly, this specification also provides an electronic device, which includes a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the method for training a language model and / or the natural language processing method according to the embodiments of this specification.

[0090] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0091] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0092] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0093] Those skilled in the art will readily appreciate other embodiments of the specification after considering the specification and practicing the invention claimed herein. The specification is intended to cover any variations, uses or adaptations of the specification that follow the general principles of the specification and include common knowledge or customary techniques in the art that are not claimed in the specification. The specification and examples are to be considered exemplary only, and the true scope and spirit of the specification are indicated by the following claims.

[0094] It should be understood that the present description is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present description is limited only by the appended claims.

[0095] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

Claims

1. A method for training a language model, characterized in that: The method comprises: Use the language model to perform natural language processing on the input text to obtain the output text; Scoring the output text using a reward model to obtain an original score; Calculating the repetition of the output text; Determining a target score according to the original score and the repetition degree; According to the target score, the parameters of the language model are adjusted.

2. The method according to claim 1, characterized in that The calculating the repetition degree of the output text comprises: determining repeated text segments in the output text; The ratio between the total number of words in the repeated text segment and the total number of words in the output text is calculated as the repetition degree of the output text.

3. The method according to claim 2, characterized in that Determining the repeated text segments in the output text comprises: Extracting a plurality of first text segments from the output text, each of the first text segments includes n consecutively appearing words, where n is a positive integer; A first text segment whose number of repeated occurrences is greater than or equal to a preset number of times among the multiple first text segments is determined as the repeated text segment.

4. The method according to claim 2, characterized in that: Determining the repeated text segments in the output text comprises: Extracting a plurality of second text segments from the output text, each of the second text segments includes m consecutively appearing words, where m is a positive integer; calculating distances between the plurality of second text segments; Second text segments whose distances between the plurality of second text segments are less than or equal to a preset distance are determined as the repeated text segments.

5. The method according to claim 1, characterized in that Determining the target score according to the original score and the repetition degree includes: Calculate the product of a preset benchmark penalty score and the repetition degree to obtain a repetition penalty score; The difference between the original score and the repetition penalty score is calculated to obtain the target score.

6. A natural language processing method, characterized in that: The method comprises: The question text is input into a language model to obtain answer data, wherein the language model is trained according to the method according to any one of claims 1-5.

7. A device for training a language model, characterized in that: The device comprises: A processing module, used to perform natural language processing on input text using a language model to obtain output text; A scoring module, used for scoring the output text using a reward model to obtain an original score; A repetition calculation module, used for calculating the repetition of the output text; A target score determination module, used to determine a target score according to the original score and the repetition degree; An adjustment module is used to adjust the parameters of the language model according to the target score.

8. The device according to claim 7, characterized in that The repetition calculation module comprises: A repeated text segment determination submodule, used to determine repeated text segments in the output text; The ratio calculation submodule is used to calculate the ratio between the total number of words in the repeated text segment and the total number of words in the output text as the repetition degree of the output text.

9. The device according to claim 8, characterized in that The repeated text segment determination submodule includes: A first extraction unit, configured to extract a plurality of first text segments from the output text, each of the first text segments comprising n consecutively appearing words, where n is a positive integer; The first determining unit is configured to determine, among the plurality of first text segments, a first text segment whose number of repeated occurrences is greater than or equal to a preset number of times as the repeated text segment.

10. A natural language processing device, characterized in that: The device comprises: An input module is used to input the question text into a language model to obtain answer data, wherein the language model is trained according to the method according to any one of claims 1-5.

Images