Language model training and natural language task processing method and apparatus, and related device
By adding placeholders and extracting global and local semantic features during the language model training process, the problem of existing language models ignoring local features is solved, resulting in more accurate feature extraction and downstream task processing.
Patent Information
- Application Number
- CN202210819636.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-13
- Publication Date
- 2025-11-11
- Estimated Expiration
- 2042-07-13
AI Technical Summary
Existing language models can only extract global features of the input text during training, while ignoring local features, resulting in inaccurate features and affecting the processing results of downstream tasks.
By adding first and second placeholders to the training text and using a language model to process the character encoding features, global and local semantic features are extracted, and the target character corresponding to the masked character is predicted until the set training termination condition is met, the trained language model is obtained.
This technology enables language models to simultaneously extract global and local semantic features from input text, resulting in richer and more accurate features. This improves the accuracy of downstream task processing results, especially the ability to identify and judge similar semantic text.
Smart Images
Figure CN115186675B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of natural language processing technology, and more specifically, to a method, apparatus and related equipment for language model training and natural language task processing. Background Technology
[0002] Natural Language Processing (NLP) is a crucial research area in artificial intelligence. Its core objective is to teach machines to "understand" the natural language used by humans in everyday life, such as inputting two sentences and having the machine determine whether they are semantically related. Traditional deep learning employs supervised learning training methods, relying on large amounts of precisely labeled data. However, precisely labeled data is scarce in practical applications and suffers from problems such as inconsistent labeling standards, long labeling times, and poor labeling accuracy. Therefore, unsupervised language models have emerged. These models first learn general semantic understanding capabilities using a large amount of unlabeled corpus, and then fine-tune them using a small amount of precisely labeled data. This allows deep learning models to demonstrate performance comparable to or even surpassing human capabilities on numerous tasks.
[0003] The most commonly used language model, taking Transformer as an example, adds a [CLS] placeholder at the beginning of the input text and models contextual semantic information based on a self-attention mechanism. Through pre-training, the [CLS] placeholder encodes the global information of the input text. Based on this, the encoded information corresponding to the [CLS] placeholder is used for downstream tasks, such as classification. However, the inventors of this case found that although this training method can encode the global information of the input text through the [CLS] placeholder, it does not consider the impact of local differences in the input text on the contextual semantics. This leads to the model's inability to distinguish similar texts well. For example, consider the two texts "high temperature and low pressure" and "high pressure and low temperature". These two texts have completely opposite semantics. However, according to the existing language model training method, it is assumed that the two texts contain the same characters and are very similar in form. Therefore, the global encoded features of the two texts are considered to be similar. After downstream comparison model processing, the result is given that the two texts are semantically similar. It is clear that this result is wrong. Summary of the Invention
[0004] In view of the above problems, this application is proposed to provide a language model training and natural language task processing method, apparatus and related equipment to solve the problem that existing language models can only extract global features of input text while ignoring local features, resulting in inaccurate extracted features, which in turn affects the processing results of downstream tasks.
[0005] Firstly, a language model training method is provided, including:
[0006] The training text is processed by adding placeholders and random character masking to obtain the edited training text. The target characters randomly selected in the training text are replaced with set mask characters in the edited training text. A first placeholder is added before the starting character of the first training text and a second placeholder is added at the end of each training text.
[0007] Determine the encoding features of each character in the edited training text, including the first and second placeholders and the mask character;
[0008] The encoding features of each character are processed using a language model to obtain global semantic features corresponding to the first placeholder and local semantic features corresponding to the second placeholder; the global semantic features are used to represent the overall semantics of the training text, and the local semantic features are used to represent the local semantics of the training text.
[0009] Based on the local semantic features corresponding to the second placeholder at the end of the training text where the mask character is located, the target character corresponding to the mask character is predicted until the set training termination condition is met, and the trained language model is obtained.
[0010] Secondly, a natural language processing method is provided, including:
[0011] Get the text of the task to be processed;
[0012] The language model obtained from the above training is used to process the task text to obtain the global semantic features corresponding to the first placeholder and the local semantic features corresponding to the second placeholder output by the language model.
[0013] The global semantic features and local semantic features are fused to obtain fused semantic features;
[0014] Using a pre-configured task processing module, a specified task is processed based on the fused semantic features to obtain the task processing result.
[0015] Thirdly, a language model training device is provided, comprising:
[0016] The training text editing unit is used to add placeholders and perform random character masking on the training text to obtain the edited training text. The target characters randomly selected in the training text are replaced with set mask characters in the edited training text. A first placeholder is added before the starting character of the first training text and a second placeholder is added at the end of each training text.
[0017] The encoding feature determination unit is used to determine the encoding features of each character in the edited training text, including the first and second placeholders and the mask character.
[0018] The encoding feature processing unit is used to process the encoding features of each character using a language model to obtain global semantic features corresponding to the first placeholder and local semantic features corresponding to the second placeholder; the global semantic features are used to represent the semantics of the training text as a whole, and the local semantic features are used to represent the semantics of a part of the training text.
[0019] The prediction unit is used to predict the target character corresponding to the mask character based on the local semantic features corresponding to the second placeholder at the end of the training text where the mask character is located, until the set training termination condition is met, and the trained language model is obtained.
[0020] Fourthly, a natural language task processing apparatus is provided, comprising:
[0021] The task text acquisition unit is used to acquire the task text to be processed.
[0022] The task text processing unit is used to process the task text using the language model trained above, and to obtain the global semantic features corresponding to the first placeholder and the local semantic features corresponding to the second placeholder output by the language model.
[0023] The feature fusion unit is used to fuse the global semantic features and local semantic features to obtain fused semantic features;
[0024] The task processing unit is used to perform specified task processing based on the fused semantic features using a pre-configured task processing module to obtain task processing results.
[0025] Fifthly, an electronic device is provided, comprising: a memory and a processor;
[0026] The memory is used to store programs;
[0027] The processor is used to execute the program to implement the language model training method as described above, or to implement the various steps of the natural language task processing method as described above.
[0028] In a sixth aspect, a storage medium is provided on which a computer program is stored, which, when executed by a processor, implements the language model training method as described above, or implements the various steps of the natural language task processing method as described above.
[0029] By employing the above technical solution, this application differs from the traditional method of predicting target characters using the encoding features corresponding to mask characters. Instead, it utilizes the encoding features corresponding to the second placeholder at the end of the training text where the mask character is located to predict the target character. This encodes local information from the training text into the second placeholder, enabling the encoding features corresponding to the second placeholder to represent local semantics within the training text. In this application, the encoding features corresponding to the second placeholder are also defined as local semantic features. Simultaneously, this application can also extract the overall semantics of the training text, i.e., global semantic features, through the first placeholder. In summary, the language model trained according to the method of this application can extract global semantic features of the input text through the first placeholder and local semantic features of the input text through the second placeholder. Compared to existing technologies that can only extract global semantic features, this application additionally extracts local semantic features, resulting in final features that simultaneously contain both global and local semantic features, making the features richer and more accurate.
[0030] Based on this, the downstream task processing module can process the specified task based on the fused semantic features obtained by fusing global and local semantic features. Since the fused semantic features contain richer and more accurate information, the downstream task processing results are also more accurate, especially for the identification and judgment of similar semantic texts. Based on the fused semantic features, it is possible to better distinguish between texts with similar semantics. Attached Figure Description
[0031] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:
[0032] Figure 1 A flowchart illustrating the language model training method provided in this application embodiment;
[0033] Figure 2 This example illustrates a language model training process.
[0034] Figure 3 This example illustrates a joint training process between a language model and a downstream task module.
[0035] Figure 4 This is a schematic diagram of the structure of a language model training device disclosed in an embodiment of this application;
[0036] Figure 5 This is a schematic diagram of the structure of a natural language task processing device disclosed in an embodiment of this application;
[0037] Figure 6This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0038] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0039] This application provides a language model training scheme and a downstream natural language task processing scheme based on the language model. The language model training scheme of this application can extract global and local semantic features of the input text based on the trained language model, making the extracted features richer and more accurate. On this basis, the global and local semantic features are fused, and the fused semantic features are used for downstream natural language tasks, such as classification tasks: semantic similarity judgment, next sentence prediction, sentence classification, and natural language inference.
[0040] The proposed solution can be implemented based on a terminal with data processing capabilities, such as a mobile phone, computer, server, or cloud platform.
[0041] Next, combined Figure 1 The language model training method of this application may include the following steps:
[0042] Step S100: Add placeholders and perform random character masking on the training text to obtain the edited training text.
[0043] Depending on the downstream natural language task, the number of training texts input to the language model at one time can be one or more. Randomly selected target characters in the training text are replaced with a set mask character in the edited training text; for example, the set mask character can be represented by [MASK]. Furthermore, a first placeholder is added before the starting character of the first training text in the edited training text, and a second placeholder is added at the end of each training text.
[0044] The first placeholder can be represented by [CLS], and the second placeholder can be represented by [SEP].
[0045] Understandably, if there is only one training text, simply add [CLS] before the training text and [SEP] after it. If there are two or more training texts, add [CLS] before the first training text and [SEP] at the end of each training text, in the following format:
[0046] [CLS] Training Text 1 [SEP] Training Text 2 [SEP] Training Text 3 [SEP]...
[0047] The two training texts are separated by a second placeholder [SEP].
[0048] Step S110: Determine the encoding features of each character in the edited training text, including the first and second placeholders and the mask character.
[0049] Specifically, the edited training text contains several different types of characters: first and second placeholders, mask characters, and all characters contained in the training text. For each character in the edited training text, the corresponding encoding features are determined, that is, the character is converted into an encoded vector representation that the language model can recognize.
[0050] Step S120: Process the encoding features of each character using a language model to obtain the global semantic features corresponding to the first placeholder and the local semantic features corresponding to the second placeholder.
[0051] The language model can include multiple hidden layers, each extracting deep-level features from the encoded features of the input characters, until the last hidden layer extracts the deep-level encoded features of each character. In this step, the encoded features corresponding to the first placeholder are defined as global semantic features, and the encoded features corresponding to the second placeholder are defined as local semantic features.
[0052] The global semantic features are used to represent the overall semantics of the training text. That is, this application can set the hidden layer structure of the language model so that the first placeholder can pay attention to the information of the other characters, thereby encoding the complete global information of each character in the training text.
[0053] The local semantic features are used to represent the local semantics in the training text. In this embodiment, the target character is predicted by using the encoding features corresponding to the second placeholder at the end of the training text where the mask character is located, through the training method in step S130 below. This encodes the local information in the training text into the second placeholder, so that the encoding features corresponding to the second placeholder can represent the local semantics in the training text. In this application, the encoding features corresponding to the second placeholder are also defined as local semantic features.
[0054] Step S130: Based on the local semantic features corresponding to the second placeholder at the end of the training text where the mask character is located, predict the target character corresponding to the mask character until the set training end condition is met, and obtain the trained language model.
[0055] Understandably, the language model mask training process, i.e., the process of recovering the character at the position of [MASK], primarily examines the language model's local information processing capabilities. Existing technologies predict the corresponding original target character based on the encoded features of the [MASK] character. This application differs from existing technologies by using the encoded features of the second placeholder [SEP] at the end of the training text where the [MASK] character is located to predict the original target character corresponding to the [MASK] character. This allows the encoded features of the second placeholder [SEP] to represent the local semantics in the training text; that is, the encoded features of the second placeholder [SEP] can serve as local semantic features of the training text.
[0056] As can be seen from the above introduction, the training method of this application can enable the language model to extract both global and local semantic features of the input text simultaneously, that is, to achieve joint enhancement of global and local features, resulting in richer and more accurate features.
[0057] Furthermore, the downstream task processing module can process specified tasks based on the fused semantic features obtained by fusing global and local semantic features. Since the fused semantic features contain richer and more accurate information, the downstream task processing results are also more accurate, especially in the identification and judgment of similar semantic texts. Based on the fused semantic features, it is possible to better distinguish similar semantic texts and improve the model's ability to distinguish and judge similar semantic texts.
[0058] In some embodiments of this application, the process of adding placeholders and performing random character masking on the training text in step S100 above to obtain the edited training text is described.
[0059] This application provides two editing methods, as follows:
[0060] The first type
[0061] S1. Add a first placeholder before the first character of the first training text, and add a second placeholder at the end of each training text to obtain training text containing placeholders.
[0062] As explained earlier, depending on the downstream natural language processing task, the training text input to the language model can be one or more texts.
[0063] In this embodiment, the downstream natural language processing task is taken as similar text judgment. Each time the training text input to the language model contains two texts, that is, two texts that need to be judged as similar texts. For example, the two texts are "high temperature and low pressure" and "high pressure and low temperature".
[0064] The training text after adding the first placeholder [CLS] and the second placeholder [SEP] is as follows:
[0065] [CLS] High Temperature Low Pressure [SEP] High Pressure Low Temperature [SEP]
[0066] S2. The training text containing placeholders is split into characters, and target characters are randomly selected and replaced with set mask characters to obtain the edited training text.
[0067] In this embodiment, the language model is trained by predicting the original characters corresponding to the mask. Therefore, this step further splits the training text containing placeholders into characters and randomly selects target characters to replace them with the set mask characters, resulting in the edited training text.
[0068] In the language model training process, this application uses the encoding feature corresponding to the second placeholder at the end of the training text where the mask character is located to predict the original target character corresponding to the mask character. Therefore, when performing random target character masking on the training text, at most one target character can be randomly selected for masking for each training text. Defining the mask character as [MASK], the edited training text obtained after masking the training text obtained in the previous step is as follows:
[0069] [CLS] High Temperature Low Pressure [SEP] High [MASK] Low Temperature [SEP]
[0070] This example shows that the first training text "high temperature and low pressure" was not masked, while the second training text "high pressure and low temperature" randomly selected the character "pressure" for masking.
[0071] The second type
[0072] S1. Split each training text according to its characters, and randomly select target characters to replace them with the set mask characters to obtain the masked training text.
[0073] Taking the training texts "high temperature and low pressure" and "high pressure and low temperature" as examples again, the masked training text obtained after this step is as follows:
[0074] Training text 1: High temperature and low pressure
[0075] Training text 2: High [MASK] Low temperature
[0076] S2. Add a first placeholder before the starting character of the training text after the first mask, and add a second placeholder at the end of the training text after each mask to obtain the edited training text.
[0077] After adding placeholders to the masked training text from the previous step, the edited training text is as follows:
[0078] [CLS] High Temperature Low Pressure [SEP] High [MASK] Low Temperature [SEP]
[0079] In some embodiments of this application, the process of determining the encoding features of each character in the edited training text, including the first and second placeholders and the mask character, in step S110 is described.
[0080] This embodiment introduces the encoding features of three types of characters, as follows:
[0081] Character encoding features: Character encoding features represent the meaning of the character itself.
[0082] Fragment coding features: Fragment coding features represent the sorting order of characters in the training text. For example, the fragment coding features of each character in the first training text can be represented as 1111, and the fragment coding features of each character in the second training text can be represented as 2222.
[0083] Positional encoding features: Positional encoding features represent the position of a character in the edited training text.
[0084] This embodiment provides two representation methods for positional coding features, as follows:
[0085] The first type
[0086] Static encoding, which is related to the length of the training text, is used. The positional encoding features of each character in the training text are related to the order of their positions within the text.
[0087] Example of edited training text as follows:
[0088] [CLS] High Temperature Low Pressure [SEP] High [MASK] Low Temperature [SEP]
[0089] The 11 characters from [CLS] to the last [SEP] can be represented by their positional encoding features from left to right as 0, 1, 2, 3...10.
[0090] Analysis shows that, according to the positional encoding feature representation method in this embodiment, for two edited training texts with different content but the same length, the encoding features of different characters at the same position are the same. This will cause the language model to be unable to learn more valuable information from the positional encoding features.
[0091] Therefore, this embodiment provides another representation of the positional encoding feature, as follows:
[0092] The second type
[0093] Determine the word segments and their order in the edited training text; for each character in the edited training text, determine the character's positional encoding features according to the order of the word segments in which the character is located.
[0094] Let's take the edited training text from the example above as an example for illustration:
[0095] [CLS] High Temperature Low Pressure [SEP] High [MASK] Low Temperature [SEP]
[0096] The segmented words in the edited training text include “[CLS]”, “high temperature”, “low pressure”, “[SEP]”, “high[MASK]”, “low temperature”, and “[SEP]”. It is worth noting that for “high[MASK]”, since the target character corresponding to the mask [MASK] is “temperature”, and “high temperature” is a segmented word, “high[MASK]” is also treated as a segmented word.
[0097] Then combine Figure 2 As shown in the example language model training process diagram, the positional encoding features of the 11 characters in the edited training text from left to right can be represented as 0, 1, 1, 2, 2, 3, 4, 4, 5, 5, 6 respectively.
[0098] Based on the three types of encoding features described in the above embodiments, after obtaining the sum of the character encoding features, fragment encoding features, and positional encoding features for each character, the character encoding features, fragment encoding features, and positional encoding features for each character can be further fused to obtain the fused encoding features for each character. (Refer to...) Figure 2 It can be seen that after fusing the character encoding features, fragment encoding features and position encoding features of each character, the resulting fused encoding features are input into the language model.
[0099] The language model extracts deep-level encoding features for each character and uses the second placeholder [SEP] at the end of the second training text containing [MASK] to predict the original character corresponding to [MASK], resulting in the predicted value "low".
[0100] In some embodiments of this application, the process of using a language model to process the encoding features of each character in step S120 above to obtain the global semantic features corresponding to the first placeholder and the local semantic features corresponding to the second placeholder is described.
[0101] To enable the first placeholder to be encoded into global semantic features, this embodiment can configure the language model's network structure so that during the processing of the edited training text, each character in the edited training text undergoes attention processing with the remaining characters, resulting in a feature representation of each character after fusing the meanings of the other characters. Based on this, contextual semantic information can be modeled.
[0102] Among them, the feature representation of the character corresponding to the first placeholder after integrating the meaning of the other characters is used as the global semantic feature, and the feature representation of the character corresponding to the second placeholder after integrating the meaning of the other characters is used as the local semantic feature.
[0103] In an optional example, the language model can employ a network structure similar to the Transformer model to achieve the goal of encoding global semantic features using the first placeholder. Unlike traditional Transformer models, this application does not predict the original target character corresponding to [MASK] based on the encoded feature representation of the [MASK] character, but rather based on the encoded feature representation of the second placeholder. This allows the second placeholder to encode local semantic features, and the encoded feature representation of the second placeholder is thus defined as a local semantic feature.
[0104] In some embodiments of this application, the language model training method of this case is further described.
[0105] The foregoing embodiments described the language model pre-training process, highlighting the ability of the pre-trained language model to extract both global and local semantic features from the input text. Building upon this foundation, this embodiment can further fine-tune the downstream task processing module based on the language model. During the fine-tuning stage, the global and local semantic features output by the trained language model are used to enhance the downstream task processing module's ability to distinguish between similar texts.
[0106] Specifically, after obtaining the trained language model, the following steps can be further included:
[0107] S1. Process the training text with the first and second placeholders added using the trained language model to obtain the global semantic features corresponding to the first placeholder and the local semantic features corresponding to the second placeholder.
[0108] Specifically, according to the language model designed in this application, the input text needs to have first and second placeholders added. The method of adding the first and second placeholders can be referred to the previous description, and will not be repeated here.
[0109] The training texts used in this step can be the same as those used in the language model training process, or other training texts can be used; there are no strict limitations on this.
[0110] After adding first and second placeholders to the training text, and inputting it into the trained language model, we can obtain the global semantic features corresponding to the first placeholder and the local semantic features corresponding to the second placeholder.
[0111] Understandably, if the input training text is a single text, there will be only one second placeholder, and thus only one corresponding local semantic feature. If the input training text is multiple texts, the number of corresponding second placeholders will be the same as the number of input training texts, and the number of obtained local semantic features will also be the same as the number of second placeholders.
[0112] S2. The global semantic features and the local semantic features are fused to obtain the fused semantic features.
[0113] S3. Input the fused semantic features into the downstream task processing module to obtain the task processing result predicted by the task processing module. With the goal of the predicted task processing result being close to the sample label corresponding to the training text, the network parameters of the task processing module are fine-tuned.
[0114] Combination Figure 3 The fine-tuning stage is explained as shown below:
[0115] Taking a training text containing the two sentences "high temperature, low pressure" and "high pressure, low temperature" as an example, after adding a first placeholder [CLS] and a second placeholder [SEP] to the training text and segmenting it according to characters, the resulting input is as follows: Figure 3 As shown.
[0116] After processing by the language model, the encoded features corresponding to the first placeholder and the two second placeholders can be obtained. The global semantic feature corresponding to the first placeholder is defined as T. [CLS] To distinguish the encoded features corresponding to two different second placeholders, the local semantic feature corresponding to the first [SEP] is defined as T. [SEP1] The local semantic feature corresponding to the second [SEP] is represented as T. [SEP2] .
[0117] T [CLS] T [SEP1] and T [SEP2] The fusion yields the fused semantic features as T. 融合 .
[0118] Downstream task modules can determine whether the semantics of two input texts are the same, and then use the fused semantic features as T. 融合Input the task module, and the task module will give the prediction result. For example, 0 indicates different semantics and 1 indicates the same semantics.
[0119] The input training text carries labels indicating whether the semantics are the same, allowing the task module to fine-tune the network parameters based on the prediction results and labels.
[0120] In some embodiments of this application, a natural language task processing method is further provided, which may include the following steps:
[0121] S1. Obtain the text of the task to be processed.
[0122] Specifically, the number of task texts to be processed in this step may vary depending on the natural language task to be handled by the task processing module. For example, if the natural language task is sentence classification, then the task text to be processed can be one sentence, that is, the sentence that needs to be classified. As another example, if the natural language task is text semantic similarity judgment, then the task text to be processed can be two texts, that is, two input texts that need to be semantically similar.
[0123] S2. Process the task text using the trained language model to obtain the global semantic features corresponding to the first placeholder and the local semantic features corresponding to the second placeholder output by the language model.
[0124] In this step, the language model can be trained using the training methods described in the preceding embodiments. When processing task text using the language model, the task text can be first supplemented with first and second placeholders as described above, and then processed by the language model. The language model can output the global semantic features corresponding to the first placeholder and the local semantic features corresponding to the second placeholder.
[0125] It is understandable that a second placeholder is added to the end of each task text, so the number of local semantic features corresponding to the second placeholder in the model output is the same as the number of task texts.
[0126] S3. The global semantic features and local semantic features are fused to obtain fused semantic features.
[0127] Specifically, feature fusion can be achieved by concatenating global and local semantic features. Through feature fusion, the fused semantic features contain richer information, resulting in a stronger ability to distinguish between texts with similar semantics.
[0128] If the task text contains one text, the language model outputs one global semantic feature and one local semantic feature.
[0129] If the task text contains N texts, where N≥2, then the language model outputs one global semantic feature and N local semantic features corresponding to the second placeholder at the end of each text. Based on this, this step fuses the global and local semantic features to obtain the fused semantic features, specifically including:
[0130] The global semantic features and N local semantic features are fused together to obtain the fused semantic features.
[0131] S4. Using the pre-configured task processing module, perform specified task processing based on the fused semantic features to obtain the task processing result.
[0132] Specifically, the pre-configured task processing module can be a network module that performs a specified natural language task, such as a classification model, and the result of the classification task is based on the processing results of the classification model. This task processing module can be a task processing module that has been fine-tuned using labeled data after the language model has been trained.
[0133] In this step, the fused semantic features can be input into the task processing module to obtain the specific task processing results output by the module.
[0134] Because the task processing module uses a fused semantic feature that combines global and local semantic features of the task text for processing, this fused semantic feature has stronger expressive power and a stronger ability to distinguish semantically similar texts, resulting in more accurate task processing results.
[0135] The language model training apparatus provided in the embodiments of this application is described below. The language model training apparatus described below can be referred to in correspondence with the language model training method described above.
[0136] See Figure 4 , Figure 4 This is a schematic diagram of the structure of a language model training device disclosed in an embodiment of this application.
[0137] like Figure 4 As shown, the device may include:
[0138] The training text editing unit 11 is used to add placeholders and perform random character masking on the training text to obtain the edited training text. The target characters randomly selected in the training text are replaced with set mask characters in the edited training text. A first placeholder is added before the starting character of the first training text and a second placeholder is added at the end of each training text.
[0139] The encoding feature determination unit 12 is used to determine the encoding features of each character in the edited training text, including the first and second placeholders and the mask character.
[0140] The encoding feature processing unit 13 is used to process the encoding features of each character using a language model to obtain global semantic features corresponding to the first placeholder and local semantic features corresponding to the second placeholder; the global semantic features are used to represent the overall semantics of the training text, and the local semantic features are used to represent the local semantics of the training text.
[0141] The prediction unit 14 is used to predict the target character corresponding to the mask character based on the local semantic features corresponding to the second placeholder at the end of the training text where the mask character is located, until the set training end condition is met, and the trained language model is obtained.
[0142] Optionally, the process by which the above-mentioned training text editing unit adds placeholders and performs random character masking on the training text to obtain the edited training text may include:
[0143] Add a first placeholder before the first character of the first training text, and add a second placeholder at the end of each training text to obtain training text containing placeholders.
[0144] The training text containing placeholders is split into characters, and target characters are randomly selected and replaced with a set mask character to obtain the edited training text;
[0145] or,
[0146] Each training text is split into characters, and target characters are randomly selected and replaced with a set mask character to obtain the masked training text;
[0147] Add a first placeholder before the first character of the training text after the first mask, and add a second placeholder at the end of the training text after each mask to obtain the edited training text.
[0148] Optionally, the process by which the encoding feature determination unit determines the encoding features of each character in the edited training text, including the first and second placeholders and the mask character, may include:
[0149] For each character in the edited training text, including the first and second placeholders and mask characters: determine the character encoding feature, fragment encoding feature and position encoding feature of the character respectively, wherein the character encoding feature represents the meaning of the character itself, the fragment encoding feature represents the sorting order of the character in the training text, and the position encoding feature represents the position of the character in the edited training text;
[0150] The character encoding features, fragment encoding features, and positional encoding features of each character are fused to obtain the fused encoding features of each character.
[0151] Optionally, the process by which the encoding feature determination unit determines the positional encoding features of a character may include:
[0152] Determine the word segmentation and the order of each word segmentation in the edited training text;
[0153] For each character in the edited training text, the positional encoding features of the character are determined according to the order of the word segments in which the character is located.
[0154] Optionally, the process by which the above-mentioned encoding feature processing unit processes the encoding features of each character using a language model to obtain the global semantic features corresponding to the first placeholder and the local semantic features corresponding to the second placeholder may include:
[0155] Each character in the edited training text is processed with attention by a language model, and the feature representation of each character after fusing the meaning of the other characters is obtained. The feature representation of the character corresponding to the first placeholder after fusing the meaning of the other characters is used as the global semantic feature, and the feature representation of the character corresponding to the second placeholder after fusing the meaning of the other characters is used as the local semantic feature.
[0156] Optionally, the apparatus of this application may further include a downstream task fine-tuning unit for:
[0157] The trained language model is used to process the training text after adding the first and second placeholders to obtain the global semantic features corresponding to the first placeholder and the local semantic features corresponding to the second placeholder.
[0158] The global semantic features and the local semantic features are fused to obtain the fused semantic features;
[0159] The fused semantic features are input into the downstream task processing module to obtain the task processing result predicted by the task processing module. The network parameters of the task processing module are fine-tuned with the goal of making the predicted task processing result close to the sample label corresponding to the training text.
[0160] The natural language task processing apparatus provided in the embodiments of this application is described below. The natural language task processing apparatus described below can be referred to in correspondence with the natural language task processing method described above.
[0161] See Figure 5 , Figure 5 This is a schematic diagram of the structure of a natural language task processing device disclosed in an embodiment of this application.
[0162] like Figure 5 As shown, the device may include:
[0163] Task text acquisition unit 21 is used to acquire the task text to be processed;
[0164] The task text processing unit 22 is used to process the task text using the language model trained in the aforementioned embodiments, and to obtain the global semantic features corresponding to the first placeholder and the local semantic features corresponding to the second placeholder output by the language model.
[0165] The feature fusion unit 23 is used to fuse the global semantic features and local semantic features to obtain fused semantic features;
[0166] The task processing unit 24 is used to perform specified task processing based on the fused semantic features using a pre-configured task processing module to obtain task processing results.
[0167] The language model training device or natural language task processing device provided in this application embodiment can be applied to electronic devices, such as terminals: mobile phones, computers, etc. Optionally, Figure 6 A hardware block diagram of the electronic device is shown, with reference to... Figure 6 The hardware structure of an electronic device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;
[0168] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4;
[0169] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.
[0170] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;
[0171] The memory stores a program, which the processor can call to execute. The program is used to execute the various steps of the aforementioned language model training method or the various steps of the aforementioned natural language task processing method.
[0172] This application embodiment also provides a storage medium that can store a program suitable for execution by a processor, the program being used to execute various steps of the aforementioned language model training method, or to execute various steps of the aforementioned natural language task processing method.
[0173] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0174] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.
[0175] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A language model training method, characterized in that, include: The training text is processed by adding placeholders and random character masking to obtain the edited training text. The target characters randomly selected in the training text are replaced with set mask characters in the edited training text. A first placeholder is added before the starting character of the first training text and a second placeholder is added at the end of each training text. Determine the encoding features of each character in the edited training text, including the first and second placeholders and the mask character; The encoding features of each character are processed using a language model to obtain global semantic features corresponding to the first placeholder and local semantic features corresponding to the second placeholder; the global semantic features are used to represent the overall semantics of the training text, and the local semantic features are used to represent the local semantics of the training text. Based on the encoding features corresponding to the second placeholder at the end of the training text where the mask character is located, the target character corresponding to the mask character is predicted. Local information in the training text is encoded into the second placeholder, so that the encoding features corresponding to the second placeholder can represent the local semantics in the training text, until the set training termination condition is reached, and the trained language model is obtained.
2. The method according to claim 1, characterized in that, The process of adding placeholders and performing random character masking on the training text to obtain the edited training text includes: Add a first placeholder before the first character of the first training text, and add a second placeholder at the end of each training text to obtain training text containing placeholders. The training text containing placeholders is split into characters, and target characters are randomly selected and replaced with a set mask character to obtain the edited training text; or, Each training text is split into characters, and target characters are randomly selected and replaced with a set mask character to obtain the masked training text; Add a first placeholder before the first character of the training text after the first mask, and add a second placeholder at the end of the training text after each mask to obtain the edited training text.
3. The method according to claim 1, characterized in that, Determine the encoding features of each character in the edited training text, including the first and second placeholders and the mask character, including: For each character in the edited training text, including the first and second placeholders and mask characters: determine the character encoding feature, fragment encoding feature and position encoding feature of the character respectively, wherein the character encoding feature represents the meaning of the character itself, the fragment encoding feature represents the sorting order of the character in the training text, and the position encoding feature represents the position of the character in the edited training text; The character encoding features, fragment encoding features, and positional encoding features of each character are fused to obtain the fused encoding features of each character.
4. The method according to claim 3, characterized in that, The process of determining the positional encoding features of a character includes: Determine the word segmentation and the order of each word segmentation in the edited training text; For each character in the edited training text, the positional encoding features of the character are determined according to the order of the word segments in which the character is located.
5. The method according to claim 1, characterized in that, The process of using a language model to process the encoding features of each character to obtain global semantic features corresponding to the first placeholder and local semantic features corresponding to the second placeholder includes: Each character in the edited training text is processed with attention by a language model, and the feature representation of each character after fusing the meaning of the other characters is obtained. The feature representation of the character corresponding to the first placeholder after fusing the meaning of the other characters is used as the global semantic feature, and the feature representation of the character corresponding to the second placeholder after fusing the meaning of the other characters is used as the local semantic feature.
6. The method according to any one of claims 1-5, characterized in that, Also includes: The trained language model is used to process the training text after adding the first and second placeholders to obtain the global semantic features corresponding to the first placeholder and the local semantic features corresponding to the second placeholder. The global semantic features and the local semantic features are fused to obtain the fused semantic features; The fused semantic features are input into the downstream task processing module to obtain the task processing result predicted by the task processing module. The network parameters of the task processing module are fine-tuned with the goal of making the predicted task processing result close to the sample label corresponding to the training text.
7. A natural language processing method, characterized in that, include: Get the text of the task to be processed; The task text is processed using the language model trained according to any one of claims 1-6, and the global semantic features corresponding to the first placeholder and the local semantic features corresponding to the second placeholder output by the language model are obtained. The global semantic features and local semantic features are fused to obtain fused semantic features; Using a pre-configured task processing module, a specified task is processed based on the fused semantic features to obtain the task processing result.
8. The method according to claim 7, characterized in that, The task processing module is a classification model; The process of using a pre-configured task processing module to perform specified task processing based on the fused semantic features, and obtaining task processing results, includes: Using the classification model, a classification task is performed based on the fused semantic features to obtain the classification task processing result.
9. The method according to claim 7, characterized in that, If the task text contains one text, then the language model outputs one global semantic feature and one local semantic feature. or, The task text contains N texts, where N≥2. The language model outputs one global semantic feature and N local semantic features, which correspond to the second placeholder at the end of each text. The process of fusing the global semantic features and local semantic features to obtain fused semantic features includes: The global semantic features and N local semantic features are fused together to obtain the fused semantic features.
10. A language model training device, characterized in that, include: The training text editing unit is used to add placeholders and perform random character masking on the training text to obtain the edited training text. The target characters randomly selected in the training text are replaced with set mask characters in the edited training text. A first placeholder is added before the starting character of the first training text and a second placeholder is added at the end of each training text. The encoding feature determination unit is used to determine the encoding features of each character in the edited training text, including the first and second placeholders and the mask character. The encoding feature processing unit is used to process the encoding features of each character using a language model to obtain global semantic features corresponding to the first placeholder and local semantic features corresponding to the second placeholder; the global semantic features are used to represent the semantics of the training text as a whole, and the local semantic features are used to represent the semantics of a part of the training text. The prediction unit is used to predict the target character corresponding to the mask character based on the encoding features corresponding to the second placeholder at the end of the training text where the mask character is located, and to encode local information in the training text into the second placeholder so that the encoding features corresponding to the second placeholder can represent the local semantics in the training text, until the set training termination condition is met, and the trained language model is obtained.
11. A natural language task processing device, characterized in that, include: The task text acquisition unit is used to acquire the task text to be processed. The task text processing unit is used to process the task text using a language model trained according to any one of claims 1-6, and obtain the global semantic features corresponding to the first placeholder and the local semantic features corresponding to the second placeholder output by the language model. The feature fusion unit is used to fuse the global semantic features and local semantic features to obtain fused semantic features; The task processing unit is used to perform specified task processing based on the fused semantic features using a pre-configured task processing module to obtain task processing results.
12. An electronic device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement the language model training method as described in any one of claims 1 to 6, or to implement the various steps of the natural language task processing method as described in any one of claims 7 to 9.
13. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the language model training method as described in any one of claims 1 to 6, or implements the various steps of the natural language task processing method as described in any one of claims 7 to 9.