Training corpus construction method, text processing method and device

Abstract

The application provides a training corpus construction method, a text processing method and a device. The training corpus construction method comprises the following steps: constructing a phrase dictionary according to at least one original sample corpus, wherein the phrase dictionary comprises phrases and corresponding phrase feature information of the phrases; replacing a target phrase in a first original sample corpus with a similar phrase to obtain an expanded sample corpus, wherein the first original sample corpus is any original sample corpus including the target phrase in the at least one original sample corpus; and constructing a training corpus according to the original sample corpus and the expanded sample corpus. The training corpus is enriched, the semantic information included in the training corpus is increased, and the quality of the training corpus is high.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method for constructing a training corpus. This application also relates to a text processing method, a training corpus construction apparatus, a text processing apparatus, a computing device, and a computer-readable storage medium. Background Technology

[0002] With the development of computer technology, artificial intelligence (AI) technology has also flourished. AI is a comprehensive technology in computer science that studies the design principles and implementation methods of various intelligent machines to enable them to have the functions of perception, reasoning, and decision-making. AI technology is a comprehensive discipline that involves a wide range of fields, such as natural language processing technology and machine learning / deep learning.

[0003] In existing technologies, training corpora can be pre-acquired, and artificial intelligence technology can be used to train text processing models. The trained text processing models can then automatically process text. However, the training of text processing models requires a large amount of training corpora to achieve good text processing results. Training corpora obtained directly through the internet and other means are relatively scarce and limited, resulting in less than ideal training results for text processing models. Summary of the Invention

[0004] In view of this, embodiments of this application provide a method for constructing a training corpus to address the technical deficiencies in the prior art. Embodiments of this application also provide a text processing method, a training corpus construction apparatus, a text processing apparatus, a computing device, and a computer-readable storage medium.

[0005] According to a first aspect of the embodiments of this application, a method for constructing a training corpus is provided, comprising:

[0006] Based on at least one original sample corpus obtained, a phrase dictionary is constructed, wherein the phrase dictionary includes phrases and phrase feature information corresponding to the phrases;

[0007] The target phrase in the first original sample corpus is replaced with a similar phrase to obtain an extended sample corpus, wherein the first original sample corpus is any original sample corpus that includes the target phrase in at least one original sample corpus;

[0008] A training corpus is constructed based on the original sample corpus and the expanded sample corpus.

[0009] Optionally, the feature is that, based on at least one obtained original sample corpus, a phrase dictionary is constructed, including:

[0010] Perform word segmentation on at least one original sample corpus to obtain at least two segmented words;

[0011] According to the preset grammar rules, extract phrases from at least two word segments and determine the phrase feature information of the phrases;

[0012] A phrase dictionary is constructed based on phrases and their corresponding phrase feature information.

[0013] Optionally, before replacing the target phrase in the first original sample corpus with a similar phrase to obtain the expanded sample corpus, the method further includes:

[0014] Identify target phrases in the phrase dictionary whose phrase feature information satisfies preset replacement conditions;

[0015] Based on the first word vector of the target phrase and the second word vector of the reference phrase in the preset phrase library, the similar phrases corresponding to the target phrase are determined.

[0016] Optionally, based on the first word vector of the target phrase and the second word vector of reference phrases in a preset phrase library, similar phrases corresponding to the target phrase are determined, including:

[0017] Determine the similarity between the first word vector and the second word vector of the reference phrase;

[0018] The obtained similarity scores are sorted, and a target number of similar phrases are selected based on the sorting results.

[0019] Optionally, the phrase feature information is word frequency, and the preset replacement condition is that the word frequency is less than the word frequency threshold;

[0020] Before sorting the obtained similarities and selecting the target number of similar phrases based on the sorting results, the following steps are also included:

[0021] The target value is determined based on the difference between the word frequency corresponding to the target phrase and the word frequency threshold.

[0022] Optionally, a training corpus is constructed based on the original sample corpus and the expanded sample corpus, including:

[0023] The expanded sample corpus was scored using an evaluation model;

[0024] Based on the scores of the extended sample corpus, the fused sample corpus is selected;

[0025] The original sample corpus and the fused sample corpus are merged to obtain the training corpus set.

[0026] Optionally, the original sample corpus is a bilingual corpus, which includes samples in the first language and corresponding samples in the second language;

[0027] Based on at least one original sample corpus obtained, construct a phrase dictionary, including:

[0028] Perform word segmentation on the first language sample to obtain at least two first language segmented words, and perform word segmentation on the second language sample to obtain at least two second language segmented words;

[0029] Word alignment processing is performed on the words segmented from the first language and the words segmented from the second language to obtain word alignment information;

[0030] According to the preset grammatical rules, extract the first language phrases from at least two first language segmented words and determine the phrase feature information of the first language phrases;

[0031] Based on word alignment information, determine the second language phrase corresponding to the first language phrase from at least two second language segmented words;

[0032] A phrase dictionary is constructed based on the phrases in the first language, the corresponding phrases in the second language, and the corresponding phrase feature information.

[0033] Optionally, the original sample corpus is a bilingual corpus, which includes samples in a first language and corresponding samples in a second language. The phrase dictionary is constructed based on word alignment information of the first and second language samples. Similar phrases corresponding to the target phrase are determined according to the first word vector of the target phrase and the second word vector of reference phrases in a preset phrase library, including:

[0034] Based on the first word vector of the target first language phrase and the second word vector of the reference first language phrase in the preset first language phrase library, the similar first language phrases corresponding to the target first language phrase are determined. The target first language phrase is any first language phrase in the phrase dictionary whose phrase feature information satisfies the preset replacement conditions.

[0035] Based on word alignment information, determine the target second language phrase corresponding to the target first language phrase;

[0036] Based on the first word vector of the target second language phrase and the second word vector of the reference second language phrase in the preset second language phrase library, determine the similar second language phrases corresponding to the target second language phrase.

[0037] Optionally, the target phrase in the first original sample corpus is replaced with a similar phrase to obtain an expanded sample corpus, including:

[0038] The target first language phrase in the first language sample of the first original sample corpus is replaced with a similar first language phrase to obtain an extended first language sample;

[0039] Based on word alignment information, the target second language phrase in the second language sample of the first original sample corpus is replaced with a similar second language phrase to obtain an extended second language sample;

[0040] The extended first language sample and the corresponding extended second language sample are used as the extended sample corpus.

[0041] According to a second aspect of the embodiments of this application, a text processing method is provided, including:

[0042] Get the text to be processed;

[0043] The text to be processed is input into the text processing model to obtain the target processing result corresponding to the text to be processed. The text processing model is trained using the training corpus obtained by the training corpus construction method in the first aspect mentioned above.

[0044] According to a third aspect of the embodiments of this application, a training corpus construction apparatus is provided, comprising:

[0045] The first construction module is configured to construct a phrase dictionary based on at least one original sample corpus obtained, wherein the phrase dictionary includes phrases and phrase feature information corresponding to the phrases;

[0046] The replacement module is configured to replace the target phrase in the first original sample corpus with a similar phrase to obtain an extended sample corpus, wherein the first original sample corpus is any original sample corpus that includes the target phrase in at least one original sample corpus;

[0047] The second construction module is configured to construct a training corpus based on the original sample corpus and the extended sample corpus.

[0048] According to a fourth aspect of the embodiments of this application, a text processing apparatus is provided, comprising:

[0049] The first acquisition module is configured to acquire the text to be processed.

[0050] The first input module is configured to input the text to be processed into the text processing model to obtain the target processing result corresponding to the text to be processed. The text processing model is trained using the training corpus obtained by the training corpus construction method described in the first aspect above.

[0051] According to a fifth aspect of the embodiments of this application, a computing device is provided, comprising:

[0052] Memory and processor;

[0053] The memory is used to store computer-executable instructions, which are the steps of the training corpus construction method or text processing method when the processor executes the computer-executable instructions.

[0054] According to a sixth aspect of the embodiments of this application, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of a training corpus construction method or a text processing method.

[0055] The training corpus construction method provided in this application constructs a phrase dictionary based on at least one obtained original sample corpus, wherein the phrase dictionary includes phrases and phrase feature information corresponding to the phrases; replaces the target phrase in the first original sample corpus with similar phrases to obtain an extended sample corpus, wherein the first original sample corpus is any original sample corpus that includes the target phrase in at least one original sample corpus; and constructs a training corpus based on the original sample corpus and the extended sample corpus.

[0056] In this scenario, the target phrase in the original sample corpus can be replaced with similar phrases to obtain an expanded sample corpus. Since the meaning of a phrase is often fixed in different sentences, the problem of polysemy is avoided, thus preventing ambiguity and grammatical errors in the expanded sample corpus and improving its quality. Then, the original sample corpus and the expanded sample corpus can be merged to construct a training corpus, which greatly enriches the training corpus, increases the semantic information included in it, and the quality of the expanded training corpus is high. Subsequently, the text processing model can be trained based on this training corpus to improve the training effect of the text processing model, ensure the accuracy of the text processing model, and thus ensure the accuracy of subsequent text processing using the text processing model.

[0057] The text processing method provided in this application can obtain the text to be processed, input the text to be processed into the text processing model, and obtain the target processing result corresponding to the text to be processed. Since the text processing model is trained by the training corpus obtained by the above training corpus construction method, rich sample corpus is used in the training process, and the training accuracy is high. Therefore, the text processing model trained by the above method has high accuracy in processing text. Attached Figure Description

[0058] Figure 1 This is a flowchart of a training corpus construction method provided in an embodiment of this application;

[0059] Figure 2 This is a flowchart of a phrase dictionary construction method provided in an embodiment of this application;

[0060] Figure 3 This is a flowchart of a method for determining similar phrases provided in an embodiment of this application;

[0061] Figure 4This is a flowchart of another method for determining similar phrases provided in an embodiment of this application;

[0062] Figure 5 This is a flowchart of another method for constructing a training corpus provided in an embodiment of this application;

[0063] Figure 6 This is a flowchart of another phrase dictionary construction method provided in an embodiment of this application;

[0064] Figure 7 This is a flowchart of another method for determining similar phrases provided in an embodiment of this application;

[0065] Figure 8 This is a flowchart of an extended sample corpus acquisition method provided in an embodiment of this application;

[0066] Figure 9 This is a flowchart of a training method for a text processing model provided in an embodiment of this application;

[0067] Figure 10 This is a flowchart of another method for training a text processing model provided in an embodiment of this application;

[0068] Figure 11 This is a flowchart of a training method for a Chinese-English translation model provided in one embodiment of this application;

[0069] Figure 12 This is a flowchart of a text processing method provided in an embodiment of this application;

[0070] Figure 13 This is a schematic diagram of the structure of a training corpus construction device provided in one embodiment of this application;

[0071] Figure 14 This is a schematic diagram of the structure of a training device for a text processing model provided in an embodiment of this application;

[0072] Figure 15 This is a schematic diagram of the structure of a text processing device provided in an embodiment of this application;

[0073] Figure 16 This is a structural block diagram of a computing device provided in one embodiment of this application. Detailed Implementation

[0074] Many specific details are set forth in the following description to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.

[0075] The terminology used in one or more embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this application. The singular forms “a” and “the” as used in one or more embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this application refers to and includes any or all possible combinations of one or more associated listed items.

[0076] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this application, and similarly, second may also be referred to as first.

[0077] First, the terminology used in one or more embodiments of the present invention will be explained.

[0078] Neural machine translation (NMT): End-to-end neural machine translation based on encoder-decoder is currently the mainstream research direction in machine translation. It uses an encoder to encode source language text into a fixed-length semantic representation, and the decoder uses this representation to generate the corresponding target translation word by word.

[0079] Data augmentation: By using training samples to change the state of the data to generate variants of the data, more training data can be generated. When training data is scarce, data augmentation is the most direct and effective way to improve the performance of low-resource neural machine translation.

[0080] Back-translation: Using existing target language data, the source language is obtained through a target language-source language translation model to obtain pseudo-parallel data.

[0081] Synonym replacement: Randomly select some words and then replace them with their synonyms to generate new sentence pairs.

[0082] n-gram language model: also known as n-gram language model, n-gram refers to a set of N words in sequence. Each byte segment is called a gram. The frequency of all grams is counted and finally filtered according to a pre-set threshold.

[0083] Wordembedding: Vectorizes data and maps it to the same semantic space.

[0084] A phrase is a fixed sentence segment consisting of two or more words that form a certain combination relationship and are frequently used together in different sentences.

[0085] This application provides a method for constructing a training corpus. This application also relates to a method for training a text processing model, a text processing method, a device for constructing a training corpus, a device for training a text processing model, a text processing device, a computing device, and a computer-readable storage medium, which will be described in detail in the following embodiments.

[0086] Figure 1 The flowchart illustrates a method for constructing a training corpus according to an embodiment of this application, which specifically includes the following steps:

[0087] Step 102: Construct a phrase dictionary based on at least one original sample corpus obtained, wherein the phrase dictionary includes phrases and phrase feature information corresponding to the phrases.

[0088] Specifically, the original sample corpus can refer to the sample corpus obtained directly. The original sample corpus is the sample corpus required by the text processing model to be trained. That is, the content and form of the original sample corpus correspond to the text processing model to be trained. For example, when the text processing model to be trained is an information annotation model, the original sample corpus is a single sentence; when the text processing model to be trained is a translation model, the original sample corpus is a bilingual corpus, that is, the original sample corpus is a parallel corpus pair including samples of the first language and samples of the second language. The translation model can be a neural machine translation model.

[0089] Furthermore, a phrase refers to a fixed sentence segment in which two or more words form a certain combination relationship and are frequently used together in different sentences. A phrase dictionary is constructed based on at least one original sample corpus, which can be a phrase lexicon obtained by combining phrases appearing in at least one original sample corpus. Phrase feature information refers to parameter information describing the relevant characteristics of a phrase in the original sample corpus. This could include the number of times a phrase appears in at least one original sample corpus, the proportion of times a phrase appears in at least one original sample corpus to the total number of phrases, or information such as the phrase's position and part of speech in the original sample corpus.

[0090] It should be noted that at least one original sample corpus required for training the text processing model can be obtained in advance through the network. The more original sample corpus obtained, the richer the content of the training corpus constructed later, and the better the training effect of the text processing model.

[0091] In addition, after obtaining the original sample corpus, data cleaning can be performed on the original sample corpus, such as deduplication, removal of blank lines, conversion of traditional Chinese to simplified Chinese, filtering of sentences with abnormal length and excessive garbled characters.

[0092] In practical applications, when constructing a phrase dictionary based on at least one original sample corpus, phrases can be extracted directly from the original sample corpus, or phrases can be constructed based on the various word segments included in the original sample corpus.

[0093] In this embodiment, at least one original sample corpus can be obtained in advance. Based on the phrases appearing in the original sample corpus and the corresponding feature information, a phrase dictionary is constructed. This allows for the determination of which phrases in the original sample corpus to be replaced based on the constructed phrase dictionary. Since the meaning of a phrase is often fixed in different sentences, the problem of polysemy is avoided, thereby preventing ambiguity and grammatical errors in the subsequently constructed extended sample corpus. This improves the quality of the extended sample corpus, thus ensuring the quality of the extended training corpus while enriching the training corpus.

[0094] Step 104: Determine the target phrase in the phrase dictionary whose phrase feature information meets the preset replacement conditions. Based on the first word vector of the target phrase and the second word vector of the reference phrase in the preset phrase library, determine the similar phrases corresponding to the target phrase.

[0095] Specifically, the preset replacement conditions are the conditions that the phrase feature information of the phrases in the phrase dictionary that need to be semantically expanded must meet. These preset replacement conditions can be customized according to the requirements of the text processing model for the training corpus.

[0096] In one possible implementation, since the fewer times a phrase appears in at least one original sample corpus, the less semantic information related to the phrase is in at least one original sample corpus, which may affect the text processing model's ability to learn the semantic information of the phrase during training, the phrase feature information can be the number of times the phrase appears in at least one original sample corpus. In this case, the preset replacement condition can be that the number of times the phrase appears is less than a threshold. Alternatively, the phrase feature information can be the phrase occurrence ratio, that is, the proportion of the number of times the phrase appears in at least one original sample corpus to the total number of phrases. In this case, the preset replacement condition can be that the phrase occurrence ratio is less than a proportion threshold.

[0097] In another possible implementation, certain phrases with defined features have a relatively small impact on semantics. Therefore, the phrase feature information can be these defined features with a relatively small impact on semantics. In this case, the preset replacement condition can be that the phrase satisfies the defined feature. That is, the target phrases in the first original sample corpus that meet the preset replacement condition can also be determined based on the defined features. Subsequently, the target phrases can be replaced by similar phrases corresponding to the target phrases. For example, the defined features can be "phrase type & sentence ending & length limit". Target phrases that meet the defined features can be selected, and subsequently, the target phrases can be replaced with corresponding similar phrases to minimize the impact on the original semantics and structure of the sentence.

[0098] In practical applications, the preset phrase library can be a collection of a large number of phrases. The preset phrase library can include phrases from the phrase dictionary constructed based on at least one original sample corpus, or it can first obtain at least one common phrase through the network, and then merge the obtained common phrase with the constructed phrase dictionary to obtain the preset phrase library. Each phrase included in the preset phrase library is a reference phrase. Subsequently, the target phrase can be compared with each phrase included in the preset phrase library, and similar phrases similar to the target phrase can be determined from each phrase included in the preset phrase library.

[0099] In this embodiment, phrases that meet preset replacement conditions can be selected from each phrase based on the phrase feature information of each phrase in the phrase dictionary. Any one of the selected phrases can be used as the target phrase. Then, based on the first word vector of the target phrase and the second word vector of each reference phrase in the preset phrase library, similar phrases corresponding to the target phrase can be determined from each reference phrase. That is, each selected phrase can be used as the target phrase and the corresponding similar phrase can be determined. The similar phrase can then be used to replace the target phrase in the original sample corpus, thereby obtaining an expanded sample corpus, expanding the training corpus, and improving the training effect of the text processing model.

[0100] Step 106: Replace the target phrase in the first original sample corpus with a similar phrase to obtain an extended sample corpus, wherein the first original sample corpus is any original sample corpus that includes the target phrase in at least one original sample corpus.

[0101] It should be noted that after identifying the target phrases in the phrase dictionary that are waiting to be replaced, at least one original sample corpus containing the target phrase can be identified. This original sample corpus is used as the first original sample corpus. Then, the target phrase in the first original sample corpus can be replaced with the corresponding similar phrase to obtain the expanded sample corpus, thereby performing data augmentation.

[0102] In practical applications, when determining the similar phrases corresponding to the target phrase, the number of similar phrases obtained can be set. That is, the number of similar phrases corresponding to the target phrase may be one or more. If there is only one similar phrase, the target phrase in the first original sample corpus can be replaced with that corresponding similar phrase to obtain an expanded sample corpus. If there are multiple similar phrases, the target phrase in the first original sample corpus can be replaced sequentially with each of the corresponding similar phrases to obtain multiple corresponding expanded sample corpora. In other words, replacing the target phrase in the first original sample corpus with a single similar phrase yields one expanded sample corpus; the number of expanded sample corpora obtained depends on the number of similar phrases selected for replacement.

[0103] In this embodiment of the application, each original sample corpus containing the target phrase can be used as the first original sample corpus. The target phrase in the first original sample corpus is replaced with a similar phrase to obtain the corresponding extended sample corpus, thereby expanding the training corpus and improving the training effect of the text processing model.

[0104] Step 108: Construct a training corpus based on the original sample corpus and the extended sample corpus.

[0105] It should be noted that after obtaining the extended sample corpus, it can be directly fused with the original sample corpus to construct a training corpus. Alternatively, since the extended sample corpus is constructed by replacing the target phrases in the original sample corpus with similar phrases, it may not conform to semantic norms. Therefore, the constructed extended sample corpus can be further filtered to remove those that clearly do not conform to semantic norms, selecting the semantically compliant extended sample corpus. Then, the selected extended sample corpus can be fused with the original sample corpus to form the training corpus.

[0106] The training corpus construction method provided in this application can identify target phrases in the original sample corpus that meet preset replacement conditions, replace the target phrases with similar phrases to obtain extended sample corpus. Since the meaning of a phrase is often fixed in different sentences, the problem of polysemy is avoided, thus avoiding ambiguity and grammatical errors in the extended sample corpus and improving its quality. Then, the original sample corpus and the obtained extended sample corpus can be merged to construct a training corpus, which greatly enriches the training corpus, increases the semantic information included in the training corpus, and the quality of the extended training corpus is high. Training a text processing model based on this training corpus can improve the training effect of the text processing model and ensure its accuracy.

[0107] It should be noted that step 102 above can be achieved through methods such as... Figure 2 The phrase dictionary construction method shown in Figure 2 is a flowchart illustrating a phrase dictionary construction method according to an embodiment of this application, which specifically includes the following steps:

[0108] Step 202: Segment at least one original sample corpus to obtain at least two segmented words.

[0109] It should be noted that at least one original sample corpus can be segmented using a preset word segmentation tool to obtain at least two segmented words. This facilitates the subsequent construction of corresponding phrases based on the segmented words, thus building a phrase dictionary. The preset word segmentation tool is any tool capable of segmenting sentences to obtain segmented words, such as Jieba, pkuseg, or NLTK (Natural Language Processing Toolkit).

[0110] In addition, since different languages ​​have different linguistic structures, when segmenting at least one original sample corpus to obtain at least two segmented words, the corresponding segmentation tool can be selected according to the language of the original sample corpus.

[0111] Step 204: Extract phrases from at least two word segments according to preset grammar rules, and determine the phrase feature information of the phrases.

[0112] It should be noted that the presupposed grammatical rules can be rules for combining words into phrases, such as nouns and verbs forming subject-predicate phrases, verbs and nouns forming verb-object phrases, and two or more content words of the same part of speech forming coordinate phrases. Alternatively, certain words can form specific phrase structures, such as idioms and proverbs, such as "meticulous" or "haphazard."

[0113] In practical applications, after segmenting the original sample corpus, words can be selected from at least two segmented words according to preset grammatical rules to form phrases, and the phrase feature information of the phrases can be determined, thereby constructing a phrase dictionary.

[0114] Step 206: Construct a phrase dictionary based on the phrases and their corresponding phrase feature information.

[0115] It should be noted that when constructing a phrase dictionary based on phrases and their corresponding phrase feature information, the phrases and their corresponding phrase feature information can be stored accordingly to obtain the phrase dictionary.

[0116] In this embodiment, at least one original sample corpus can be segmented to obtain at least two segmented words. Then, according to preset grammatical rules, words are selected from the at least two segmented words to form phrases, thereby constructing a phrase dictionary. Since phrases are composed of words, the meaning of a phrase is often fixed in different sentences. There are no multiple meanings similar to words, which avoids the problem of polysemy. Therefore, ambiguity and grammatical errors in the subsequent extended sample corpus are avoided, and the quality of the extended sample corpus is improved.

[0117] It should be noted that step 104 above can be achieved through methods such as... Figure 3 The method for determining similar phrases is shown in Figure 3, which is a flowchart of a method for determining similar phrases according to an embodiment of this application, specifically including the following steps:

[0118] Step 302: Input the target phrase into the feature extraction model to obtain the first word vector corresponding to the target phrase.

[0119] It should be noted that the feature extraction model is a pre-trained model that can perform feature recognition on the input phrase and output a feature vector. Therefore, the target phrase can be input into the feature extraction model, which can output the feature vector of the target phrase. This feature vector is used as the first word vector of the target phrase, which facilitates the subsequent determination of the similarity between each reference phrase and the target phrase based on the first word vector of the target phrase, thereby identifying the similar phrases corresponding to the target phrase.

[0120] In practical applications, the target phrase is input into the feature extraction model to obtain the first word vector corresponding to the target phrase. This is the process of vectorizing the target phrase, also known as word embedding: vectorizing data and mapping it to the same semantic space.

[0121] Step 304: Obtain the second word vector of the pre-stored reference phrase, or input the reference phrase into the feature extraction model to obtain the second word vector corresponding to the reference phrase.

[0122] It should be noted that the feature vectors of each reference phrase can also be identified by a pre-trained feature extraction model, thereby obtaining the second word vectors corresponding to each reference phrase. That is, each reference phrase can be pre-input into the feature extraction model to obtain the feature vectors corresponding to each reference phrase output by the feature extraction model, and the feature vectors corresponding to each reference phrase can be used as the second word vectors.

[0123] In practical applications, the second word vectors corresponding to each reference phrase can be obtained in advance through the feature extraction model and stored. When it is necessary to obtain the second word vectors corresponding to each reference phrase again to determine the similarity between each reference phrase and the target phrase, the second word vectors corresponding to each reference phrase can be directly obtained from the storage space without having to analyze and extract them again through the feature extraction model.

[0124] In this embodiment, the system first checks whether the second word vectors corresponding to each reference phrase are stored in the storage space. If so, the second word vectors of the pre-stored reference phrases are directly obtained; otherwise, the reference phrases are input into the feature extraction model to obtain the second word vectors corresponding to the reference phrases. That is, the word vectors of each reference phrase only need to be identified once by the feature extraction model, and can be directly obtained from the storage space afterward, saving processing resources and improving the efficiency of similar phrase determination.

[0125] Step 306: Determine the similarity between the first word vector and the second word vector of the reference phrase.

[0126] It should be noted that the cosine value of two vectors can represent the distance between the two vectors, that is, the degree of similarity between the two vectors. The higher the cosine value, the more similar the two vectors are. Therefore, we can determine the cosine value between the first word vector and the second word vector of each reference phrase, and use this cosine value as the similarity. Subsequently, based on the similarity between the first word vector of the target phrase and the second word vector of each reference phrase, we can determine the similar phrases corresponding to the target phrase.

[0127] Of course, in practical applications, in addition to calculating the similarity between the first word vector and the second word vector based on the cosine value between the two vectors, other methods can be used to calculate the similarity between the first word vector and the second word vector, such as Euclidean distance, Pearson correlation coefficient, Manhattan distance or Mahalanobis distance.

[0128] Step 308: Sort the obtained similarity scores and select the target number of similar phrases based on the sorting results.

[0129] It should be noted that higher similarity indicates a greater similarity between the corresponding reference phrase and the target phrase. Therefore, the similarity scores can be ranked, and a target number of similar phrases can be selected based on the ranking results. In practical applications, the target number is the number of expanded sample corpora corresponding to the target phrase. This target number can be preset based on the specific details of the original sample corpus. For example, the target number could be 3, 5, 8, etc.

[0130] For example, for the target phrase A, its target value is determined to be 3. Based on the sorting results, three similar phrases are selected, namely A1, A2, and A3. At this time, the target phrase A in the first original sample corpus can be replaced with A1, A2, and A3 respectively, and three extended sample corpora corresponding to the target phrase A can be obtained.

[0131] In this embodiment, the first word vector of the target phrase and the second word vector of each reference phrase can be obtained accurately and efficiently through the feature analysis model. Based on the similarity between the two word vectors, the similarity between the target phrase and the reference phrases is represented. Then, similar phrases that are similar to the target phrase are selected from each reference phrase. The process of determining similar phrases is simple and efficient.

[0132] It should be noted that the phrase feature information can be word frequency, and the preset replacement condition is that the word frequency is less than the word frequency threshold. In this case, step 104 above can also be achieved through, for example... Figure 4 The method for determining similar phrases shown is implemented as follows: Figure 4 The flowchart illustrates another method for determining similar phrases according to an embodiment of this application, which specifically includes the following steps:

[0133] Step 402: Input the target phrase into the feature extraction model to obtain the first word vector corresponding to the target phrase.

[0134] Step 404: Obtain the second word vector of the pre-stored reference phrase, or input the reference phrase into the feature extraction model to obtain the second word vector corresponding to the reference phrase.

[0135] Step 406: Determine the similarity between the first word vector and the second word vector of the reference phrase.

[0136] Step 408: Determine the target value based on the difference between the word frequency corresponding to the target phrase and the word frequency threshold.

[0137] It should be noted that the word frequency corresponding to the target phrase is the frequency of the target phrase appearing in at least one original sample corpus. The word frequency threshold is a value used to determine whether the word frequency of the target phrase is too low, thereby determining whether there is too little semantic information related to the target phrase in at least one original sample corpus. This word frequency threshold can be set relatively small. If the word frequency threshold is set too high, it may result in too few or even no target phrases being selected for replacement, leading to a low degree of expansion of the training corpus. For example, the word frequency threshold can be set to 5, 10, 15, etc.

[0138] In practical applications, the target phrase is a phrase whose frequency has not reached the frequency threshold. Therefore, the target value can be determined based on the difference between the frequency of the target phrase and the frequency threshold. That is, based on the difference between the frequency of the target phrase and the frequency threshold, we can determine how many similar phrases need to be selected, and then construct several expanded sample corpora to increase the frequency of the target phrase in the training corpus. In other words, the determined target value is the number of similar phrases that need to be selected for the target phrase, and the expanded sample corpus is expanded according to this number.

[0139] As an example, the difference between the word frequency corresponding to the target phrase and the word frequency threshold can be used as the target value, or a predetermined multiple of the difference between the word frequency corresponding to the target phrase and the word frequency threshold can be used as the target value.

[0140] Step 410: Sort the obtained similarity scores and select the target number of similar phrases based on the sorting results.

[0141] In this embodiment, the number of similar phrases can be determined based on the difference between the word frequency of the target phrase and the word frequency threshold, thereby filtering out a corresponding number of similar phrases. Subsequently, the target phrase in the original sample corpus can be replaced with the corresponding number of similar phrases in turn, increasing the frequency of semantic information related to the target phrase in the training corpus, enriching the training corpus and semantic information, and improving the training effect of the text processing model.

[0142] It should be noted that step 108 above can also be achieved through methods such as... Figure 5 The method for constructing the training corpus shown is implemented. Figure 5 A flowchart of another training corpus construction method according to an embodiment of this application is shown, which specifically includes the following steps:

[0143] Step 502: Score the expanded sample corpus using the evaluation model.

[0144] It should be noted that the evaluation model can refer to a pre-trained model that can evaluate whether the input sentence conforms to the grammatical rules. In other words, the evaluation model can analyze the semantic information and grammatical structure of the input extended sample corpus and output a score for the extended sample corpus. This score can represent the reasonableness of the extended sample corpus. The higher the score, the more the extended sample corpus conforms to the real grammatical rules.

[0145] The evaluation model can be an n-gram language model.

[0146] In practical applications, an evaluation model capable of scoring extended sample corpora can be pre-trained. This evaluation model can be trained using the following method:

[0147] Obtain an evaluation training sample set, wherein the evaluation training sample set includes at least one evaluation sample corpus, and each evaluation sample corpus carries a score label;

[0148] Input each evaluation sample corpus from the evaluation training sample set into the initial evaluation model to obtain the prediction score corresponding to each evaluation sample corpus output by the initial evaluation model.

[0149] Based on the score labels and predicted scores corresponding to each evaluation sample corpus, the loss value of the initial evaluation model is calculated. According to the loss value, the model parameters of the initial evaluation model are adjusted, and the operation steps of obtaining the evaluation training sample set are returned until the training stopping condition is reached, and the trained evaluation model is obtained.

[0150] Among them, the score label can refer to the score manually annotated in advance based on the semantic information and grammatical structure of the evaluation sample corpus.

[0151] In one possible implementation, the training stopping condition can be determined solely based on the relationship between the loss value and the loss threshold. Specifically, if the loss value is greater than or equal to the loss threshold, it indicates a significant difference between the score labels and predicted scores for each evaluation sample corpus, suggesting poor recognition ability of the initial evaluation model. In this case, it can be determined that the training stopping condition has not been met. The model parameters of the initial evaluation model are adjusted, and the steps to obtain the evaluation training sample set are returned to continue training the initial evaluation model until the loss value is less than the loss threshold. This indicates a smaller difference between the score labels and predicted scores for each evaluation sample corpus, confirming that the training stopping condition has been met. Training is then stopped, and the trained evaluation model is obtained.

[0152] It should be noted that the loss value of the initial evaluation model can be calculated based on the score labels and predicted scores corresponding to each evaluation sample corpus. The score label refers to the actual result that the initial evaluation model wants to output, that is, the score label is the true result. When the evaluation sample corpus is input into the initial evaluation model, the output predicted score is the prediction result of the initial evaluation model. When the difference between the predicted result and the true result is small enough, it means that the predicted result is close enough to the true result. At this time, the initial evaluation model training is completed, and the trained evaluation model is obtained.

[0153] The loss threshold is a critical value for the loss. If the loss value is greater than or equal to the loss threshold, it indicates that there is still a certain deviation between the prediction result of the initial evaluation model and the actual result, and the model parameters of the initial evaluation model still need to be adjusted. At this point, the training stopping condition has not been met. If the loss value is less than the loss threshold, it indicates that the prediction result of the initial evaluation model is close enough to the actual result, and training can be stopped. At this point, the training stopping condition can be met. The specific value of the loss threshold is selected according to the actual situation, and this application embodiment does not impose any limitation on it.

[0154] In this embodiment of the application, the difference between the prediction result and the actual result of the initial evaluation model can be intuitively shown by calculating the loss value. Subsequently, the initial evaluation model can be trained in a targeted manner based on the difference, and the parameters of the initial evaluation model can be adjusted, thereby effectively improving the training rate and training effect of the initial evaluation model.

[0155] In another possible implementation, besides comparing the relationship between the loss value and the loss threshold, the number of iterations can also be considered to determine whether the training stopping condition has been met. Specifically, if the loss value is greater than the loss threshold, it can be further determined whether the current number of iterations has reached the preset number of iterations. If the current number of iterations has not reached the preset number of iterations, it can be determined that the training stopping condition has not been met. The model parameters of the initial evaluation model can be adjusted, and the operation steps of obtaining the evaluation training sample set can be returned to continue training the initial evaluation model until the loss value is less than or equal to the loss threshold, or the number of iterations reaches the preset number of iterations. At this point, the training stopping condition is determined to have been met, the iteration is stopped, and the trained evaluation model is obtained.

[0156] The preset number of iterations is set according to the actual situation. This application embodiment does not limit it in any way. When the number of training iterations reaches the preset number, it means that the training of the initial evaluation model has been sufficient. At this time, the prediction result of the initial evaluation model is very close to the real result, and training can be stopped.

[0157] In practical applications, there are many loss functions used to calculate the loss value, such as cross-entropy loss function, L1 norm loss function, maximum loss function, mean squared error loss function, and log loss function. Loss functions can be used to evaluate the degree of difference between the model's predictions and the actual results. Generally, the better the loss function, the better the model's performance. Different types of models typically use different loss functions.

[0158] The cross-entropy loss function is essentially a log-likelihood function, which can be used in binary and multi-class classification tasks. When the calculated loss value is large, it indicates that the error between the true result and the predicted result is large. At this time, the model weights can be updated quickly. When the calculated loss value is small, it indicates that the error between the true result and the predicted result is small. At this time, the model weights can be updated slowly. Using the cross-entropy loss function can avoid gradient vanishing when calculating gradients.

[0159] In practical applications, models are generally divided into two categories: classification models and regression models. For classification models, the cross-entropy loss function is generally preferred to avoid gradient vanishing. The initial evaluation model in this application is actually designed to identify the scores of the evaluation sample corpus, which is essentially a classification model. Therefore, the cross-entropy loss function is preferred to train the initial evaluation model in this embodiment.

[0160] In practice, based on the score labels and predicted scores corresponding to each evaluation sample corpus, the loss value of the initial evaluation model can be calculated using the following formula (1):

[0161]

[0162] Where LOSS represents the loss value of the initial evaluation model, N represents the number of categories that need to be classified by the initial evaluation model (i.e., how many score levels there are), y(i) represents the score label corresponding to each evaluation sample corpus, i.e. the true result, and f(x(i)) represents the predicted score corresponding to each evaluation sample corpus output by the classification function in the initial evaluation model, i.e. the prediction result.

[0163] In one possible implementation, the initial evaluation model is a neural network model. When adjusting the parameters of the initial evaluation model based on the calculated loss value, the weights of the initial evaluation model can be adjusted through backpropagation of the loss value. Specifically, after calculating the loss value of the initial evaluation model, this loss value can be passed to the last layer of the initial evaluation model. This loss value can represent the magnitude of the current error, so the last layer can automatically adjust its own weights according to the received loss value. Then, the last layer can pass the loss value back to the previous layer, and the previous layer, upon receiving the loss value, can also automatically adjust its own weights according to the received loss value. This process continues until the loss value is passed to the first layer of the initial evaluation model, completing one parameter adjustment cycle.

[0164] In this embodiment, the specific training status of the initial evaluation model can be determined based on the loss value, and the model parameters of the initial evaluation model can be adjusted in reverse based on the loss value if the training is unqualified, so as to improve the error detection capability of the initial evaluation model, resulting in a high training rate and good training effect.

[0165] Step 504: Select the fused sample corpus based on the scores of the extended sample corpus.

[0166] It should be noted that after scoring the extended sample corpus through the evaluation model, the top-ranked extended sample corpus with a set score can be selected as the fusion sample corpus based on the scores of each extended sample corpus, or the extended sample corpus with a score greater than the threshold can be selected as the fusion sample corpus.

[0167] Step 506: Merge the original sample corpus and the fused sample corpus to obtain the training corpus set.

[0168] In this embodiment, after obtaining the extended sample corpus, the extended sample corpus can be scored using an evaluation model. Based on the scores of the extended sample corpus, fused sample corpus is selected. Then, the original sample corpus and the fused sample corpus are merged to obtain a training corpus set. Since the extended sample corpus is constructed by replacing the target phrases in the original sample corpus with similar phrases, the extended sample corpus may not conform to semantic norms. Therefore, the constructed extended sample corpus can be further screened to remove those that do not conform to semantic norms. Then, the screened extended sample corpus that conforms to semantic norms is merged with the original sample corpus. That is, the union of the original sample corpus and the fused sample corpus is used as the training corpus set, which ensures the quality of the constructed training corpus set and thus guarantees the training effect of the subsequent text processing model.

[0169] It should be noted that the original sample corpus can be a bilingual corpus, which includes samples in the first language and corresponding samples in the second language. In this case, step 102 above can be performed as follows: Figure 6 The phrase dictionary construction method shown is implemented as follows: Figure 6 The flowchart illustrates another phrase dictionary construction method according to an embodiment of this application, which specifically includes the following steps:

[0170] Step 602: Perform word segmentation on the first language sample to obtain at least two first language segmented words, and perform word segmentation on the second language sample to obtain at least two second language segmented words.

[0171] In practical applications, at least one original sample corpus can be segmented using a pre-defined segmentation tool to obtain at least two segmented words. This facilitates the subsequent construction of corresponding phrases and a phrase dictionary based on the segmented words. The pre-defined segmentation tool is any tool capable of segmenting sentences to obtain segmented words, such as Jieba, pkuseg, or NLTK (Natural Language Processing Toolkit).

[0172] It should be noted that word segmentation methods can be divided into string matching-based word segmentation methods, understanding-based word segmentation methods, and statistical word segmentation methods, etc.

[0173] Among them, the word segmentation method based on string matching is also called the mechanical word segmentation method. It matches the Chinese character string to be analyzed with the entries in a "sufficiently large" machine dictionary according to a certain strategy. If a certain string is found in the dictionary, the match is successful (a word is identified). It includes forward maximum matching (from left to right), backward maximum matching (from right to left), minimum segmentation (minimizing the number of words segmented in each sentence), and bidirectional maximum matching (performing two scans from left to right and from right to left).

[0174] Understanding-based word segmentation methods achieve word recognition by having computers simulate human sentence comprehension. The basic idea is to perform syntactic and semantic analysis simultaneously with word segmentation, using syntactic and semantic information to handle ambiguity. It typically consists of three parts: a word segmentation subsystem, a syntactic-semantic subsystem, and a central control unit. Under the coordination of the central control unit, the word segmentation subsystem obtains syntactic and semantic information about words and sentences to determine word segmentation ambiguities; that is, it simulates the human process of sentence comprehension. This word segmentation method requires the use of a large amount of linguistic knowledge and information.

[0175] Statistical word segmentation methods take a large amount of already segmented text as input and use statistical machine learning models to learn the rules of word segmentation (called training) to segment unknown text. Examples include maximum probability segmentation and maximum entropy segmentation. Major statistical models include N-gram models, Hidden Markov Models (HMMs), Maximum Entropy Models (MEs), and Conditional Random Fields (CRFs).

[0176] In practical applications, string matching-based word segmentation methods can be combined with statistical word segmentation methods. This leverages the fast segmentation speed of string matching-based methods while utilizing the advantages of statistical word segmentation in identifying new words and automatically eliminating ambiguity by combining context, as exemplified by Jieba word segmentation.

[0177] It should be noted that bilingual corpora include data in two languages, namely the source language and the corresponding translation, such as Chinese-English parallel corpora, Chinese-French parallel corpora, etc. A bilingual corpus includes a first language sample and a corresponding second language sample. The first language sample is the source language, and the second language sample is the corresponding translation sample of the first language sample. Therefore, the first language sample can be segmented to obtain at least two first language segmented words, and the second language sample can be segmented simultaneously to obtain at least two second language segmented words. This facilitates the subsequent construction of a phrase dictionary based on the segmentation results of the first and second language samples.

[0178] Step 604: Perform word alignment processing on the words segmented in the first language and the words segmented in the second language to obtain word alignment information.

[0179] It should be noted that when segmenting the first language sample and the second language sample, in addition to obtaining the segmented words of the first language and the second language, the position information of each word can also be obtained.

[0180] In practical applications, sentences in different languages ​​have special grammatical structures, and the words obtained from the segmentation may not correspond one-to-one. Therefore, based on the grammatical structures corresponding to the first language sample and the second language sample, word alignment processing can be performed on the segmented words of the first language and the segmented words of the second language to obtain word alignment information. This word alignment information can be the correspondence information between the segmented words of the first language and the segmented words of the second language. After obtaining the word alignment information, it can be stored for easy querying when replacing the target phrase later.

[0181] Step 606: According to the preset grammar rules, extract the first language phrase from at least two first language segmented words and determine the phrase feature information of the first language phrase.

[0182] Step 608: Based on word alignment information, determine the second language phrase corresponding to the first language phrase from at least two second language segmented words.

[0183] It should be noted that since the second language sample is the translation sample corresponding to the first language sample, after extracting the first language phrase from at least two first language segmented words according to the preset grammatical rules, the second language segmented words corresponding to the first language phrase can be determined from at least two second language segmented words based on word alignment information. The determined second language segmented words can form the second language phrase corresponding to the first language phrase.

[0184] Step 610: Construct a phrase dictionary based on the first language phrase, the corresponding second language phrase, and the corresponding phrase feature information.

[0185] In this embodiment, the first language sample and the second language sample in the bilingual corpus can be segmented into words respectively to obtain the corresponding first language segmented words and second language segmented words. Words are selected from the first language segmented words to form first language phrases. Then, based on the second language segmented words corresponding to the first language phrases, second language phrases corresponding to the first language phrases are constructed, thereby building a phrase dictionary. Since the meaning of a phrase is often fixed in different sentences, the problem of polysemy is avoided, thus avoiding ambiguity and grammatical errors in the subsequent extended sample corpus and improving the quality of the extended sample corpus.

[0186] It should be noted that the original sample corpus is a bilingual corpus, which includes samples in the first language and corresponding samples in the second language. The phrase dictionary is constructed based on the word alignment information of the first and second language samples. Step 104 above can be achieved through methods such as... Figure 7 The method for determining similar phrases shown is implemented as follows: Figure 7 The flowchart illustrates another method for determining similar phrases according to an embodiment of this application, which specifically includes the following steps:

[0187] Step 702: Based on the first word vector of the target first language phrase and the second word vector of the reference first language phrase in the preset first language phrase library, determine the similar first language phrases corresponding to the target first language phrase, wherein the target first language phrase is any first language phrase in the phrase dictionary whose phrase feature information satisfies the preset replacement conditions.

[0188] It should be noted that, firstly, based on the phrase feature information of each first language phrase in the phrase dictionary, the first language phrases that meet the preset replacement conditions can be selected from each first language phrase. Any one of the selected first language phrases to be replaced can be taken as the target first language phrase. Then, based on the first word vector of the target first language phrase and the second word vector of each reference first language phrase in the preset first language phrase library, the similar first language phrases corresponding to the target first language phrase can be determined from each reference first language phrase. That is, each first language phrase to be replaced selected can be used as the target first language phrase, and the corresponding similar first language phrase can be determined. This similar first language phrase can then be used to replace the target first language phrase in the first language sample of the original sample corpus.

[0189] When determining similar first language phrases corresponding to the target first language phrase from various reference first language phrases, the similar first language phrases can be determined based on the vector similarity between the first word vector of the target first language phrase and the second word vectors of various reference first language phrases in the preset first language phrase library. For specific determination methods, please refer to the relevant content in Figure 3 above.

[0190] Step 704: Based on the word alignment information, determine the target second language phrase corresponding to the target first language phrase.

[0191] It should be noted that since the second language sample is the translation sample corresponding to the first language sample, after the target first language phrase in the first language sample is determined, the target second language phrase corresponding to the target first language phrase can be determined based on word alignment information, which facilitates the subsequent determination of similar second language phrases of the target second language phrase.

[0192] Step 706: Based on the first word vector of the target second language phrase and the second word vector of the reference second language phrase in the preset second language phrase library, determine the similar second language phrases corresponding to the target second language phrase.

[0193] It should be noted that after obtaining the similar first language phrases corresponding to the target first language phrase, the similar second language phrases corresponding to the target second language phrases can be determined from each reference second language phrase based on the first word vector of the target second language phrase and the second word vectors of each reference second language phrase in the preset second language phrase library. These similar second language phrases can then be used to replace the target second language phrases in the second language samples of the original sample corpus, thereby obtaining the expanded bilingual corpus after replacing the similar phrases.

[0194] When determining similar second language phrases corresponding to the target second language phrase from various reference second language phrases, the similarity can be determined based on the vector similarity between the first word vector of the target second language phrase and the second word vectors of various reference second language phrases in the preset second language phrase library. The specific determination method can also be found above. Figure 3 Related content.

[0195] In this embodiment, the feature analysis model can accurately and efficiently obtain the first word vectors of the target first language phrase and the target second language phrase, as well as the second word vectors of each reference first language phrase and reference second language phrase. The similarity between the first word vector of the target first language phrase and the second word vector of the reference first language phrase can represent the similarity between the target first language phrase and the reference first language phrase; the similarity between the first word vector of the target second language phrase and the second word vector of the reference second language phrase can also represent the similarity between the target second language phrase and the reference second language phrase. Therefore, similar first language phrases and similar second language phrases that are similar to the target first language phrase and the target second language phrase can be selected from each reference first language phrase and reference second language phrase. The process of determining similar phrases is simple and efficient.

[0196] It should be noted that when the original sample corpus is a bilingual corpus, and the bilingual corpus includes samples in the first language and corresponding samples in the second language, step 106 above can be performed as follows: Figure 8 The method for obtaining the extended sample corpus shown is implemented as follows: Figure 8 The flowchart illustrates a method for obtaining extended sample corpus according to an embodiment of this application, which specifically includes the following steps:

[0197] Step 802: Replace the target first language phrase in the first language sample of the first original sample corpus with a similar first language phrase to obtain an extended first language sample.

[0198] Step 804: Based on word alignment information, replace the target second language phrase in the second language sample of the first original sample corpus with a similar second language phrase to obtain an extended second language sample.

[0199] Step 806: Combine the extended first language sample with the corresponding extended second language sample as the extended sample corpus.

[0200] In this embodiment, each original sample corpus containing a target first language phrase in its first language sample can be used as the first original sample corpus. The target first language phrase in the first language sample of the first original sample corpus is replaced with a similar first language phrase to obtain an extended first language sample. Similarly, each original sample corpus containing a target second language phrase in its second language sample can be used as the first original sample corpus. The target second language phrase in the second language sample of the first original sample corpus is replaced with a similar second language phrase to obtain an extended second language sample. Only the first language samples and the second language samples are combined (by taking the union) to obtain an extended bilingual corpus, i.e., an extended sample corpus, thereby expanding the training corpus and improving the training effect of the text processing model.

[0201] Figure 9 The flowchart illustrates a training method for a text processing model according to an embodiment of this application, specifically including the following steps:

[0202] Step 902: Obtain the training corpus set, wherein the training corpus set is constructed using the above-described training corpus set construction method, and the training corpus set includes at least one sample corpus, the sample corpus including sample labels.

[0203] Step 904: Input at least one sample corpus into the text processing model to obtain the prediction processing result corresponding to at least one sample corpus.

[0204] Step 906: Based on the prediction results and sample labels, determine the loss value of the text processing model, adjust the parameters of the text processing model based on the loss value, and return to the operation steps of obtaining the training corpus until the training stopping condition is met, and obtain the trained text processing model.

[0205] The training method for the text processing model provided in this application obtains a training corpus containing rich sample data, that is, rich semantic information, and the quality of the training corpus is high. Training the text processing model based on this training corpus can improve the training effect of the text processing model and ensure the training accuracy of the text processing model.

[0206] Figure 10 A flowchart illustrating a training method for another text processing model according to an embodiment of this application is shown. This text processing model is a translation model, and specifically includes the following steps:

[0207] Step 1002: Obtain the training corpus set, wherein the training corpus set is constructed using the above-described training corpus set construction method, and the training corpus set includes at least one sample corpus, the sample corpus includes a first language sample and a corresponding second language sample, and the second language sample is the sample label of the first language sample.

[0208] Step 1004: Input the first language sample from at least one sample corpus into the text processing model to obtain the predicted translation result corresponding to the first language sample.

[0209] Step 1006: Based on the predicted translation results corresponding to the first language sample and the second language sample, determine the loss value of the text processing model, adjust the parameters of the text processing model based on the loss value, and return to execute step 1002 until the training stopping condition is met, and obtain the trained text processing model.

[0210] The training method for the text processing model provided in this application obtains a training corpus containing rich sample data, that is, rich semantic information, and the quality of the training corpus is high. Training the text processing model based on this training corpus can improve the training effect of the text processing model and ensure the training accuracy of the text processing model.

[0211] Figure 11 The flowchart illustrates a training method for a Chinese-English translation model according to an embodiment of this application, specifically including the following steps:

[0212] Step 1102: Obtain the initial Chinese-English parallel corpus, perform data cleaning on the obtained Chinese-English parallel corpus, and obtain the original Chinese-English parallel corpus.

[0213] Data cleaning mainly includes deduplication, removal of blank lines, conversion of traditional Chinese to simplified Chinese, filtering of sentences with abnormal length and excessive garbled characters.

[0214] Step 1104: Perform word segmentation and word alignment processing on the original Chinese-English parallel corpus to obtain Chinese word segmentation words, English word segmentation words, and word alignment information.

[0215] For Chinese word segmentation, the open-source word segmentation tool pkuseg can be used, while for English word segmentation, NLTK can be used.

[0216] Step 1106: Extract Chinese phrases from the Chinese word segmentation words and determine the frequency of each Chinese phrase. Based on word alignment information and English word segmentation words, extract the corresponding English phrases for the Chinese phrases and count the frequency of each English phrase. Based on each Chinese phrase and its corresponding frequency, and each English phrase and its frequency, construct a phrase dictionary.

[0217] It should be noted that a frequency threshold can be preset, and phrases in the phrase dictionary with a frequency below the threshold will be the main phrases to be replaced later.

[0218] Step 1108: Identify the target Chinese phrases in the original Chinese-English parallel corpus that have a frequency less than a threshold, and use the Chinese BERT model to generate the vector of the target Chinese phrase, as well as the vectors of each reference Chinese phrase in the preset Chinese phrase library.

[0219] Step 1110: Determine the similarity between the vector of the target Chinese phrase and the vectors of each reference Chinese phrase, sort the similarity from high to low, and select the top-ranked reference Chinese phrases as the similar Chinese phrases corresponding to the target Chinese phrase.

[0220] The target value is the difference between the word frequency of the target phrase in the constructed phrase dictionary and the word frequency threshold.

[0221] Step 1112: Based on word alignment information, determine the target English phrase corresponding to the target Chinese phrase in the original Chinese-English parallel corpus.

[0222] Step 1114: Use the English BERT model to generate the vector of the target English phrase, as well as the vectors of each reference English phrase in the preset English phrase library.

[0223] Step 1116: Determine the similarity between the vector of the target English phrase and the vectors of each reference English phrase, sort the similarity from high to low, and select the top-ranked reference English phrases as the similar English phrases corresponding to the target English phrase.

[0224] Step 1118: Replace the target Chinese phrase in the original Chinese-English parallel corpus with similar Chinese phrases, and replace the target English phrase in the original Chinese-English parallel corpus with similar English phrases to obtain the expanded Chinese-English parallel corpus.

[0225] Step 1120: Score each extended Chinese-English parallel corpus using an n-gram language model, and select the fused Chinese-English parallel corpus based on the scores of each extended Chinese-English parallel corpus.

[0226] Step 1122: Merge the original Chinese-English parallel corpus and the fused Chinese-English parallel corpus to obtain the training corpus set.

[0227] Step 1124: Train the translation model based on the training corpus.

[0228] The text processing model training method provided in this application obtains a training corpus containing rich Chinese-English parallel corpora, which contains rich semantic information and is of high quality. Training the translation model based on this training corpus can improve the training effect of the translation model, ensure the training accuracy of the translation model, and thus ensure the accuracy of subsequent translation.

[0229] Figure 12 A flowchart of a text processing method according to an embodiment of this application is shown, which specifically includes the following steps:

[0230] Step 1202: Obtain the text to be processed.

[0231] Step 1204: Input the text to be processed into the text processing model to obtain the target processing result corresponding to the text to be processed. The text processing model is trained using the training corpus obtained by the above training corpus construction method.

[0232] The text processing method provided in this application can obtain the text to be processed, input the text to be processed into the text processing model, and obtain the target processing result corresponding to the text to be processed. Since the text processing model is trained by the training corpus obtained by the above training corpus construction method, rich sample corpus is used in the training process, and the training accuracy is high. Therefore, the text processing model trained by the above method has high accuracy in processing text.

[0233] Corresponding to the above method embodiments, this application also provides an embodiment of a training corpus construction apparatus. Figure 13 A schematic diagram of a training corpus construction apparatus according to an embodiment of this application is shown. Figure 13 As shown, the device includes:

[0234] The first construction module 1302 is configured to construct a phrase dictionary based on at least one original sample corpus obtained, wherein the phrase dictionary includes phrases and phrase feature information corresponding to the phrases;

[0235] The replacement module 1304 is configured to replace the target phrase in the first original sample corpus with a similar phrase to obtain an extended sample corpus, wherein the first original sample corpus is any original sample corpus that includes the target phrase in at least one original sample corpus.

[0236] The second construction module 1306 is configured to construct a training corpus set based on the original sample corpus and the extended sample corpus.

[0237] Optionally, the first building module 1302 is further configured as follows:

[0238] Perform word segmentation on at least one original sample corpus to obtain at least two segmented words;

[0239] According to the preset grammar rules, extract phrases from at least two word segments and determine the phrase feature information of the phrases;

[0240] A phrase dictionary is constructed based on phrases and their corresponding phrase feature information.

[0241] Optionally, the device further includes a determining module configured to:

[0242] Identify target phrases in the phrase dictionary whose phrase feature information satisfies preset replacement conditions;

[0243] Based on the first word vector of the target phrase and the second word vector of the reference phrase in the preset phrase library, the similar phrases corresponding to the target phrase are determined.

[0244] Optionally, the determination module is further configured as follows:

[0245] Determine the similarity between the first word vector and the second word vector of the reference phrase;

[0246] The obtained similarity scores are sorted, and a target number of similar phrases are selected based on the sorting results.

[0247] Optionally, the phrase feature information is word frequency, and the preset replacement condition is that the word frequency is less than the word frequency threshold;

[0248] Optionally, the determination module is further configured as follows:

[0249] The target value is determined based on the difference between the word frequency corresponding to the target phrase and the word frequency threshold.

[0250] Optionally, the second building module 1306 is further configured as follows:

[0251] The expanded sample corpus was scored using an evaluation model;

[0252] Based on the scores of the extended sample corpus, the fused sample corpus is selected;

[0253] The original sample corpus and the fused sample corpus are merged to obtain the training corpus set.

[0254] Optionally, the original sample corpus is a bilingual corpus, which includes samples in the first language and corresponding samples in the second language; the first construction module 1302 is further configured as follows:

[0255] Perform word segmentation on the first language sample to obtain at least two first language segmented words, and perform word segmentation on the second language sample to obtain at least two second language segmented words;

[0256] Word alignment processing is performed on the words segmented from the first language and the words segmented from the second language to obtain word alignment information;

[0257] According to the preset grammatical rules, extract the first language phrases from at least two first language segmented words and determine the phrase feature information of the first language phrases;

[0258] Based on word alignment information, determine the second language phrase corresponding to the first language phrase from at least two second language segmented words;

[0259] A phrase dictionary is constructed based on the phrases in the first language, the corresponding phrases in the second language, and the corresponding phrase feature information.

[0260] Optionally, the original sample corpus is a bilingual corpus, which includes samples in a first language and corresponding samples in a second language. The phrase dictionary is constructed based on word alignment information of the first language samples and the second language samples. The determining module is further configured to:

[0261] Based on the first word vector of the target first language phrase and the second word vector of the reference first language phrase in the preset first language phrase library, the similar first language phrases corresponding to the target first language phrase are determined. The target first language phrase is any first language phrase in the phrase dictionary whose phrase feature information satisfies the preset replacement conditions.

[0262] Based on word alignment information, determine the target second language phrase corresponding to the target first language phrase;

[0263] Based on the first word vector of the target second language phrase and the second word vector of the reference second language phrase in the preset second language phrase library, determine the similar second language phrases corresponding to the target second language phrase.

[0264] Optionally, the replacement module 1304 is further configured as follows:

[0265] The target first language phrase in the first language sample of the first original sample corpus is replaced with a similar first language phrase to obtain an extended first language sample;

[0266] Based on word alignment information, the target second language phrase in the second language sample of the first original sample corpus is replaced with a similar second language phrase to obtain an extended second language sample;

[0267] The extended first language sample and the corresponding extended second language sample are used as the extended sample corpus.

[0268] The training corpus construction device provided in this application can replace target phrases in the original sample corpus based on similar phrases to obtain extended sample corpus. Since the meaning of a phrase is often fixed in different sentences, it avoids the problem of polysemy, thus avoiding ambiguity and grammatical errors in the extended sample corpus and improving its quality. Then, the original sample corpus and the obtained extended sample corpus can be merged to construct a training corpus, which greatly enriches the training corpus, increases the semantic information included in the training corpus, and the quality of the extended training corpus is high. Training a text processing model based on this training corpus can improve the training effect of the text processing model, ensure the accuracy of the text processing model, and thus ensure the accuracy of subsequent text processing using the text processing model.

[0269] The above is a schematic scheme of a training corpus construction device according to this embodiment. It should be noted that the technical solution of this training corpus construction device and the technical solution of the training corpus construction method described above belong to the same concept. For details not described in detail in the technical solution of the training corpus construction device, please refer to the description of the technical solution of the training corpus construction method described above.

[0270] Corresponding to the above method embodiments, this application also provides embodiments of a training device for a text processing model. Figure 14 This diagram illustrates the structure of a training apparatus for a text processing model according to an embodiment of this application. Figure 14 As shown, the device includes:

[0271] The second acquisition module 1402 is configured to acquire a training corpus set, wherein the training corpus set is constructed by the above-mentioned training corpus set construction method, and the training corpus set includes at least one sample corpus, the sample corpus including sample labels.

[0272] The second input module 1404 is configured to input at least one sample corpus into the text processing model to obtain the prediction processing result corresponding to at least one sample corpus.

[0273] The adjustment module 1406 is configured to determine the loss value of the text processing model based on the prediction processing results and sample labels, adjust the parameters of the text processing model based on the loss value, and return to execute the operation steps of obtaining the training corpus until the training stopping condition is met, thereby obtaining the trained text processing model.

[0274] Optionally, the text processing model is a translation model, and the sample corpus includes samples in a first language and corresponding samples in a second language, wherein the second language samples are sample labels of the first language samples; the second input module 1404 is further configured as follows:

[0275] Input the first language sample from at least one sample corpus into the text processing model to obtain the predicted translation result corresponding to the first language sample;

[0276] Accordingly, the adjustment module 1406 is further configured as follows:

[0277] The loss value of the text processing model is determined based on the predicted translation results corresponding to the first language sample and the second language sample.

[0278] The training apparatus for the text processing model provided in this application can acquire a training corpus constructed using the aforementioned training corpus construction method. This training corpus includes at least one sample corpus, each containing sample labels. Then, the at least one sample corpus can be input into the text processing model to obtain a prediction processing result corresponding to that sample corpus. Subsequently, based on the prediction processing result and the sample labels, the loss value of the text processing model is determined. The parameters of the text processing model are adjusted based on the loss value, and the process of acquiring the training corpus is repeated until the training stopping condition is met, resulting in a fully trained text processing model. In this case, the acquired training corpus includes rich sample corpus, i.e., it contains rich semantic information. Training the text processing model based on this training corpus can improve the training effect of the text processing model and ensure its training accuracy.

[0279] The above is a schematic scheme of a text processing model training device according to this embodiment. It should be noted that the technical solution of this text processing model training device and the technical solution of the text processing model training method described above belong to the same concept. For details not described in detail in the technical solution of the text processing model training device, please refer to the description of the technical solution of the text processing model training method described above.

[0280] Corresponding to the above method embodiments, this application also provides text processing apparatus embodiments. Figure 15 A schematic diagram of the structure of a text processing apparatus according to an embodiment of this application is shown. Figure 15 As shown, the device includes:

[0281] The first acquisition module 1502 is configured to acquire the text to be processed.

[0282] The first input module 1504 is configured to input the text to be processed into the text processing model to obtain the target processing result corresponding to the text to be processed. The text processing model is trained using the training corpus obtained by the above-mentioned training corpus construction method.

[0283] The text processing apparatus provided in this application can acquire the text to be processed, input the text to be processed into the text processing model, and obtain the target processing result corresponding to the text to be processed. Since the text processing model is trained by the training corpus obtained by the above-mentioned training corpus construction method, rich sample corpus is used in the training process, and the training accuracy is high. Therefore, the text processing model trained by the above method has high accuracy in processing text.

[0284] The above is an illustrative scheme of a text processing device according to this embodiment. It should be noted that the technical solution of this text processing device and the technical solution of the above-described text processing method belong to the same concept. For details not described in detail in the technical solution of the text processing device, please refer to the description of the technical solution of the above-described text processing method.

[0285] Furthermore, the components in the device embodiment should be understood as functional modules necessary to implement each step of the program flow or method, and these functional modules are not actual functional divisions or separations. A device claim defined by such a set of functional modules should be understood as a functional module architecture that primarily implements the solution through the computer program described in the specification, and not as a physical device that primarily implements the solution through hardware.

[0286] Figure 16 A structural block diagram of a computing device 1600 according to an embodiment of this application is shown. The components of the computing device 1600 include, but are not limited to, a memory 1610 and a processor 1620. The processor 1620 is connected to the memory 1610 via a bus 1630, and a database 1650 is used to store data.

[0287] The computing device 1600 also includes an access device 1640, which enables the computing device 1600 to communicate via one or more networks 1660. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 1640 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Controller (NIC)), such as an IEEE 802.11 Wireless Local Area Networks (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0288] In one embodiment of this application, the aforementioned components of the computing device 1600 and Figure 16 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 16 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this application. Those skilled in the art can add or replace other components as needed.

[0289] The computing device 1600 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 1600 can also be a mobile or stationary server.

[0290] The processor 1620 is used to execute computer-executable instructions for methods of constructing training corpora, training text processing models, or text processing methods.

[0291] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the above-described training corpus construction method, text processing model training method, or text processing method. For details not described in detail in the technical solution of the computing device, please refer to the description of the above-described training corpus construction method, text processing model training method, or text processing method.

[0292] One embodiment of this application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, are used to implement a training corpus construction method, a text processing model training method, or a text processing method.

[0293] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the above-described training corpus construction method, text processing model training method, or text processing method. For details not described in detail in the technical solution of the storage medium, please refer to the description of the above-described training corpus construction method, text processing model training method, or text processing method.

[0294] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0295] Computer instructions include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0296] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0297] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0298] The preferred embodiments disclosed above are merely illustrative of this application. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this application. These embodiments are selected and specifically described in this application to better explain the principles and practical applications of this application, thereby enabling those skilled in the art to better understand and utilize this application. This application is limited only by the claims and their full scope and equivalents.

Claims

1. A training corpus construction method, characterized by, The method comprises the following steps: According to the obtained at least one original sample corpus, a phrase dictionary is constructed, wherein the phrase dictionary comprises a phrase and corresponding phrase feature information of the phrase, the phrase feature information is parameter information describing the relevant characteristics of the phrase in the at least one original sample corpus, and the relevant characteristic parameter information comprises at least one of the number of occurrences of the phrase in the at least one original sample corpus, the proportion of the number of occurrences of the phrase in the at least one original sample corpus in the total number of phrases, the position of the phrase in the at least one original sample corpus, and the part of speech; Replace the target phrase in the first original sample corpus with a target number of similar phrases to obtain an expanded sample corpus, wherein the first original sample corpus is any original sample corpus in the at least one original sample corpus that includes the target phrase, and the target number is determined based on the difference between the word frequency corresponding to the target phrase and a word frequency threshold; According to the original sample corpus and the expanded sample corpus, a training corpus set is constructed. 2.The training corpus construction method of claim 1, wherein, The method comprises the following steps: Tokenizing the at least one original sample corpus to obtain at least two tokenized words; According to a preset grammar rule, a phrase is extracted from the at least two tokenized words, and phrase feature information of the phrase is determined; According to the phrase and the corresponding phrase feature information of the phrase, the phrase dictionary is constructed. 3.The training corpus construction method of claim 1, wherein, Before the target phrase in the first original sample corpus is replaced with a similar phrase to obtain an expanded sample corpus, the method further comprises the following steps: Determine the target phrase in the phrase dictionary whose phrase feature information meets a preset replacement condition; According to the first word vector of the target phrase and the second word vector of a reference phrase in a preset phrase library, determine the similar phrase corresponding to the target phrase. 4.The training corpus construction method of claim 3, wherein, The method comprises the following steps: Determine the similarity between the first word vector and the second word vector of the reference phrase, respectively; Sort the obtained similarities, and select a target number of similar phrases according to the sorting result. 5.The training corpus construction method of claim 4, wherein, The phrase feature information is word frequency, and the preset replacement condition is that the word frequency is less than a word frequency threshold; Before the obtained similarities are sorted and a target number of similar phrases are selected according to the sorting result, the method further comprises the following steps: Determine the target number according to the difference between the word frequency corresponding to the target phrase and the word frequency threshold.

6. The training corpus construction method according to any one of claims 1-5, wherein, The method comprises the following steps: Score the expanded sample corpus by evaluating the model; Select a fusion sample corpus according to the score of the expanded sample corpus; Merge the original sample corpus and the fusion sample corpus to obtain the training corpus set.

7. The training corpus construction method according to any one of claims 1-5, wherein, The original sample corpus is a bilingual corpus, and the bilingual corpus comprises a first language sample and a corresponding second language sample; The method comprises the following steps: segmenting the first language sample to obtain at least two first language segmented words, and segmenting the second language sample to obtain at least two second language segmented words; performing word alignment processing on the first language segmented words and the second language segmented words to obtain word alignment information; extracting a first language phrase from the at least two first language segmented words according to a preset grammar rule, and determining phrase feature information of the first language phrase; determining a second language phrase corresponding to the first language phrase from the at least two second language segmented words according to the word alignment information; constructing the phrase dictionary according to the first language phrase, the corresponding second language phrase, and the corresponding phrase feature information. 8.The training corpus construction method of claim 3, wherein, The original sample corpus is a bilingual corpus, the bilingual corpus includes a first language sample and a corresponding second language sample, and the phrase dictionary is constructed based on word alignment information of the first language sample and the second language sample; The method for determining the similar phrase corresponding to the target phrase according to the first word vector of the target phrase and the second word vector of a reference phrase in a preset phrase library includes: determining a similar first language phrase corresponding to a target first language phrase according to a first word vector of the target first language phrase and a second word vector of a reference first language phrase in a preset first language phrase library, wherein the target first language phrase is any first language phrase in the phrase dictionary whose phrase feature information satisfies a preset replacement condition; determining a target second language phrase corresponding to the target first language phrase according to the word alignment information; determining a similar second language phrase corresponding to a target second language phrase according to a first word vector of the target second language phrase and a second word vector of a reference second language phrase in a preset second language phrase library. 9.The training corpus construction method of claim 8, wherein, The method for replacing the target phrase in the first original sample corpus with the similar phrase to obtain an expanded sample corpus includes: replacing a target first language phrase in a first language sample of the first original sample corpus with the similar first language phrase to obtain an expanded first language sample; replacing a target second language phrase in a second language sample of the first original sample corpus with the similar second language phrase to obtain an expanded second language sample based on the word alignment information; using the expanded first language sample and the corresponding expanded second language sample as the expanded sample corpus.

10. A text processing method characterized by, The method includes: obtaining a text to be processed; inputting the text to be processed into a text processing model to obtain a target processing result corresponding to the text to be processed, wherein the text processing model is trained by a training corpus set constructed by the training corpus set construction method of any one of claims 1-9.

11. A training corpus construction apparatus characterized by comprising: The method includes: The first construction module is configured to construct a phrase dictionary according to the obtained at least one original sample corpus, wherein the phrase dictionary comprises a phrase and corresponding phrase feature information of the phrase, the phrase feature information is parameter information describing relevant characteristics of the phrase in the at least one original sample corpus, and the relevant characteristic parameter information comprises at least one of the number of occurrences of the phrase in the at least one original sample corpus, the proportion of the number of occurrences of the phrase in the at least one original sample corpus in the total number of phrases, the position of the phrase in the at least one original sample corpus, and the part of speech; The replacement module is configured to replace a target phrase in a first original sample corpus with a target number of similar phrases to obtain an expanded sample corpus, wherein the first original sample corpus is any original sample corpus comprising the target phrase in the at least one original sample corpus, and the target number is determined based on a difference between the word frequency corresponding to the target phrase and a word frequency threshold; The second construction module is configured to construct a training corpus set according to the original sample corpus and the expanded sample corpus.

12. A text processing apparatus characterized by comprising: It comprises: The first acquisition module is configured to acquire a text to be processed; The first input module is configured to input the text to be processed into a text processing model to obtain a target processing result corresponding to the text to be processed, wherein the text processing model is trained by using a training corpus set constructed by the training corpus set construction method of any one of claims 1-9.

13. A computing device, comprising: It comprises: A memory and a processor; The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions to realize the steps of the training corpus set construction method of any one of claims 1-9 or the text processing method of claim 10.

14. A computer-readable storage medium storing computer instructions, wherein, The instructions are executed by the processor to realize the steps of the training corpus set construction method of any one of claims 1-9 or the text processing method of claim 10.

Images