Language expression model system, pre-training method, device, equipment, and medium

Abstract

[0009] Embodiments of the present disclosure provide a language representation model system, a language representation model pre-training method, a natural language processing method, a device, an apparatus, and a medium, wherein the language representation model system includes a character-granular language representation sub-model using characters as division units and a word-granular language representation sub-model using words as division units, wherein the character-granular language representation sub-model is used to output, based on a sentence using characters as division units, a first semantic vector corresponding to the meaning expressed by each divided character in the sentence, and the word-granular language representation sub-model is used to output, based on the sentence using words as division units, a second semantic vector corresponding to the meaning expressed by each divided word in the sentence. [0010] Solutions according to embodiments of the present disclosure provide a hybrid granular language representation model, providing a model foundation for downstream natural language processing tasks, helping to improve the processing accuracy of downstream natural language processing tasks and enhancing the transfer effect of the language representation model.

Description

[Technical field]

[0001] TECHNICAL FIELD The embodiments of the present disclosure relate to the field of computer technology, and in particular to a language expression model system, a language expression model pre-training method, a natural language processing method, an apparatus, an electronic device, and a storage medium. [Background technology]

[0002] With the development of artificial intelligence technology, applications related to natural language processing have been seen everywhere. Common applications related to natural language processing include machine translation, intelligent question-and-answer robots, machine reading comprehension, etc. The rapid development of applications related to natural language processing is largely due to the idea of ​​realizing transfer learning by language representation models. In the field of natural language processing, the essence of transfer learning is to first pre-train a language representation model using a sample set, and then perform secondary fine-tuning training on the language representation model obtained by pre-training according to a specific natural language processing task, so that the trained language representation model can perform natural language processing tasks with different functions, such as semantic analysis, text classification, or language reasoning. Summary of the Invention [Problem to be solved by the invention]

[0003] The embodiments of the present disclosure provide a language expression model system, a language expression model pre-training method, a natural language processing method, an apparatus, an electronic device, and a storage medium, collocation Units for natural language collocation To provide granular representation, and word Units for natural language word To provide a hybrid granular language representation model capable of performing granular representation, which provides a model foundation for downstream natural language processing tasks, helps to improve the processing accuracy of downstream natural language processing tasks, and enhances the transfer effect of the language representation model. [Means for solving the problem]

[0004] In a first aspect, an embodiment of the present disclosure provides a language expression model system, the system comprising: word is the division unit word A granular language representation sub-model; collocation is the division unit collocation A granular language representation sub-model; The above word Granular language representation sub-models are word Based on the sentence with each division unit, word is used to output a first semantic vector corresponding to a semantic expression expressed in the sentence; The above collocation Granular language representation sub-models are collocation Based on the above sentence with each division unit, collocation is used to output a second semantic vector corresponding to the semantic expression expressed in the sentence.

[0005] In a second aspect, an embodiment of the present disclosure further provides a method for pre-training a language representation model, the method including: determining a pre-training corpus sample; For the above language samples, collocation Units and word Divide by unit and collocation The division result and word obtaining a segmentation result; The above collocation Using the segmentation results, we use the language representation model collocation is the division unit collocation Pre-training is performed on the granular language representation sub-model. collocation obtaining a granular language representation sub-model; The above word Using the segmentation results, we use the language representation model word is the division unit word Pre-training is performed on the granular language representation sub-model. word Obtaining a granular linguistic representation sub-model.

[0006] In a third aspect, an embodiment of the present disclosure further provides a natural language processing method, the method comprising: determining a fine-tuning sample corpus based on a natural language processing task; For the fine-tuned sample language corpus, collocation Units and word Divide by unit and fine-tune collocation Segmentation results and fine-tuning word obtaining a segmentation result; Fine-tuning collocation Segmentation results and fine-tuning word using the segmentation results to fine-tune a pre-trained language representation model; and processing the target natural language using the fine-tuned language representation model.

[0007] In a fourth aspect, the present disclosure further provides an apparatus for pre-training a language representation model, the apparatus comprising: a determination module for determining pre-training corpus samples; For the above language samples, collocation Units and word Divide by unit and collocation The division result and word a segmentation module for obtaining a segmentation result; The above collocation Using the segmentation results, we use the language representation model collocation is the division unit collocation Pre-training is performed on the granular language representation sub-model. collocation a first pre-training module for obtaining a granular language representation sub-model; The above word Using the segmentation results, we use the language representation model word is the division unit word Pre-training is performed on the granular language representation sub-model. word and a second pre-training module for obtaining a granular linguistic representation sub-model.

[0008] In a fifth aspect, an embodiment of the present disclosure further provides a natural language processing apparatus, the apparatus comprising: a determination module for determining a fine-tuning sample corpus based on a natural language processing task; For the fine-tuned sample language corpus, collocation Units and word Divide by unit and fine-tune collocation Segmentation results and fine-tuning word a segmentation module for obtaining a segmentation result; Fine-tuning collocation Segmentation results and fine-tuning word a fine-tuning module for fine-tuning the pre-trained language representation model using the segmentation result; and a processing module for processing the target natural language using the fine-tuned language representation model.

[0009] In a sixth aspect, embodiments of the present disclosure further provide an apparatus, the apparatus comprising: one or more processors; a storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the language representation model pre-training method described in any embodiment of the present disclosure.

[0010] In a seventh aspect, an embodiment of the present disclosure further provides a storage medium having computer-executable instructions stored thereon, the computer-executable instructions, when executed by a processor of a computer, being used to perform a method for pre-training a language representation model as described in any embodiment of the present disclosure.

[0011] In an eighth aspect, an embodiment of the present disclosure further provides a storage medium having computer-executable instructions stored thereon, the computer-executable instructions, when executed by a processor of a computer, being used to perform a natural language processing method as described in any embodiment of the present disclosure.

[0012] In a ninth aspect, the embodiments of the present disclosure further provide a computer program product including a computer program carried on a non-transitory computer-readable medium, the computer program including a program code used to execute the method for pre-training a language representation model described in any one of the embodiments of the present disclosure.

[0013] In a tenth aspect, the embodiments of the present disclosure further provide a computer program product including a computer program carried on a non-transitory computer readable medium, the computer program including program code used to execute the natural language processing method described in any one of the embodiments of the present disclosure.

[0014] In an eleventh aspect, the embodiments of the present disclosure further provide a computer program carried on a non-transitory computer-readable medium, the computer program including a program code used to execute the method for pre-training a language representation model described in any one of the embodiments of the present disclosure.

[0015] In a twelfth aspect, the embodiments of the present disclosure further provide a computer program carried on a non-transitory computer readable medium, the computer program including a program code used to execute a natural language processing method as described in any one of the embodiments of the present disclosure. Effect of the Invention

[0016] The language expression model system provided by the embodiment of the present disclosure comprises: word is the division unit word A granular language representation sub-model; collocation is the division unit collocation A granular language representation sub-model, word Granular language representation sub-models are word Based on the sentence with each division unit, wordis used to output a first semantic vector corresponding to a semantic expression expressed in the sentence, collocation Granular language representation sub-models are collocation Based on the above sentence with each division unit, collocation is used to output a second semantic vector corresponding to the semantic expression expressed in the sentence, and an embodiment of the present disclosure collocation Units for natural language collocation To provide granular representation, and word Units for natural language word To provide a hybrid granular language representation model capable of performing granular representation, which provides a model foundation for downstream natural language processing tasks, helps to improve the processing accuracy of downstream natural language processing tasks, and enhances the transfer effect of the language representation model. [Brief description of the drawings]

[0017] The above and other features, advantages and aspects of each embodiment of the present disclosure will become more apparent with reference to the following specific embodiments in conjunction with the accompanying drawings, in which the same or similar reference numerals refer to the same or similar elements throughout the drawings. It should be understood that the drawings are schematic and that the elements and components are not necessarily drawn to scale.

[0018] [Figure 1] FIG. 2 is a structural schematic diagram of a language expression model system provided by Example 1 of the present disclosure. [Diagram 2] FIG. 11 is a structural schematic diagram of another language expression model system provided by Example 2 of the present disclosure. [Diagram 3] 1 is a schematic flowchart of a method for pre-training a language representation model provided by Example 3 of the present disclosure; [Figure 4] 11 is a schematic flowchart of a method for pre-training another language representation model provided by Example 3 of the present disclosure. [Diagram 5] 1 is a schematic flowchart of a natural language processing method provided by Example 4 of the present disclosure. [Figure 6]11 is a schematic flowchart of another natural language processing method provided by Example 4 of the present disclosure. [Figure 7] FIG. 11 is a structural schematic diagram of a pre-training device for a language representation model provided by Example 5 of the present disclosure. [Figure 8] FIG. 11 is a structural schematic diagram of a natural language processing apparatus provided by Example 6 of the present disclosure. [Figure 9] FIG. 11 is a structural schematic diagram of an electronic device provided according to Example 7 of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Hereinafter, the embodiments of the present disclosure will be described in more detail with reference to the drawings. Although the drawings show some embodiments of the present disclosure, it should be understood that the present disclosure can be realized in various forms and should not be interpreted as being limited to the embodiments described herein, but on the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure serve merely as examples and are not intended to limit the scope of protection of the present disclosure.

[0020] The steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. Also, method embodiments may include additional steps and / or omit performing steps that are shown. The scope of the present disclosure is not limited in this respect.

[0021] As used herein, the term "comprising" and variations thereof are intended to mean an open inclusion, i.e., "including, but not limited to." The term "based on" means "based at least in part on." The term "in one embodiment" means "at least one embodiment," the term "in another embodiment" means "at least one other embodiment," and the term "in some embodiments" means "at least some embodiments." Other terms are defined as explained below.

[0022] In addition, concepts such as "first" and "second" in this disclosure are used to distinguish different devices, modules or units, and are not intended to limit the order of functions performed by these devices, modules or units or the mutual dependencies between them.

[0023] It should be noted that the modifiers "one" and "multiple" in this disclosure are illustrative rather than limiting, and as would be understood by one of ordinary skill in the art, unless otherwise specified, they should be understood as "one or more."

[0024] Currently, the language representation models that are commonly used are usually single-grained language representation models, e.g. word is the division unit word Granular language expression model or collocation is the division unit collocation Granular language expression models include:

[0025] In implementing this application, the inventors: word The above with the division unit word The granular language representation model does not take into account the problem of incorrect attention weighting of characters, and when a character is unrelated to another character in a specific context, but the two characters are relatively closely related in the training sample language corpus, an unreasonable attention weighting is given to the character, thus causing the meaning of the character to be incorrectly represented in that specific context. collocation The above with the division unit collocation Granular language representation models can lead to incorrect semantic analysis due to incorrect division.

[0026] Example 1

[0027] FIG 1 is a structural schematic diagram of a language expression model system provided by the first embodiment of the present disclosure. As shown in FIG 1, the system includes: word is the division unit word A granular language representation sub-model 110; collocation is the division unit collocation and a granular language representation sub-model 120.

[0028] The above word The granular language representation sub-model 110 is word Based on the sentence with each division unit, word is used to output a first semantic vector corresponding to a semantic expression expressed in the sentence; The above collocation The granular language expression sub-model 120 is collocation Based on the above sentence with each division unit, collocation is used to output a second semantic vector corresponding to the semantic expression expressed in the sentence.

[0029] " [Outside 1] For example, take the Chinese sentence "TIFF0007662291000001.tif9127 (Japanese translation: All the ping pong bats in the store are sold out)": word When the sentence is divided into two parts, word The result of the division is that the ping pong bats in the store have been sold out, collocation When the sentence is divided into two parts, collocation The result of the division is that the ping-pong auction in the store has ended. In the context of the above sentence, collocation If we use the above as a division unit, we will misunderstand the meaning of the sentence. This is because we mistakenly associate the word "bat" in "ping pong bat" with the word "sell" that follows it, and divide it accordingly. Therefore, collocation When the granular language representation sub-model 120 is used alone, the meaning of a sentence similar to the above sentence cannot be accurately understood, and an incorrect sentence is generated. collocation Mis-analysis due to division is common.

[0030] For example, take the English sentence "we want to hang a portrait in the drawing room." word When the sentence is divided into two parts, wordThe division result is we / want / to / hang / a / portrait / in drawing / room(Japanese translation: We want to hang a portrait in the room) collocation When the sentence is divided into two parts, collocation The division result is we / want to / hang / a / portrait / in drawing room(Japanese translation: We want to hang a portrait in the drawing room). In the context of the above sentence, word If the sentence is divided into two parts, the meaning of the sentence may be misunderstood. This is due to the misunderstanding of "drawing". Since "portrait" comes before "drawing", the system is likely to understand the meaning of "drawing" as being associated with "portrait", i.e., as drawing a picture. However, in the above sentence, "drawing" should be understood as "drawing room" associated with "room". Therefore, word If the granular language representation sub-model 110 is used alone, it will not be possible to accurately understand the meaning of a sentence similar to the above sentence, "we want to hang a portrait in drawing room."

[0031] In response to the above problem, the language expression model provided by this embodiment is as follows: word is the division unit word As well as granular language representation sub-models, collocation is the division unit collocation Since it also includes a granular language representation submodel, for the same sentence, each division word If we can express a first semantic vector corresponding to the sense expressed by collocationIt can also represent a second semantic vector that corresponds to the meaning expressed in the sentence, providing a semantic foundation for downstream natural language processing tasks, and in specific language processing tasks word By particle size characteristics collocation Regulating grain size characteristics, collocation By particle size characteristics word It enriches the representation of granular features, helps improve the processing accuracy of downstream natural language processing tasks, and enhances the transfer effect of language representation models.

[0032] Example 2

[0033] FIG. 2 is a structural schematic diagram of another language expression model system provided by the second embodiment of the present disclosure. Based on the above embodiments, the embodiment of the present disclosure includes: collocation A granular language representation sub-model; word The granular language representation sub-model and the structure of are further described. collocation Granular language expression submodel and word The structure of the granular language expression submodel is shown.

[0034] As shown in FIG. word The granular language representation sub-model 210 is word is the division unit word A granular input layer 211; word A granularity coding layer 212; word and a granularity output layer 213. word The granularity input layer 211 is word connected to the granularity coding layer 212; word Receive a sentence with each division unit word Each division is performed by referring to the position of the sentence in the paragraph in which the sentence is located. word Corresponding word Convert it to a vector, word Vector word It is used to send to the granularity coding layer 212. word The granularity coding layer 212 is word The granular output layer 213 is connected to the received word Split each part based on the vector worda first semantic vector corresponding to a meaning expressed in the sentence; and word It is used to output to the granularity output layer 213. word The granular output layer 213 outputs the first semantic vector received. word The number of granularity coding layers 212 is at least two, and each word The structure of the granularity coding layer is the Transformer structure.

[0035] The above collocation The granular language representation sub-model 220 is collocation is the division unit collocation A granular input layer 221; collocation A granularity coding layer 222; collocation a granularity output layer 223; The above collocation The granularity input layer 221 is collocation connected to the granularity coding layer 222; collocation Receive a sentence with each division unit collocation Each division is performed by referring to the position of the sentence in the paragraph in which the sentence is located. collocation Corresponding collocation To convert each division into a vector, collocation Corresponding to collocation The vector is divided collocation The position vector, the paragraph vector, and the corresponding segmentation collocation and an embedding vector of collocation Vector collocation The granularity coding layer 222 is used to transmit the granularity coding layer 222. collocation The granularity coding layer 222 is collocation The granular output layer 223 is connected to the received collocation Split each part based on the vector collocation a second semantic vector corresponding to the meaning expressed in the sentence, and the second semantic vector is collocation It is used to output to the granularity output layer 223. collocation The granular output layer 223 is used to output the received second semantic vector. collocationThe number of granularity coding layers 222 is at least two, and each collocation The structure of the granular coding layer 222 is a Transformer structure. The Transformer structure includes an encoding assembly, a decoding assembly, and a connection relationship between them. The encoding assembly is usually composed of at least two encoders, and the decoding assembly is composed of the same number of decoders as the encoders. The encoder includes two layers, a self-attention layer and a feedforward neural network layer. The self-attention layer is a layer that is generated by applying the current collocation It helps to obtain the meaning of the context, rather than just focusing on the current node. The decoder also includes a self-attention layer and a feedforward neural network layer, but between these two layers there is an attention layer that helps to obtain the important content that should be currently focused on at the current node. The Transformer structure has been widely applied in the field of natural language processing (NLP), such as machine translation, question-and-answer systems, text summarization, and speech recognition, and has shown excellent performance.

[0036] In FIG. 2, "cls" indicates the beginning of a sentence, and "sep" indicates the end of a sentence.

[0037] According to the solution according to the embodiment of the present disclosure, word Granular language representation sub-models are word is the division unit word A granular input layer; word A granularity coding layer; word a granularity output layer; collocation Granular language representation sub-models are collocation is the division unit collocation A granular input layer; collocation A granularity coding layer; collocation A granular output layer and a collocation Each division of granularity collocation Semantic expression of word Each granularity wordThe present invention achieves the purpose of semantic representation of the language representation model, provides an understanding basis for downstream natural language processing tasks, helps improve the processing accuracy of downstream natural language processing tasks, and enhances the transfer effect of the language representation model.

[0038] Example 3

[0039] FIG. 3 is a method for pre-training a language expression model provided by the third embodiment of the present disclosure, in which pre-training is performed on the language expression model described in the above embodiments, and the language expression model is collocation Granularity and word The method is used to provide a granular semantic representation of words and provide an understanding basis for downstream natural language processing tasks. As shown in Figure 3, the method includes the following steps:

[0040] In step 310, a pre-training corpus sample is determined.

[0041] The corpus samples can be taken from commonly used websites, or the ambiguous sentences can be collected and organized manually. [Outside 2] Examples include the Chinese sentence "TIFF0007662291000002.tif9127 (Japanese translation: The store is out of ping-pong bats)" or the English sentence "we want to hang a portrait in drawing room."

[0042] In step 320, for each of the corpus samples, collocation Units and word Divide by unit and collocation The division result and word The division result is obtained.

[0043] " [Outside 3] For example, take the Chinese sentence "TIFF0007662291000003.tif9127 (Chinese) (Japanese translation: All the ping pong bats in the store are sold out)": word When the sentence is divided into two parts, word The result of the division will be that the ping pong bats in the store have been sold out. collocation When the sentence is divided into two parts, collocation The result of the division is that the ping-pong auction in the store has ended.

[0044] For example, take the English sentence "we want to hang a portrait in the drawing room." word When the sentence is divided into two parts, word The division result is we / want / to / hang / a / portrait / in drawing / room(Japanese translation: We want to hang a painted portrait in the / room). collocation When the sentence is divided into two parts, collocation The division result is we / want to / hang / a / portrait / in drawing room(Japanese translation: We would like to hang a portrait in our drawing room).

[0045] In step 330, collocation Using the segmentation results, we use the language representation model collocation is the division unit collocation Pre-training is performed on the granular language representation sub-model. collocation Obtain the granular language representation submodel.

[0046] For example, refer to the schematic flow chart of another language expression model pre-training method shown in FIG. collocation Using the segmentation results, we use the language representation model collocation is the division unit collocation The step of pre-training the granular language representation sub-model includes: collocation In units of collocation Splitting with a given percentage in the split result collocation A step of masking (MASK) collocation Splits that are masked in the split result collocation and the remaining unmasked split collocation The above collocation Granular language representation sub-model collocation Input to the granular input layer, collocation By the granular input layer, each of the above divisions collocation Each of the segments is then processed by referring to the location of the corpus sample in the corpus and the paragraph in which the corpus is located. collocation Corresponding collocation Convert it to a vector, collocation The vector of the collocation granular language representation sub-model is collocation sending to a granularity coding layer; The above collocation The granularity coding layer collocation determining a second semantic vector corresponding to the meaning expressed in the corpus; and collocation Granular language representation sub-model collocation outputting the granular output layer; The above collocation By the granularity output layer, each of the above divisions collocation Based on the second semantic vector of collocation outputting the The above collocation The masked division output from the granular output layer collocation and terminating the pre-training when the accuracy of reaches a second threshold.

[0047] The predetermined percentage is usually 15%. collocation15% split from split result collocation Randomly determine these divisions collocation Mask the partition to be masked using embedding vectors that do not represent any set meaning. collocation represents each division collocation For the current split, whether it is something that should be masked or not, collocation A position identifier that indicates the position in the current sentence of collocation A paragraph identifier indicating the paragraph in which the sentence is located is added.

[0048] The above collocation The granular output layer is specifically, the division to be masked collocation Masked division based on the position identifier and paragraph identifier of collocation Split before and after collocation Masked division based on the meaning of the word collocation Specifically, we predict the partition to be masked. collocation is a specific division collocation Calculate the probability that collocation As shown in Figure 4, collocation The granular output layer finally outputs the masked division collocation Output "shop", word The granular output layer is finally masked. word Output "commerce" and "store".

[0049] In step 340, word Using the segmentation results, we use the language representation model word is the division unit word Pre-training is performed on the granular language representation sub-model. word Obtain the granular language representation submodel.

[0050] word The process of pre-training the granular language representation sub-model is as follows: collocation Similar to the process of pre-training the granular language representation sub-model, for example, The above word Using the segmentation results, we use the language representation model word is the division unit word The step of pre-training the granular language representation sub-model includes: word In units of word Splitting with a given percentage in the split result word masking the word Masked splits in split results word and the remaining unmasked split word The above word Granular language representation sub-model word Input to the granular input layer, word By the granular input layer, each of the above divisions word Each of the segments is then processed by referring to the location of the segment in the corpus and the paragraph in which the segment is located. word Corresponding word Convert it to a vector, word Vector word Granular language representation sub-model word sending to a granularity coding layer; The above word The granularity coding layer word determining a first semantic vector corresponding to a meaning expressed in the corpus; and computing the first semantic vector as word Granular language representation sub-model word outputting the granular output layer; The above word By the granularity output layer, each of the above divisions word Based on the first semantic vector of word outputting the The above word The masked division output from the granular output layer word and terminating the pre-training when the accuracy of reaches a first threshold.

[0051] Furthermore, word The number of granularity coding layers is at least two, and each word The structure of the granularity coding layer is a Transformer structure, collocation The number of granularity coding layers is at least two, and each collocation The structure of the granular coding layer is a Transformer structure. The parameters of the language expression model are word Corresponding to granular language expression submodel word A vector parameter, collocation Corresponding to granular language expression submodel collocation A vector parameter, word Granular language expression submodel and collocation The granular language expression sub-model and the related parameters of the Transformer shared by the granular language expression sub-model, the position vector, and the paragraph vector. word Specifically, the vector parameters are word or an associated matrix for determining said embedding vector. collocation Specifically, the vector parameters are collocation The embedding vector of the vector may be expressed as a vector of the ... collocation or word The position vector and the paragraph vector represent the position in the sentence, and the paragraph vector represents the paragraph information in which the sentence to be divided is located. word Granular language expression submodel and collocation Since the granular language representation can be shared by sub-models, word Granular language expression submodel and collocation It is not necessary to store each of the granular language representation sub-models, but only one of them needs to be reserved, which can reduce the scale of the model parameters and reduce the complexity of the language representation model. The relevant parameters of the Transformer refer to the parameters learned during pre-training, such as the parameters of the fully connected layer.

[0052] As can be understood, there is no chronological relationship between step 330 and step 340, and the operation of step 330 may be performed first and the operation of step 340 may be performed later, or the operation of step 340 may be performed first and the operation of step 330 may be performed later, or the operation of step 330 and the operation of step 340 may be performed in parallel.

[0053] The pre-training method for a language representation model provided by the embodiment of the present disclosure divides the same sentence into division units of different granularity, and pre-trains language representation sub-models of different granularity based on the divisions of different granularity, respectively, to convert an input sentence into: collocation Granular division collocation and the semantic vector of word Granular division word We obtain a hybrid granular language representation model that can simultaneously represent the semantic vectors of the language, and provide a semantic understanding foundation for downstream natural language processing tasks. word By particle size characteristics collocation Regulating grain size characteristics, collocation By particle size characteristics word It enriches the representation of granular features, helps improve the processing accuracy of downstream natural language processing tasks, and enhances the transfer effect of language representation models.

[0054] Example 4

[0055] FIG. 5 is a schematic flowchart of a natural language processing method provided by Example 4 of the present disclosure. Specifically, the language representation model disclosed based on the above examples performs specific natural language processing tasks, and the natural language processing tasks include at least one of semantic similarity calculation, text classification, linguistic reasoning, keyword identification, and reading comprehension.

[0056] As shown in FIG. 5, the natural language processing method includes the following steps.

[0057] In step 510, a fine-tuning sample corpus is determined based on the natural language processing task.

[0058] The language representation model disclosed in the above embodiments is only used to provide different granularity semantic representation for natural language sentences and provide an understanding basis for all natural language processing tasks. When a specific natural language processing task needs to be performed, the language representation model needs to be further fine-tuned and trained according to the specific natural language processing task.

[0059] For example, if the natural language processing task is semantic similarity calculation, the fine-tuned sample language data will have the format of sentence 1:sentence 2:similarity label, where "0" indicates that sentence 1 and sentence 2 have different meanings, i.e., the two sentences are not similar in meaning, and "1" indicates that sentence 1 and sentence 2 have the same meaning, i.e., the two sentences are similar in meaning.

[0060] If the natural language processing task is text classification, the fine-tuning sample corpus is in the format of classification label:classification name:text to be classified.

[0061] In step 520, the fine-tuned sample corpus is collocation Units and word Divide by unit and fine-tune collocation Segmentation results and fine-tuning word The division result is obtained.

[0062] In step 530, the fine adjustment collocation Segmentation results and fine-tuning word The segmentation results are used to fine-tune a pre-trained language representation model.

[0063] That is, the fine adjustment collocation Segmentation results and fine-tuning word The segmentation results are used to perform secondary training on the pre-trained language representation model so that the language representation model learns specific task processing strategies.

[0064] In step 540, the fine-tuned language representation model is used to process the target natural language.

[0065] Further, referring to the schematic flowchart of another natural language processing method shown in FIG. 6, numeral 610 indicates a pre-training step of a hybrid granular language representation model, which corresponds to the content disclosed in the third embodiment of the present disclosure, and the hybrid granular language representation model is the language representation model described in any of the above embodiments; word is the division unit word A granular language representation sub-model; collocation is the division unit collocation The hybrid granular language representation model includes a granular language representation sub-model and a granular language representation sub-model. Number 620 indicates a step of performing secondary training on the pre-trained language representation model, i.e., a fine-tuning step, specifically, fine-tuning the pre-trained language representation model according to a specific natural language processing task. Number 630 indicates downstream application scenarios of the hybrid granular language representation model, specifically including semantic similarity calculation task AFQMC, short text classification task TNEWS, long text classification task IFLYTEK, linguistic reasoning task CMNLI, Chinese coreference analysis task CLUEWSC2020, paper keyword identification task CSL, simplified Chinese reading task CMRC2018, Chinese idiom reading blank fill-in task CHID, and Chinese multiple choice reading task C3.

[0066] The semantic similarity calculation task AFQMC specifically refers to a task such as {"sentence1":"Where can I apply for an increase in my credit card credit limit for Double Eleven?", "sentence2":"My credit card credit limit can be increased at...", "label":"0"}. There are three attributes for each data, which are sentence 1, sentence 2, and sentence similarity label, from the front to the back. As for the label, the sentence similarity label 1 indicates that the meanings of sentence 1 and sentence 2 are similar, and the sentence similarity label 0 indicates that the meanings of sentence 1 and sentence 2 are different. During the application of a specific task, the pre-trained language representation model described in the above embodiment is first fine-tuned using the language materials related to the specific task that have been collected in advance, that is, the pre-trained language representation model is re-trained to be suitable for the specific task application. Taking the specific task as an example of a semantic similarity calculation task, first, semantically similar and dissimilar language materials are collected, and the expression format of the language materials is {"sentence1": "Where can I apply for an increase in my credit card limit for Double Eleven?", "sentence2": "My credit card limit will be increased at...", "label": "0"}, and each data has three attributes, which are sentence 1, sentence 2, and sentence similarity label (i.e., as a label, 1 indicates that sentence 1 and sentence 2 are similar in meaning, and 0 indicates that sentence 1 and sentence 2 are different in meaning). Then, using the collected language materials, the pre-trained language representation model is re-trained, and during the re-training, for sentences of the same language materials, the trained language representation model is used to re-train each segmentation. word A first semantic vector corresponding to the sense expressed in the sentence, and each partition collocation and a second semantic vector corresponding to the meaning expressed in the sentence, word By particle size characteristics collocation Regulating grain size characteristics, collocation By particle size characteristics word By enriching the representation of granular features, the probability of misunderstanding the meaning of a sentence and the number of incorrect collocation It can reduce the probability of erroneous analysis due to division, improve the learning effect of specific natural language processing tasks, increase the task processing accuracy, and the pre-trained language representation model has high model transfer characteristics. For example, the short text classification task TNEWS is a selfie with a donut, which is beautiful even at a mysterious angle. I am obsessed with beautiful things. Each data has three attributes, which are the classification ID, classification name, and news string (only the title is included).

[0067] Long text classification task IFLYTEK, specifically, for example, {“label”:“110”, “label_des”:“Community Supermarket”, “sentence”:“Happy Delivery Supermarket was founded in 2016 and is dedicated to building a one-stop shopping platform that can deliver in 30 minutes with just one mobile device. Product categories include fruits, vegetables, meat, eggs, dairy products, seafood, grains, edible oils, seasonings, wine, water, beverages, casual food, daily necessities, takeout, and more. Happy Company is committed to becoming a faster, better, more, and more cost-saving online retail platform with a new business model, a more efficient and faster warehouse delivery model, to bring consumers a better consumption experience while promoting food safety processes and becoming a respected Internet company in society. Happy, comfortable and fast, 1. The delivery time is presented more clearly and friendly 2. Several optimizations to ensure user privacy 3. Other adjustments to improve usability 4. Fixed some known bugs”}. Each piece of data has three attributes: type ID, type name, and text content.

[0068] Specifically, the language reasoning task CMNLI is, for example, {“sentence1”:“The new rights are already good enough”, “sentence2”:“Everyone likes the latest welfare”, “label”:“neutral”}. Each data has three attributes, which are sentence 1, sentence 2, and inclusion label from the front. There are three labels: neutral, entailment, and contradiction.

[0069] Specifically, the Chinese coreference analysis task CLUEWSC2020 determines what specific target the pronoun coreferences. For example, {“target”:{“span2_index”:37, “span1_index”:5, “span1_text”:“bed”, “span2_text”:“it”}, “idx”:261, “label”:“false”, “text”:“At this time, the mobile phone next to the pillow on the bed rang, and I thought it was strange, but it had been off for two months because of overdue bills, and then it suddenly rang.”} “true” indicates that the pronoun indeed coreferences the noun in span1_text, and “false” indicates that the pronoun does not coreference the noun in span1_text.

[0070] The paper keyword identification task of CSL was specifically conducted by extracting keywords from the abstracts of Chinese papers, and the papers were partly selected from core academic journals in the fields of social science and natural science in China. For example, {“id”:1,”abst”:“To solve the problem of reduced resolution in 3D sonar imaging caused by the traditional uniform FFT beamforming algorithm, this paper proposes a regional FFT beamforming algorithm. Under far-field conditions, the constraint is to ensure the imaging resolution, and the goal is to minimize the number of divisions. A genetic algorithm is used as an optimization method to divide the imaging area into multiple areas. A beam direction is selected in each area, and the demodulated output when each receiving array element receives the echo in that direction is obtained, and traditional uniform FFT beamforming is performed in the area as the original data. The FFT calculation process is optimized, the calculation amount of the new algorithm is reduced, and the real-time requirements of 3D imaging sonar are met. Simulation and experimental results show that the imaging resolution of the regional FFT beamforming algorithm is significantly improved compared to the traditional uniform FFT beamforming algorithm, and the real-time requirements are met.”, “keyword”:[“underwater acoustics”, “FFT”, “beamforming”, “3D imaging sonar”], “label”:“1”}. Each data set has four attributes: data ID, paper abstract, keywords, and true / false label.

[0071] The Simplified Chinese reading comprehension task CMRC2018 specifically refers to being given a sentence and a question, and needing to find the answer in the sentence.

[0072] The Chinese idiom reading comprehension fill-in-the-blank task CHID specifically involves giving a sentence, extracting some idioms, giving candidate idioms, and having the language representation model select the appropriate idiom to fill in the blanks.

[0073] The Chinese multiple-choice reading comprehension task C3 specifically refers to completing multiple-choice questions based on dialogues and incorporating the content of the dialogues.

[0074] In testing and evaluating the above nine tasks, the task processing effect of the language representation model provided by the embodiment of the present disclosure is significantly higher than the task processing effect of a single-granularity language representation model trained using the same language material, and the task processing effect of a language representation model of the same or larger scale that has been published. This is because the language representation model provided by the embodiment of the present disclosure is a hybrid granularity language representation model, word is the division unit word As well as granular language representation sub-models, collocation is the division unit collocation This is because the granular language representation sub-model is also included. For the same natural language sentence, each segmentation is performed using the language representation model provided by the embodiment of the present disclosure. word If we can express a first semantic vector corresponding to the sense expressed by collocation It is also possible to express a second semantic vector corresponding to the sense that word By particle size characteristics collocation Regulating grain size characteristics, collocation By particle size characteristics word By enriching the representation of granular features, the probability of misunderstanding the meaning of a sentence and the number of incorrect collocation This reduces the probability of erroneous analysis due to segmentation, improving the processing accuracy of natural language processing tasks.

[0075] Acc. stands for accuracy, EM stands for maximum matching accuracy, and a comparison table of task processing performance of each language representation model is shown in Table 1. Models BERT, RoBERTa, ALBERT-xlarge, XLNet-Mid and ERNIE are test evaluation results that have already been published. After the usage data of some processing tasks (e.g., short text classification task TNEWS, Chinese coreference analysis task CLUEWSC2020 and paper keyword identification task CSL) has been increased, they are re-tested and evaluated using the language representation models provided by the embodiments of the present disclosure. BERT reproduction refers to the task processing results obtained by the applicant using BERT to evaluate the above processing tasks, and the hybrid granular language representation model is provided by the embodiments of the present disclosure. word is the division unit word Not only granular language representation submodels collocation is the division unit collocation A language representation model that includes a granular language representation submodel. Hybrid granular language representation refers to the idea of ​​hybrid granular language representation, i.e., the idea of ​​hybrid granular language representation for the same natural sentence. word Semantic vectors for each collocation and the semantic vector for each word, respectively; the hybrid granularity language representation model can be obtained by extending any single-granularity language representation model, and the single-granularity language representation model may be, for example, BERT, RoBERTa, or ALBERT. From Table 1, the performance and average score shown by the hybrid granularity language representation model in most task processing applications are far superior to other conventional single-granularity language representation models, and when the pre-training data is the same, the processing performance of the hybrid granularity language representation model also far superior to the processing performance of the single-granularity language representation model.

[0076] [Table 1]

[0077] The natural language processing method provided by the embodiments of the present disclosure includes: fine-tuning and training the hybrid granular language representation model described in the above embodiments according to a specific processing task; and performing natural language task processing based on the fine-tuned and trained hybrid granular language representation model; word By particle size characteristics collocation Regulating grain size characteristics, collocation By particle size characteristics word Enhancing the representation of granular features and improving the processing accuracy of natural language processing tasks.

[0078] Example 5

[0079] FIG. 7 shows a pre-training device for a language expression model provided by the fifth embodiment of the present disclosure, specifically including: a determining module 710, a dividing module 720, a first pre-training module 730, and a second pre-training module 740; The determination module 710 is used to determine a corpus sample for pre-training, and the segmentation module 720 performs segmentation on the corpus sample, respectively. collocation Units and word Divide by unit and collocation The division result and word The first pre-training module 730 is used to obtain the segmentation result. collocation Using the segmentation results, we use the language representation model collocation is the division unit collocation Pre-training is performed on the granular language representation sub-model. collocation The second pre-training module 740 is used to obtain a granular language representation sub-model. word Using the segmentation results, we use the language representation model word is the division unit word Pre-training is performed on the granular language representation sub-model. word Granular language expressions are used to obtain submodels.

[0080] Based on the above solution, the first pre-training module 730: collocation In units of collocation Splitting with a specified ratio in the split result collocation a mask unit for masking the collocation Masked splits in split results collocation and the remaining unmasked split collocation The above collocation Granular language representation sub-model collocation Input to the granular input layer, collocation By the granular input layer, each of the above divisions collocation Each of the segments is then processed by referring to the location of the corpus sample in the corpus and the paragraph in which the corpus is located. collocation Corresponding collocationConvert it to a vector, collocation The vector of the collocation granular language representation sub-model is collocation an input unit for sending to the granularity coding layer; The above collocation The granularity coding layer collocation determining a second semantic vector corresponding to the meaning expressed in the corpus; and collocation Granular language representation sub-model collocation a decision unit for outputting to a granular output layer; The above collocation By the granularity output layer, each of the above divisions collocation A segmentation masked based on the second semantic vector of collocation and outputting the collocation The masked division output from the granular output layer collocation and an output unit that is used to terminate pre-training when the accuracy of reaches a second threshold.

[0081] Based on the above solution, the second pre-training module 740: word In units of word Splitting with a given percentage in the split result word a mask unit for masking the word Masked splits in split results word and the remaining unmasked split word The above word Granular language representation sub-model word Input to the granular input layer, word By the granular input layer, each of the above divisions word Each of the segments is then processed by referring to the location of the corpus sample in the corpus and the paragraph in which the corpus is located. word Corresponding word Convert it to a vector, word Vector word Granular language representation sub-model word an input unit for sending to the granularity coding layer; The above word Each division by the granularity coding layer word determining a first semantic vector corresponding to a meaning expressed in the corpus; and computing the first semantic vector as word Granular language representation sub-model word a decision unit for outputting to a granular output layer; The above word Each of the above divisions is done by the granular output layer. word A segmentation masked based on the first semantic vector of word and outputting the word The masked division output from the granular output layer word and an output unit that is used to terminate pre-training when the accuracy of reaches a first threshold.

[0082] According to the above solution, the parameters of the language expression model are word Corresponding to granular language expression submodel word A vector parameter, collocation Corresponding to granular language expression submodel collocation A vector parameter, word Granular language expression submodel and collocation The granular language expression sub-model includes the relevant parameters of the Transformer shared with the sub-model, the position vector, and the paragraph vector.

[0083] The solution according to the embodiment of the present disclosure divides the same sentence into division units of different granularity, and pre-trains language expression sub-models of different granularity based on the division of different granularity, respectively, to convert the input sentence into: collocation Granular division collocation and the semantic vector of word Granular division word We obtain a hybrid granular language representation model that can simultaneously represent the semantic vectors of the language, and provide a semantic understanding foundation for downstream natural language processing tasks. word By particle size characteristics collocation Regulating grain size characteristics, collocation By particle size characteristics wordIt enriches the representation of granular features, helps improve the processing accuracy of downstream natural language processing tasks, and enhances the transfer effect of language representation models.

[0084] The language representation model pre-training device provided by the embodiments of the present disclosure can execute the language representation model pre-training method provided by any embodiment of the present disclosure, and has functional modules for executing the method and beneficial effects.

[0085] Although each unit and module included in the above device is classified based on functional logic, it is sufficient that the corresponding function can be realized. The classification is not limited to the above. The specific names of each functional unit are merely for distinguishing each other, and are not intended to limit the scope of protection of the embodiments of the present disclosure.

[0086] Example 6

[0087] FIG. 8 shows a natural language processing apparatus provided by the sixth embodiment of the present disclosure, specifically including: a determining module 810, a dividing module 820, a fine-tuning module 830, and a processing module 840; The determination module 810 is used to determine a fine-tuning sample corpus based on a natural language processing task, and the segmentation module 820 divides the fine-tuning sample corpus into collocation Units and word Divide by unit and fine-tune collocation Segmentation results and fine-tuning word The fine-tuning module 830 obtains the fine-tuning result. collocation Segmentation results and fine-tuning word The segmentation results are used to fine-tune a pre-trained language representation model, and the processing module 840 is used to process the target natural language using the fine-tuned language representation model.

[0088] Based on the above solution, the natural language processing task includes at least one of semantic similarity calculation, text classification, linguistic reasoning, keyword identification, and reading comprehension.

[0089] The solution according to the embodiment of the present disclosure includes fine-tuning and training the hybrid granular language representation model described in the above embodiment according to a specific processing task, and performing natural language task processing based on the fine-tuned and trained hybrid granular language representation model; word By particle size characteristics collocation Regulating grain size characteristics, collocation By particle size characteristics word Enhancing the representation of granular features and improving the processing accuracy of natural language processing tasks.

[0090] The natural language processing apparatus provided by the embodiments of the present disclosure can execute the natural language processing method provided by any embodiment of the present disclosure, and has functional modules for executing the method and beneficial effects.

[0091] Although each unit and module included in the above device is classified based on functional logic, it is sufficient that the corresponding function can be realized. The classification is not limited to the above. The specific names of each functional unit are merely for distinguishing each other, and are not intended to limit the scope of protection of the embodiments of the present disclosure.

[0092] Example 7

[0093] Referring to FIG. 9 below, FIG. 9 shows a structural schematic diagram of an electronic device (e.g., a terminal device or server in FIG. 9) 400 suitable for implementing the embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in FIG. 9 is merely an example and should not impose any limitations on the functions and scope of use of the embodiments of the present disclosure.

[0094] 9, the electronic device 400 may include a processing unit (such as a central processing unit or a graphic processor) 401, which may perform various appropriate operations and processes according to a program stored in a read only memory (ROM) 402 or loaded from a storage device 408 into a random access memory (RAM) 403. The RAM 403 also stores various programs and data necessary for the operation of the electronic device 400. The processing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0095] Typically, input devices 406 including a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc., output devices 407 including a liquid crystal display (LCD), speaker, vibrator, etc., storage devices 408 including magnetic tape, hard disk, etc., and communication devices 409 can be connected to the I / O interface 405. The communication devices 409 can enable the electronic device 400 to communicate and exchange data with other devices wirelessly or via wires. Although FIG. 9 shows the electronic device 400 with various devices, it should be understood that not all of the devices shown need be implemented or deployed. Alternatively, more or fewer devices can be implemented or deployed.

[0096] In particular, according to an embodiment of the present disclosure, the above processes described with reference to the flowcharts can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product including a computer program embodied in a non-transitory computer-readable medium, the computer program including program code for performing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above functions defined in the method according to the embodiment of the present disclosure are performed.

[0097] The terminal device provided by the embodiments of the present disclosure belongs to the same inventive concept as the language expression model pre-training method and natural language processing method provided by the above embodiments, and the technical details not detailed in this embodiment may refer to the above embodiments, and the embodiments of the present disclosure achieve the same beneficial effects as the above embodiments.

[0098] Example 8

[0099] An embodiment of the present disclosure provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, realizes the method for pre-training a language representation model and the method for natural language processing provided by the above embodiment.

[0100] It should be noted that the computer readable medium described in this disclosure may be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of computer readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a Random Access Memory (RAM), a Read-Only Memory (ROM), an Erasable Programmable Read-Only Memory (EPROM or Flash memory), an optical fiber, a portable Compact Disc Read-Only Memory (CD-ROM), an optical storage device, a magnetic memory component, or any suitable combination of the above. In this disclosure, a computer readable storage medium may be any tangible medium that contains or stores a program that can be used by or in combination with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with computer-readable program code embedded therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.The program code contained in the computer readable medium may be transmitted using any suitable medium, including but not limited to electrical wire, fiber optic cable, RF (Radio Frequency), etc., or any suitable combination of the above.

[0101] In some embodiments, the client terminals and servers may communicate using any conventional or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network (LAN), a wide area network (WAN), an internetwork (e.g., the Internet), an end-to-end network (e.g., an ad hoc end-to-end network), and any conventional or future developed network.

[0102] The computer-readable medium may be included in the electronic device, or may exist independently without being assembled in the electronic device.

[0103] The computer-readable medium includes one or more programs stored therein, and when the one or more programs are executed by the electronic device, the electronic device: determining a pre-training corpus sample; For the above language samples, collocation Units and word Divide by unit and collocation The division result and word obtaining a segmentation result; The above collocation Using the segmentation results, we use the language representation model collocation is the division unit collocation Pre-training is performed on the granular language representation sub-model. collocation obtaining a granular language representation sub-model; The above word Using the segmentation results, we use the language representation model word is the division unit word Pre-training is performed on the granular language representation sub-model. word obtaining a granular language representation sub-model; determining a fine-tuning sample corpus based on a natural language processing task; For the fine-tuned sample language corpus, collocation Units and word Divide by unit and fine-tune collocation Segmentation results and fine-tuning word obtaining a segmentation result; Fine-tuning collocation Segmentation results and fine-tuning word fine-tuning a pre-trained language representation model using the segmentation results; A step of processing the target natural language using the fine-tuned language representation model is performed.

[0104] Computer program code for carrying out the operations of the present disclosure can be written in one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages, or a combination thereof. The program code can run entirely on the user's computer, partly on the user's computer, as a standalone software package, partly on the user's computer, partly on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer via any kind of network, such as a local area network (LAN) or wide area network (WAN), or can be connected to an external computer (e.g., connecting via the Internet using an Internet Service Provider).

[0105] The flowcharts and block diagrams in the drawings illustrate architecture, functions, and operations that can be realized by the system, method, computer program, and computer program product according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, program segment, or part of code that includes one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions shown in the blocks may be executed in a different order than that shown in the figures. For example, two blocks shown as connected may actually be executed in parallel, or the blocks may be executed in the reverse order depending on the related functionality. It should be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified functions or operations, or may be implemented using a combination of dedicated hardware and computer instructions.

[0106] The units described in the embodiments of the present disclosure can be realized in software or hardware. The names of these units may not be intended to limit the units themselves under certain circumstances, for example, an editable content display unit may be further described as an "edit unit". The embodiments of the present disclosure further include a computer program, which, when executed in an electronic device or executed by a processor, performs the above-mentioned functions defined in the manner according to the embodiments of the present disclosure.

[0107] The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that may be used may include, but are not limited to, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), Systems on Chips (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.

[0108] In the context of this disclosure, a machine-readable medium may be a tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of machine-readable storage media may include electrical connections based on one or more wires, portable computer disks, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic memory components, or any suitable combination of the above.

[0109] According to one or more embodiments of the present disclosure, in [Example 1], a language expression model system is provided, the system comprising: word is the division unit word A granular language representation sub-model; collocation is the division unit collocation A granular language representation sub-model; The above word Granular language representation sub-models are word Based on the sentence with each division unit, word is used to output a first semantic vector corresponding to a semantic expression expressed in the sentence; The above collocation Granular language representation sub-models are collocation Based on the above sentence with each division unit, collocation is used to output a second semantic vector corresponding to the semantic expression expressed in the sentence.

[0110] According to one or more embodiments of the present disclosure, based on [Example 1], [Example 2] provides a language expression model system, optionally comprising: word Granular language representation sub-models are word is the division unit word A granular input layer; word A granularity coding layer; word a granular output layer; The above word The granular input layer is word connected to the granularity coding layer, word Receive a sentence with each division unit word Each division is performed by referring to the position of the sentence in the paragraph in which the sentence is located. word Corresponding word Convert it to a vector, word Vector word Used to send to the granularity coding layer, The above word The granularity coding layer is word Connected to the granular output layer, the received word Split each part based on the vector word a first semantic vector corresponding to a meaning expressed in the sentence; and word It is used to output to the granular output layer, The above word The granular output layer is used to output the received first semantic vector.

[0111] According to one or more embodiments of the present disclosure, based on [Example 2], [Example 3] provides a language expression model system, and optionally word The number of granularity coding layers is at least two, and each word The structure of the granularity coding layer is the Transformer structure.

[0112] According to one or more embodiments of the present disclosure, based on any of the above examples, in [Example 4], a language expression model system is provided, optionally including: The above collocation Granular language representation sub-models are collocation is the division unit collocation A granular input layer; collocation A granularity coding layer; collocation a granular output layer; The above collocation The granular input layer is collocation connected to the granularity coding layer, collocation Receive a sentence with each division unit collocation Each division is performed by referring to the position of the sentence in the paragraph in which the sentence is located. collocation Corresponding collocation Convert it to a vector, collocation Vector collocation Used to send to the granularity coding layer, The above collocation The granularity coding layer is collocation Connected to the granular output layer, the received collocation Split each part based on the vector collocation a second semantic vector corresponding to the meaning expressed in the sentence, and the second semantic vector is collocation It is used to output to the granular output layer, The above collocation The granular output layer is used to output the received second semantic vector.

[0113] According to one or more embodiments of the present disclosure, based on [Example 4], [Example 5] provides a language expression model system, and optionally collocation The number of granularity coding layers is at least two, and each collocation The structure of the granularity coding layer is the Transformer structure.

[0114] According to one or more embodiments of the present disclosure, in [Example 6], a method for pre-training a language representation model is provided, the method including: determining a pre-training corpus sample; For the above language samples, collocation Units and word Divide by unit and collocation The division result and word obtaining a segmentation result; The above collocation Using the segmentation results, we use the language representation model collocation is the division unit collocation Pre-training is performed on the granular language representation sub-model. collocation obtaining a granular language representation sub-model; The above word Using the segmentation results, we use the language representation model word is the division unit word Pre-training is performed on the granular language representation sub-model. word Obtaining a granular linguistic representation sub-model.

[0115] According to one or more embodiments of the present disclosure, based on [Example 6], [Example 7] provides a method for pre-training a language expression model, optionally including: The above collocation Using the segmentation results, we use the language representation model collocation is the division unit collocation The step of pre-training the granular language representation sub-model includes: collocation In units of collocation Splitting with a specified ratio in the split result collocation masking the collocation Masked splits in split results collocation and the remaining unmasked split collocation The above collocation Granular language representation sub-model collocation Input to the granular input layer, collocation By the granular input layer, each of the above divisions collocation Each of the segments is then processed by referring to the location of the corpus sample in the corpus and the paragraph in which the corpus is located. collocation Corresponding collocation Convert it to a vector, collocation Vector collocation Granular language representation sub-model collocation sending to a granularity coding layer; The above collocation The granularity coding layer collocation determining a second semantic vector corresponding to the meaning expressed in the corpus; and collocation Granular language representation sub-model collocation outputting the granular output layer; The above collocation By the granularity output layer, each of the above divisions collocation A segmentation masked based on the second semantic vector of collocation outputting the The above collocation The masked division output from the granular output layer collocation and terminating the pre-training when the accuracy of reaches a second threshold.

[0116] According to one or more embodiments of the present disclosure, based on [Example 7], [Example 8] provides a method for pre-training a language expression model, optionally including: The above word Using the segmentation results, we use the language representation model word is the division unit word The step of pre-training the granular language representation sub-model includes: word In units of word Splitting with a given percentage in the split result word masking the word Masked splits in split results word and the remaining unmasked split word The above word Granular language representation sub-model word Input to the granular input layer, word By the granular input layer, each of the above divisions word Each of the segments is then processed by referring to the location of the corpus sample in the corpus and the paragraph in which the corpus is located. word Corresponding word Convert it to a vector, word Vector word Granular language representation sub-model word sending to a granularity coding layer; The above word Each division by the granularity coding layer word determining a first semantic vector corresponding to a meaning expressed in the corpus; and computing the first semantic vector as word Granular language representation sub-model word outputting the granular output layer; The above word Each of the above divisions is done by the granular output layer. word A segmentation masked based on the first semantic vector of word outputting the The above word The masked division output from the granular output layer word and terminating the pre-training when the accuracy of reaches a first threshold.

[0117] According to one or more embodiments of the present disclosure, based on [Example 8], [Example 9] provides a method for pre-training a language expression model, optionally including: The above word The number of granularity coding layers is at least two, and each word The structure of the granular coding layer is a Transformer structure, The above collocation The number of granularity coding layers is at least two, and each collocation The structure of the granularity coding layer is the Transformer structure.

[0118] According to one or more embodiments of the present disclosure, based on [Example 9], [Example 10] provides a method for pre-training a language expression model, optionally including: The parameters of the language expression model are word Corresponding to granular language expression submodels word A vector parameter, collocation Corresponding to granular language expression submodels collocation A vector parameter, word Granular language expression submodel and collocation The granular language expression sub-model includes the relevant parameters of the Transformer shared with the sub-model, the position vector, and the paragraph vector.

[0119] According to one or more embodiments of the present disclosure, in [Example 11], a method for natural language processing is provided, the method comprising: determining a fine-tuning sample corpus based on a natural language processing task; For the fine-tuned sample language corpus, collocation Units and word Divide by unit and fine-tune collocation Segmentation results and fine-tuning word obtaining a segmentation result; Fine-tuning collocation Segmentation results and fine-tuning word fine-tuning a pre-trained language representation model using the segmentation results; and processing the target natural language using the fine-tuned language representation model.

[0120] According to one or more embodiments of the present disclosure, based on [Example 11], [Example 12] provides a natural language processing method, optionally wherein the natural language processing task includes at least one of semantic similarity calculation, text classification, linguistic reasoning, keyword identification, and reading comprehension.

[0121] According to one or more embodiments of the present disclosure, in [Example 13], a pre-training device for a language representation model is provided, the device comprising: a determination module for determining pre-training corpus samples; For the above language samples, collocation Units and word Divide by unit, collocation The division result and worda segmentation module for obtaining a segmentation result; The above collocation Using the segmentation results, we use the language representation model collocation is the division unit collocation Pre-training is performed on the granular language representation sub-model. collocation a first pre-training module for obtaining a granular language representation sub-model; The above word Using the segmentation results, we use the language representation model word is the division unit word Pre-training is performed on the granular language representation sub-model. word and a second pre-training module for obtaining a granular linguistic representation sub-model.

[0122] According to one or more embodiments of the present disclosure, in [Example 14], a natural language processing apparatus is provided, the apparatus including: a determination module for determining a fine-tuning sample corpus based on a natural language processing task; For the fine-tuned sample language corpus, collocation Units and word Divide by unit and fine-tune collocation Segmentation results and fine-tuning word a segmentation module for obtaining a segmentation result; Fine-tuning collocation Segmentation results and fine-tuning word a fine-tuning module for fine-tuning the pre-trained language representation model using the segmentation result; and a processing module for processing the target natural language using the fine-tuned language representation model.

[0123] According to one or more embodiments of the present disclosure, in [Example 15], one or more processors; a storage device for storing one or more programs, The one or more programs, when executed by the one or more processors, cause the one or more processors to: determining a pre-training corpus sample; For the above language samples, collocation Units and word Divide by unit and collocation The division result and word obtaining a segmentation result; The above collocation Using the segmentation results, we use the language representation model collocation is the division unit collocation Pre-training is performed on the granular language representation sub-model. collocation obtaining a granular language representation sub-model; The above word Using the segmentation results, we use the language representation model word is the division unit word Pre-training the granular language representation sub-model word A method for pre-training a language representation model, comprising the steps of: obtaining a granular language representation sub-model; determining a fine-tuning sample corpus based on a natural language processing task; For the fine-tuned sample language corpus, collocation Units and word Divide by unit and fine-tune collocation Segmentation results and fine-tuning word obtaining a segmentation result; Fine-tuning collocation Segmentation results and fine-tuning word fine-tuning a pre-trained language representation model using the segmentation results; and processing a target natural language using the fine-tuned language representation model.

[0124] According to one or more embodiments of the present disclosure, in [Example 16], a storage medium is provided that includes computer-executable instructions, the computer-executable instructions, when executed by a processor of a computer, determining a pre-training corpus sample; For the above language samples, collocationUnits and word Divide by unit and collocation The division result and word obtaining a segmentation result; The above collocation Using the segmentation results, we use the language representation model collocation is the division unit collocation Pre-training the granular language representation sub-model collocation obtaining a granular language representation sub-model; The above word Using the segmentation results, we use the language representation model word is the division unit word Pre-training the granular language representation sub-model word A method for pre-training a language representation model, comprising the steps of: obtaining a granular language representation sub-model; determining a fine-tuning sample corpus based on a natural language processing task; For the fine-tuned sample language corpus, collocation Units and word Divide by unit and fine-tune collocation Segmentation results and fine-tuning word obtaining a segmentation result; Fine-tuning collocation Segmentation results and fine-tuning word fine-tuning a pre-trained language representation model using the segmentation results; and processing a target natural language using the fine-tuned language representation model.

[0125] The above description is merely a description of some preferred embodiments of the present disclosure and a description of the applied technical principles.Those skilled in the art should understand that the disclosure scope of the present disclosure is not limited to the solution formed by a specific combination of the above technical features, but should also cover other solutions formed by any combination of the above technical features or their equivalent features without departing from the concept of the above disclosure, for example, by replacing the above features with technical features having similar functions (but not limited to) disclosed in the present disclosure.

[0126] Additionally, although operations are described in a particular order, this should not be understood as requiring that these operations be performed in the particular order or sequence shown. Multitasking or parallel processing may be advantageous in certain circumstances. Similarly, although the above description includes some specific implementation details, these should not be construed as limiting the scope of the disclosure. Certain features that are described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments, either alone or in any suitable subcombination.

[0127] Although the present subject matter has been described using language specific to structural features and / or methodological logical operations, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or operations described above. Rather, the specific features and operations described above are merely example forms for implementing the claims.

[0128] This application claims priority to a Chinese patent application filed on July 29, 2020, bearing application number 202010746066.0 and entitled "Language Representation Model System, Pre-Training Method, Apparatus, Device and Medium," the entire contents of which are incorporated herein by reference.

Claims

1. 1. A method for natural language processing, comprising: determining a pre-training corpus sample; segmenting the linguistic material samples into collocations and words, respectively, to obtain collocation segmentation results and word segmentation results; A step of performing pre-training on a collocation granularity language expression sub-model in which collocations are used as a division unit in the language expression model by using the collocation division result, and obtaining a pre-trained collocation granularity language expression sub-model; A step of performing pre-training on a word-granularity language expression sub-model in which a word is a division unit of the language expression model using the word segmentation result, and obtaining a pre-trained word-granularity language expression sub-model; performing a natural language processing task according to the pre-trained language representation model; The step of performing pre-training on a word-granularity language expression sub-model in which a word is a division unit of the language expression model using the word division result, A step of masking segmented words having a predetermined ratio in the word segmentation result in units of words; inputting the masked segmented words and the remaining unmasked segmented words in the word segmentation result into a word-granularity input layer of the word-granularity language representation sub-model; converting each of the segmented words into a corresponding word vector by the word-granularity input layer with reference to the position of the segmented word in the corpus of the corpus sample and the paragraph in which the corpus is located, and sending the word vector to a word-granularity encoding layer of the word-granularity language representation sub-model; determining a first semantic vector corresponding to the meaning expressed by each segmented word in the linguistic material by the word granularity encoding layer, and outputting the first semantic vector to a word granularity output layer of the word granularity language representation sub-model; outputting segmented words masked based on the first semantic vector of each of the segmented words by the word granularity output layer; When the accuracy of the masked segmented words output from the word-granularity output layer reaches a first threshold, the pre-training of the word-granularity language representation sub-model is terminated; A step of pre-training a collocation granular language expression sub-model in which collocations are used as a division unit in the language expression model by using the collocation division result, A step of masking divided collocations having a predetermined ratio in the result of the collocation division using the collocation as a unit; inputting the masked divided collocations in the collocation segmentation result and the remaining unmasked divided collocations into a collocation granularity input layer of the collocation granularity language representation sub-model; converting each of the divided collocations into a corresponding collocation vector by using the collocation granularity input layer, with reference to the position of each of the divided collocations in the corpus of the corpus sample and the paragraph in which the corpus is located, and transmitting the collocation vector to a collocation granularity encoding layer of the collocation granularity language representation sub-model; determining a second semantic vector corresponding to the meaning expressed by each divided collocation in the linguistic material by the collocation granularity coding layer, and outputting the second semantic vector to a collocation granularity output layer of the collocation granularity language representation sub-model; outputting, by the collocation granularity output layer, a segmented collocation that is masked based on the second semantic vector of each of the segmented collocations; When the accuracy of the masked divided collocations output from the collocation granularity output layer reaches a second threshold, the pre-training of the collocation granularity language representation sub-model is terminated; The number of the word granularity coding layers is at least two, and the structure of each word granularity coding layer is a Transformer structure; The number of the collocation granularity coding layers is at least two, and each collocation granularity coding layer has a transformer structure; A natural language processing method characterized in that the parameters of the language expression model include word vector parameters corresponding to the word-granularity language expression sub-model, collocation vector parameters corresponding to the collocation granularity language expression sub-model, and related parameters of a Transformer, a position vector, and a paragraph vector shared by the word-granularity language expression sub-model and the collocation granularity language expression sub-model.

2. The step of performing a natural language processing task according to the pre-trained language representation model includes: determining a fine-tuning sample corpus based on the natural language processing task; Segmenting the fine-tuned sample linguistic material into collocations and words, respectively, to obtain fine-tuned collocation segmentation results and fine-tuned word segmentation results; fine-tuning a pre-trained language representation model using the fine-tuned collocation segmentation result and the fine-tuned word segmentation result; and processing the target natural language using the fine-tuned language representation model.

3. 3. The method of claim 2, wherein the natural language processing tasks include at least one of semantic similarity calculation, text classification, linguistic reasoning, Chinese coreference analysis, keyword identification, and reading comprehension.

4. A natural language processing device, a determination module for determining pre-training corpus samples; a segmentation module for segmenting the corpus sample into collocations and words, respectively, to obtain collocation segmentation results and word segmentation results; a first pre-training module for performing pre-training on a collocation granularity language expression sub-model in which collocations are used as a division unit of the language expression model, by using the collocation division result, to obtain a pre-trained collocation granularity language expression sub-model; a second pre-training module for performing pre-training on a word-granularity language expression sub-model in which a word is a division unit of the language expression model by using the word segmentation result, and obtaining a pre-trained word-granularity language expression sub-model; a natural language processing module for performing a natural language processing task according to a pre-trained language representation model; The first pre-training module includes: a first masking unit for masking divided collocations having a predetermined ratio in a result of the collocation division; a first input unit for inputting masked split collocations and remaining unmasked split collocations in the collocation segmentation result to a collocation granularity input layer of the collocation granularity language representation sub-model, converting each of the split collocations into a corresponding collocation vector by the collocation granularity input layer with reference to a position of each of the split collocations in the linguistic material of the linguistic material sample and a paragraph in which the linguistic material is located, and transmitting the collocation vector to a collocation granularity encoding layer of the collocation granularity language representation sub-model; a first determination unit for determining a second semantic vector corresponding to a meaning expressed by each divided collocation in the corpus by the collocation granularity coding layer, and outputting the second semantic vector to a collocation granularity output layer of the collocation granularity language representation sub-model; a first output unit for outputting, by the collocation granularity output layer, a segmented collocation that is masked based on the second semantic vector of each of the segmented collocations, and terminating pre-training of the collocation granularity language representation sub-model when the accuracy of the masked segmented collocation outputted from the collocation granularity output layer reaches a second threshold value; The second pre-training module includes: a second masking unit for masking divided words having a predetermined ratio in the word segmentation result in units of words; a second input unit for inputting masked segmented words and remaining unmasked segmented words in the word segmentation result to a word-granularity input layer of the word-granularity language representation sub-model, converting each of the segmented words into a corresponding word vector by the word-granularity input layer with reference to a position of each of the segmented words in the corpus of the corpus sample and a paragraph in which the corpus is located, and transmitting the word vector to a word-granularity encoding layer of the word-granularity language representation sub-model; a second determination unit for determining a first semantic vector corresponding to a meaning expressed in the linguistic material by each divided word by the word granularity encoding layer, and outputting the first semantic vector to a word granularity output layer of the word granularity language representation sub-model; a second output unit for outputting segmented words masked based on the first semantic vector of each of the segmented words by the word granularity output layer, and terminating pre-training of the word granularity language representation sub-model when the accuracy of the masked segmented words output from the word granularity output layer reaches a first threshold value; The number of the word granularity coding layers is at least two, and the structure of each word granularity coding layer is a Transformer structure; The number of the collocation granularity coding layers is at least two, and each collocation granularity coding layer has a transformer structure; A natural language processing device characterized in that the parameters of the language expression model include word vector parameters corresponding to the word-granularity language expression sub-model, collocation vector parameters corresponding to the collocation granularity language expression sub-model, and related parameters of a Transformer, a position vector, and a paragraph vector shared by the word-granularity language expression sub-model and the collocation granularity language expression sub-model.

5. The natural language processing module includes: a third determining unit for determining a fine-tuning sample corpus based on the natural language processing task; a segmentation unit for segmenting the fine-tuned sample linguistic material into collocations and words, respectively, to obtain fine-tuned collocation segmentation results and fine-tuned word segmentation results; a fine-tuning unit for fine-tuning a pre-trained language representation model using the fine-tuned collocation result and the fine-tuned word segmentation result; The natural language processing apparatus according to claim 4, further comprising: a processing unit for processing a target natural language using the fine-tuned language expression model.

6. An electronic device, comprising: one or more processors; a storage device for storing one or more programs; An electronic device characterized in that, when the one or more programs are executed by the one or more processors, the one or more processors realize the natural language processing method described in any one of claims 1 to 3.

7. A storage medium containing computer-executable instructions, the computer-executable instructions being used to execute the natural language processing method according to any one of claims 1 to 3 when executed by a processor of a computer.

8. A computer program, which, when executed by a processing device, causes the natural language processing method according to any one of claims 1 to 3 to be executed.

Images