Training method of language model, text matching method and related device

Abstract

The application discloses a language model training method, a text matching method and related devices. The application provides a scheme that can divide the granularity, integrate high-frequency words in the target field into the first training text, and use the SOP task and the MSP task as the first stage training task. The first training text is used to pre-train the language model. Thus, in the pre-training stage, the method can not only enable the language model to learn the knowledge in the general field, but also learn the knowledge in the target field. Moreover, the method can use the text matching task as the second stage training task, and use the second training text obtained from the target field to train the language model, so that the language model can learn the basic knowledge in the target field more widely. Thus, the generalization ability of the language model can be improved, thereby ensuring the training effect of the model.

Description

Technical Field

[0001] This application relates to the field of Natural Language Processing (NLP), and in particular to a method for training a language model, a text matching method, and related apparatus. Background Technology

[0002] In natural language processing, an initial model can be pre-trained using a large amount of unlabeled text to obtain a language model independent of specific downstream tasks. Then, based on specific downstream tasks (such as text entailment recognition, dialogue matching, or question-answering search), the language model can be trained using manually labeled text relevant to those tasks to obtain a target language model capable of performing those tasks.

[0003] However, the language models trained using the above methods are of poor quality. Summary of the Invention

[0004] This application provides a language model training method, a text matching method, and related apparatus, which can effectively improve the training effect of the language model. The technical solution is as follows.

[0005] On the one hand, a method for training a language model is provided, the method comprising:

[0006] Two sentence samples are obtained from a first document in a document library containing documents in the target domain;

[0007] Each sentence sample is divided into segments at a granular level. Each segment includes multiple sentence segments, at least one of which includes multiple characters and at least one word from a high-frequency word library composed of the multiple characters. The high-frequency word library contains high-frequency words from the target domain.

[0008] At least one of the sentence segments in the two sentence samples after granular division is masked to obtain the first training text;

[0009] The first training stage uses the Sentence Order Prediction (SOP) task and the Mask Segment Prediction (MSP) task as training tasks, and the first training text is used to pretrain the language model. The SOP task is used to predict the order of the two sentence samples in the first document, and the MSP task is used to predict at least one sentence segment that has been masked.

[0010] The language model is trained using a text matching task as the second training task. The second training text includes a first input text in the target domain and the identifier text of a second document in the target domain. The text matching task is used to predict the matching degree between the second document and the first input text.

[0011] On the other hand, a text matching method is provided, the method comprising:

[0012] Get the target input text;

[0013] The target input text is input into the text matching model to obtain the identifier text of the recommended document that matches the target input text, output by the text matching model;

[0014] The text matching model is obtained by training the language model using the language model training method described above.

[0015] In another aspect, a language model training device is provided, the device comprising:

[0016] The acquisition module is used to acquire two sentence samples from a first document in a document library containing documents in the target domain;

[0017] The segmentation module is used to perform granular segmentation on each of the sentence samples. The granular segmented sentence samples include multiple sentence segments, at least one of the sentence segments includes multiple characters, and at least one word belonging to a high-frequency word library composed of the multiple characters. The high-frequency word library contains high-frequency words of the target domain.

[0018] The masking module is used to mask at least one of the sentence segments in the two sentence samples after granular division to obtain the first training text;

[0019] The training module is used to train a language model using the first training text as the first training task and the SOP task as the first training task. The SOP task is used to predict the order of the two sentence samples in the first document, and the MSP task is used to predict at least one sentence segment that has been masked.

[0020] The training module is further configured to train the language model using a text matching task as the second-stage training task, and to use a second training text, wherein the second training text includes a first input text in the target domain and the identifier text of a second document in the target domain, and the text matching task is used to predict the matching degree between the second text and the first input text.

[0021] In another aspect, a text matching device is provided, the device comprising:

[0022] The acquisition module is used to acquire the target input text;

[0023] The matching module is used to input the target input text into the text matching model and obtain the identifier text of the recommended document that matches the target input text output by the text matching model;

[0024] The text matching model is trained using the language model training device described above.

[0025] In another aspect, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the language model training method as described above, or the text matching method as described above.

[0026] In another aspect, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored therein, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement a language model training method as described above, or a text matching method as described above.

[0027] In another aspect, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the language model training method or the text matching method described above.

[0028] The beneficial effects of the technical solution provided in this application include at least the following:

[0029] This application discloses a language model training method, a text matching method, and related apparatus. The scheme provided in this application can integrate high-frequency words from the target domain into a first training text through granular segmentation, and use the SOP and MSP tasks as the first-stage training tasks to pre-train the language model using this first training text. Therefore, in the pre-training stage, this method enables the language model to learn not only general domain knowledge but also target domain knowledge. Furthermore, this method can also use a text matching task as the second-stage training task, using a second training text obtained from the target domain to train the language model, thereby enabling the language model to learn a wider range of fundamental knowledge from the target domain. This increases the generalization ability of the language model, thus ensuring the training effect of the model. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a schematic diagram of the structure of a language model training system provided in an embodiment of this application;

[0032] Figure 2 This is a flowchart of a language model training method provided in an embodiment of this application;

[0033] Figure 3 This is a flowchart of another language model training method provided in an embodiment of this application;

[0034] Figure 4 This is a schematic diagram illustrating a language model trained using the SOP task as the training task, as provided in an embodiment of this application.

[0035] Figure 5 This is a schematic diagram illustrating the masking process of a sentence segment using a language model provided in an embodiment of this application;

[0036] Figure 6 This is a schematic diagram of a pre-trained language model provided in this application embodiment, with SOP and MSP tasks as the first stage of training tasks.

[0037] Figure 7 This is a schematic diagram illustrating a language model trained using a second training text, as provided in an embodiment of this application.

[0038] Figure 8 This is a schematic diagram illustrating the training process of a language model provided in an embodiment of this application;

[0039] Figure 9 This is a visualization of the target language model and the basic reference model provided in this application embodiment, presented in the form of attention.

[0040] Figure 10 This is a visualization of another target language model and basic reference model provided in this application embodiment in the form of attention.

[0041] Figure 11 This is a visualization of another target language model and basic reference model provided in this application embodiment in the form of attention.

[0042] Figure 12 This is a visualization of another target language model and basic reference model provided in the embodiments of this application, presented in the form of attention.

[0043] Figure 13 This is a flowchart of a text matching method provided in an embodiment of this application;

[0044] Figure 14 This is a structural block diagram of a language model training device provided in an embodiment of this application;

[0045] Figure 15 This is a structural block diagram of a training device for another language model provided in an embodiment of this application;

[0046] Figure 16 This is a structural block diagram of a text matching device provided in an embodiment of this application;

[0047] Figure 17 This is a structural block diagram of a server provided in an embodiment of this application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0049] The solutions provided in this application can be applied to various scenarios such as cloud technology, artificial intelligence, and smart transportation. First, a brief introduction to the terms used in this application's embodiments will be given.

[0050] Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0051] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0052] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0053] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instruction-based learning.

[0054] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close relationship with linguistic research. NLP technologies typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0055] Language Model (LM): In the field of Natural Language Processing (NLP), a model is used to analyze and process language text. It can generally be categorized into grammatical rule-based language models, statistical language models, and neural network language models. Among these, the language model capable of performing specific downstream tasks is obtained by fine-tuning pre-trained language models (PLMs).

[0056] Pre-training involves using various unsupervised training methods and a large corpus to train an initial language model. During pre-training, the language model learns general language knowledge, reducing the training workload for downstream tasks and avoiding the need to train a new model from scratch. Fine-tuning involves training the language model obtained in the pre-training stage with a certain amount of manually annotated target domain knowledge, resulting in a language model capable of performing specific downstream tasks (such as text entailment recognition, dialogue matching, or question-answering search).

[0057] However, because the training samples used in the pre-training phase are general language knowledge, the language model lacks large-scale training closely related to downstream tasks. Therefore, it is difficult for the language model to learn target domain knowledge during pre-training. Furthermore, since the training task in the pre-training phase differs significantly from the training task in the fine-tuning phase (i.e., the specific downstream task), a larger amount of finely labeled corpus (i.e., language knowledge related to the downstream task and with manually annotated target domain information) is required to train the language model in the fine-tuning phase. In related technologies, acquiring finely labeled data typically requires significant human and material resources, resulting in high costs during language model training.

[0058] Figure 1 This is a schematic diagram of the structure of a language model training system provided in an embodiment of this application. See also... Figure 1 The system includes a server 110 and a terminal 120. A wired or wireless communication connection is established between the server 110 and the terminal 120. Optionally, the server 110 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal 120 can be a personal computer (PC), in-vehicle terminal, tablet computer, smartphone, wearable device, intelligent robot, or other terminal with data computing, processing, and storage capabilities.

[0059] In this embodiment, the terminal 120 in the system can be used to acquire training text (e.g., sentence samples, second training text, and third training text) and send the training text to the server 110. The server 110 can then use the training text to train the language model (e.g., pre-training and fine-tuning) to obtain a language model capable of performing specific tasks (e.g., a text matching model in the medical field).

[0060] Alternatively, the training text used to train the language model can also be pre-stored in the server 110. Correspondingly, the language model training system may not include the terminal 120.

[0061] Alternatively, the system can also perform specific tasks, such as text matching. Accordingly, terminal 120 in the system can be used to acquire input text (e.g., target input text) and send it to server 110 for searching. Server 110 pre-stores pre-trained and fine-tuned language models, such as text matching models. After acquiring the input text, the server can input it into the language model, which can then process the input text and output the result text. For example, the result text output by the language model could be the identifier text of recommended documents that match the input text. Then, server 110 can send the result text to terminal 120.

[0062] Optionally, the terminal 120 may also store a pre-trained and fine-tuned language model. After obtaining input text, the terminal 120 can directly input the input text into the language model, which can then process the input text and output the result text. Correspondingly, the training system for this language model may not include the server 110.

[0063] Based on the above introduction to terminology and application scenarios, the training method for the language model provided in this application embodiment will be described. This method can be applied to a computer device, which can be... Figure 1 Server 110 is shown in the scenario. The following explanation uses the application of this language model training method to a server as an example. Figure 2 As shown, the method includes the following steps.

[0064] Step 101: Obtain two sentence samples from the first document in the document library.

[0065] In this embodiment, the server pre-stores a document library, which contains not only multiple documents in a general domain but also multiple documents in a target domain. The target domain can be the domain in which the trained language model will be applied, such as the medical field. At the start of training, the server can retrieve multiple first documents from the document library and obtain two sentence samples from each first document.

[0066] Understandably, for each first document, the server can divide it into multiple sentences using a period as a delimiter. Then, the server can randomly select two sentences before and after a period from these multiple sentences as sentence samples.

[0067] Step 102: Perform granular segmentation on each sentence sample.

[0068] The server pre-stores a high-frequency word database, which includes not only general high-frequency words but also high-frequency words specific to the target domain. Based on this database, the server can perform granular segmentation on each sentence sample. Each granularly segmented sentence sample can be divided into multiple sentence segments, at least one of which includes multiple characters and at least one word from the high-frequency word database composed of those characters.

[0069] It is understood that the high-frequency word library can include words that are commonly used in daily life and work, as well as words that are frequently used in the target domain. The words in this embodiment include at least two characters. It is also understood that among the multiple sentence segments after this granular division, there may be a case where a certain sentence segment contains only a single character.

[0070] Optionally, the sentence sample in this embodiment can be Chinese text, and correspondingly, the characters in the first sentence text can refer to Chinese characters. Alternatively, the sentence sample can be English text, and correspondingly, the characters in the first sentence text can refer to words.

[0071] Step 103: Mask at least one sentence segment from the two sentence samples after granular division to obtain the first training text.

[0072] In this embodiment, the server can input two sentence samples after granular segmentation into a language model for masking processing. For the two sentence samples, the language model can perform masking processing on at least one sentence segment of the two sentence samples to obtain the first training text. Among them, some characters or words in each sentence segment after masking processing are replaced with a mask.

[0073] Optionally, the language model can randomly mask a certain percentage (e.g., 15%) of sentence segments in the two sentence samples. Furthermore, at least one sentence segment in the two sentence samples that is replaced with the mask [MASK] is a sentence segment that the language model needs to predict during pre-training.

[0074] Step 104: Using the SOP task and MSP task as the first stage of training tasks, the language model is pre-trained using the first training text.

[0075] The SOP task is used to predict the order of the two sentence samples in the first document, and the MSP task is used to predict at least one sentence segment that has been masked.

[0076] It is understandable that the order of the two sentence samples after granular segmentation in the first training text may be the same as or different from the order of the two sentence samples in the first document. Pre-training the language model using the SOP task as the training task allows the language model to learn the coherence between the two sentence samples.

[0077] Pre-training a language model using the MSP task as its training task allows the language model to learn some high-frequency words from sentence samples.

[0078] It is also understood that, since the document library provided in this application embodiment contains documents in the target domain and the high-frequency word library contains high-frequency words in the target domain, pre-training the language model with the SOP task and MSP task enables the language model to learn knowledge in the target domain as well as general domain knowledge.

[0079] Optionally, in embodiments of this application, a Bidirectional Encoder Representations from Transformers (BERT) model can be used as the initial language model, and the initial language model can be pre-trained using the first training text.

[0080] Step 105: Using the text matching task as the training task for the second stage, train the language model using the second training text.

[0081] After completing the first phase of training, the server can continue training the language model using a second training text. This second training text includes the first input text from the target domain and the identifier text of a second document in the target domain. The text matching task is used to predict the matching degree between the second document and the first input text.

[0082] Understandably, the text matching task in this second stage is the initial downstream task. Based on the target domain to which the language model obtained in the first stage (i.e., the pre-training stage) will be applied, and the text matching task to be performed by the language model, the server can retrieve multiple first input texts related to the text matching task from the database of that target domain, as well as the identifier texts of second documents related to the first input texts. The identifier text can be the title of the second document. Then, the server can use the first input texts and the identifier texts of the second documents as second training texts to train the language model, thereby achieving initial fine-tuning of the language model.

[0083] It is understandable that directly using the first input text and the second document as the second training text would result in an excessively large amount of data for the second training text, significantly reducing training efficiency. Therefore, in this embodiment, the first input text and the identifier text of the second document can be used as the second training text for the text matching task. Accordingly, after the second stage of training, the language model can learn the matching degree between the identifier text of the second document and the first input text, and this matching degree can be used as the matching degree between the second document and the first input text.

[0084] Optionally, the text matching task may specifically be a text entailment recognition task, a dialogue matching task, or a question-and-answer search task, etc.

[0085] It's also understandable that this second stage of training allows the language model to learn some fundamental knowledge of the target domain, thereby increasing its generalization ability. This enables the language model to quickly and effectively learn target domain knowledge using limited, precisely labeled data (i.e., manually annotated training text) when training on specific downstream tasks (i.e., text matching tasks). Thus, while ensuring the training effectiveness of the language model, the training cost can be effectively reduced.

[0086] In summary, this application provides a method for training a language model. This method integrates high-frequency words from the target domain into a first training text through granular partitioning, and uses the SOP and MSP tasks as the first-stage training tasks to pre-train the language model using this first training text. Therefore, in the pre-training stage, this method enables the language model to learn not only general domain knowledge but also target domain knowledge. Furthermore, this method can use a text matching task as the second-stage training task, employing a second training text obtained from the target domain to train the language model, thereby enabling the language model to learn a wider range of fundamental knowledge from the target domain. This increases the generalization ability of the language model, thus ensuring the training effect of the model.

[0087] Figure 3 This is a flowchart illustrating another language model training method provided in this application embodiment. This embodiment uses the application of this method to a server as an example for illustration. See also... Figure 3 The method includes:

[0088] Step 201: Obtain two sentence samples from the first document in the document library.

[0089] In this embodiment, the server pre-stores a document library containing multiple documents not only from a general domain but also from a target domain. At the start of training, the server can retrieve multiple first documents from the document library and obtain one or more sentence pairs from each first document, where each sentence pair includes two sentence samples. Optionally, the documents in the document library can originate from books, journals, encyclopedias, forums, and social media platforms, etc.

[0090] In this context, a general domain refers to an open domain that can provide a large number of training samples for the pre-training process of a language model. Pre-training a language model using multiple training samples from a general domain allows the language model to learn general syntactic and semantic knowledge. The target domain can be the area where the trained language model will be applied, such as the medical field.

[0091] Understandably, for each first document, the server can divide it into multiple sentences using a period as a delimiter. Then, the server can randomly select two sentences before and after a period from these multiple sentences to form a sentence pair.

[0092] For example, assuming the target domain is the medical field, the document library contains multiple documents related to this medical field. The server can retrieve k first documents from this document library. The document set D of these k first documents can be represented as D = {d1, d2, d3, ..., dk}. n ...d k-1 d k}, where k is the total number of the first documents contained in the document set D, and k is a positive integer greater than 1, d n Let n be the nth document in the document set D, where n is a positive integer greater than or equal to 1 and less than or equal to k.

[0093] For the nth first document d in the document set D n The server can use a period as a delimiter to define the first document d. n The document is divided into multiple sentences to obtain the first document d. n Sentence sequence d n ={d n,1 d n,2 ...d n,m}. Where m is the first document d n The total number of sentences included, where m is an integer greater than 1, and dn,i is the first document d n The i-th sentence in the document, where i is an integer not greater than m. The server obtains this first document d. n After obtaining the sentence sequence, the server can randomly select two consecutive sentences from that sequence as sentence samples. For example, the server can select the first document d. n The first sentence d n,1 And the second sentence d n,2 As a sentence sample.

[0094] Step 202: Perform granular segmentation on each sentence sample.

[0095] The server pre-stores a high-frequency word database, which includes not only general high-frequency words but also high-frequency words specific to the target domain. Based on this database, the server can perform granular segmentation on each sentence sample. Each granularly segmented sentence sample can be divided into multiple sentence segments, at least one of which includes multiple characters and at least one word from the high-frequency word database composed of those characters.

[0096] It is understood that words in the high-frequency word library include at least two characters. The general high-frequency words refer to words that appear frequently in daily life and work, while the target domain high-frequency words refer to words that appear frequently in the target domain. The words in this application embodiment include at least two characters. It is also understood that among the multiple sentence segments after this granular division, there may be a case where a certain sentence segment only includes a single character.

[0097] Optionally, for each sentence sample, the server can first perform word segmentation to identify the words contained in the sentence. Then, the server can match the identified words with various high-frequency words in a high-frequency word database. If a word matches any high-frequency word in the database, the server can insert that word into the sentence sample. The inserted word is located after the last character contained in that word. If several consecutive characters in the sentence sample can form multiple words belonging to the high-frequency word database, then for each word, the server can insert it after the last character contained in that word.

[0098] For example, assume that the sentence sample includes the three characters "motion sickness patch". Since these three characters can form two words, namely "motion sickness" and "motion sickness patch", the server can insert the word "motion sickness" after the character "车" and insert the word "motion sickness patch" after the character "贴". Thus, the server can obtain the sentence sample after granularity division: "晕 - 车 - 晕车 - 贴 - 晕车贴".

[0099] It can be understood that the sentence sample after granularity division can include multiple sentence segments. Each sentence segment includes multiple characters and one or more words formed by these multiple characters. That is, if several consecutive characters in the sentence sample can form multiple words, then these consecutive characters and these multiple words all belong to one sentence segment. Or, if a certain character in the sentence sample cannot form a word with its adjacent characters, then this character can be a sentence segment alone. Each sentence segment in the sentence sample can also be called a sub - graph. And based on the above - mentioned division method, it can be known that after the server performs granularity division on each sentence sample to obtain multiple sub - graphs, two adjacent sub - graphs do not intersect, that is, the last character or word in the previous sub - graph among two adjacent sub - graphs cannot form a new word with the first character or word in the subsequent sub - graph.

[0100] For example, for the sentence sample "motion sickness patch usage method", after the server performs word segmentation on it, it can be determined that the following words are included in this sentence sample: "motion sickness", "motion sickness patch", "usage", and "method". Assume that the target domain is the medical field, and the words "motion sickness", "usage", and "method" are all general high - frequency words in the high - frequency word library, and "motion sickness patch" is a high - frequency word in the medical field of the high - frequency word library. Based on the high - frequency words existing in the high - frequency word library in this sentence sample, after the server performs granularity division on this sentence sample, it can obtain "晕 - 车 - 晕车 - 贴 - 晕车贴 - 使 - 用 - 使用 - 方 - 法 - 方法".

[0101] The sentence sample after the above - mentioned granularity division includes three sentence segments: "晕 - 车 - 晕车 - 贴 - 晕车贴", "使 - 用 - 使用", and "方 - 法 - 方法". Referring to this example, it can be seen that there is no direct semantic connection among the three sentence segments in this sentence sample (that is, the sub - graphs do not interact with each other). And each sentence segment contains sentence components at the character granularity (such as the character "晕") and sentence components at the word granularity (such as the word "motion sickness").

[0102] It is also understandable that sentence components at the character level often possess multiple semantic meanings. If a language model is trained based on sentence samples divided at the character level, numerous ambiguities will arise during the training process. However, this embodiment combines character-level and word-level segmentation, introducing high-frequency words from a high-frequency word library as word-level sentence components into the sentence samples, effectively mitigating the ambiguities inherent in character-level sentence components. For example, the words "car sticker" and "motion sickness sticker" differ by only one character but have completely different meanings. This embodiment, by introducing high-frequency words, can accurately identify and learn the high-frequency medical term "motion sickness sticker." Furthermore, due to the introduction of more granular knowledge, the language model can learn alignment information between two sentence samples from a larger number of samples, which is beneficial for improving the performance of the language model in downstream tasks (such as text matching tasks).

[0103] Optionally, the sentence sample in this embodiment can be Chinese text, and correspondingly, the characters in the first sentence text can refer to Chinese characters. Alternatively, the sentence sample can be English text, and correspondingly, the characters in the first sentence text can refer to words.

[0104] Step 203: Mask at least one sentence segment from the two sentence samples after granular division to obtain the first training text.

[0105] For two sentence samples after granular segmentation, the server can concatenate the two sentence samples and input them into the language model. The language model can extract at least one sentence segment from the concatenated two sentence samples for masking, thereby obtaining the first training text. For example, the language model can extract a certain proportion (e.g., 15%) of the sentence segments included in the concatenated two sentence samples for masking.

[0106] Optionally, for each obtained sentence segment, the language model can replace the first sentence component in the segment with a mask and the second sentence component with other sentence components, while the remaining components in the segment remain unchanged. Here, the first and second sentence components in the sentence segment are both single characters or words. The words in the first and second sentence components must include at least two characters. That is, the server can replace some single characters (or words) in each sentence segment with a mask and replace some single characters (or words) with other single characters (or other words).

[0107] Understandably, the sentence components in the granularized sentence sample are arranged in a combination of character-level and word-level granularity. This arrangement results in a significant amount of redundant information in the sentence sample. If all sentence components in this segment are replaced with a mask, the language model will directly predict the masked sentence segment based on this redundant information in the MSP task, leading to difficulty in convergence. Therefore, the language model can replace only the first sentence component in the segment with a mask. Furthermore, to reduce the difference between the pre-training task and the downstream task, the language model can replace the second sentence component in the segment with sentence components common in downstream tasks (e.g., high-frequency words in the target domain), while leaving the remaining components in the segment unchanged.

[0108] Optionally, for each obtained sentence segment, the language model can randomly select a first proportion (e.g., 80%) of sentence components in that segment as the first sentence component and replace this first sentence component with a mask [MASK]. Furthermore, the language model can select a second proportion (e.g., 10%) of sentence components in that segment as the second sentence component and replace this second sentence component with other sentence components. The remaining sentence components in the segment remain unchanged.

[0109] For example, suppose the server obtains two sentence samples, d. n,m-1 and d n,m Then refer to Figure 4 The server can concatenate the two sentence samples using [SEP] and append them to "[CLS]d". n,m-1 [SEP]d n,m The input is given in the form of "[CLS]". Here, "[SEP]" is a special separator, also known as a concatenation character, which the language model uses to distinguish between the two sentence samples. "[CLS]" is a special tag that combines the semantic information from the two sentence samples into a unified semantic information, which is then used as the output of the language model.

[0110] Then, for the sentence segment S1 = {w} in the two sentence samples b-2 w b-1 w b},like Figure 5 As shown, the language model can classify the sentence segment S1={w b-2 w b-1 w b The first sentence component w in} b Replace with the mask [MASK], and change the second sentence component w b-2 Replace with high-frequency words from the high-frequency word database. c The remaining component w in sentence segment S1 b-1This remains unchanged. In subsequent MSP tasks, the language model is able to predict the individual sentence components included in the sentence segment.

[0111] Optionally, for each obtained sentence segment, the language model can also replace all sentence components in that sentence segment with a mask.

[0112] Step 204: Determine the order labels of the two sentence samples based on their order in the first training text.

[0113] For each pair of sentence samples in the first training text, if the server determines that the concatenation order of the two sentence samples in the first training text is the same as their order in the first document, then the server can set the order label of the two sentence samples as the first label. If the server determines that the order of the two sentence samples in the first training text is different from their order in the first document, then the server can set the order label of the two sentence samples as the second label. The first label and the second label are different; for example, the first label can be "1" and the second label can be "0".

[0114] For example, if the first document d n Two sentence samples d n,m-1 and d n,m The concatenation order in the first training text is "d n,m-1 [SEP]d n,m ", then because the splicing order is different from that of the two sentence samples in the first document d n Since the order of the two sentence samples is the same, their sequential labels can be set as the first label (e.g., "1"). If the concatenation order of the two sentence samples in the first training text is d... n,m [SEP]d n,m-1 Then, because the splicing order is different from that of the two sentence samples in the first document d, n Since the order of the two sentence samples is different, the order label of the two sentence samples can be set as the second label (e.g., "0").

[0115] It is understandable that the first training sample, consisting of two sentence samples with the first sequential label, can also be called a positive sample. Similarly, the first training sample, consisting of two sentence samples with the second sequential label, can also be called a negative sample.

[0116] Step 205: Using the SOP task and MSP task as the first stage of training tasks, the language model is pre-trained using the first training text.

[0117] The SOP task is used to predict the order of the two sentence samples in the first document, and the MSP task is used to predict at least one sentence segment that has been masked.

[0118] During the process of pre-training the language model using the first training text, the server can input the first training text into the language model to obtain the predicted order and predicted sentence segments output by the language model. Then, the server can determine the first loss value of the language model in the SOP task based on the predicted order output by the language model and the sequence labels of two sentence samples in the first training text. The predicted order output by the language model can include: the probability that the concatenation order of the two sentence samples in the first training text matches the order indicated by the first label, and the probability that the concatenation order of the two sentence samples matches the order indicated by the second label.

[0119] Furthermore, the server can determine the second loss value of the language model in the MSP task based on the predicted sentence segment and the masked sentence segments in the two sentence samples after granular segmentation. After determining the first and second loss values ​​of the language model in the first stage, the server can pre-train the language model based on these first and second loss values.

[0120] It is understandable that the order of the two sentence samples after granular segmentation in the first training text may be the same as or different from the order of the two sentence samples in the first document. Pre-training the language model using the SOP task as the training task allows the language model to learn the coherence between the two sentence samples.

[0121] In this embodiment, the server can divide the determined multiple first training texts into multiple batches for training. The server can determine the first loss value of a batch by summing the first loss values ​​of all the first training texts within a batch, and determine the second loss value of a batch by summing the second loss values ​​of all the first training texts within a batch.

[0122] In this embodiment, the first loss value of the language model in the SOP task and the second loss value in the MSP task can both be calculated using the cross-entropy loss function. That is, the loss function in the SOP task and the loss function in the MSP task can both be the cross-entropy loss function. The loss function L for the SOP task... SOP It can be represented as:

[0123]

[0124] Where x represents the first training text. p(x) represents the true probability distribution of the order labels of the first training text on the first and second labels. q(x) represents the probability distribution of the predicted order of the first training text output by the language model on the first and second labels. The first loss value L SOP It can be the sum of the loss values ​​of the language model within a training batch.

[0125] In this embodiment, the probability distribution of the sequence label of the first training text on the first label and the second label can be represented as [a1, a2]. a1 represents the probability that the sequence label is the first label, and a2 represents the probability that the sequence label is the second label. The probability distribution of the predicted order output by the language model on the first label and the second label can be represented as [b1, b2]. b1 represents the probability that the predicted order is the order indicated by the first label, and b2 represents the probability that the predicted order is the order indicated by the second label. Therefore, this SOP task is used to predict whether the concatenation order of two sentence samples in the first training text is the same as the order of the two sentence samples in the first document. Alternatively, it can be understood as: this SOP task is used to predict whether the concatenation order of two sentence samples in the first training text is correct.

[0126] For example, suppose the first training text x has the first label as its sequential label, then a1 = 1, a2 = 0. Correspondingly, the probability distribution p(x) of the sequential label of the first training text x on the first and second labels is [1, 0]. If the probability that the concatenation order of two sentence samples in the first training text x output by the language model is the order indicated by the first and second labels is 0.95 and 0.05 respectively, i.e., b1 = 0.95, b2 = 0.05, then the probability distribution q(x) of the predicted order on the first and second labels can be [0.95, 0.05].

[0127] The second loss value of the language model in the MSP task can be calculated using the loss function of the MSP task. The loss function L for the MSP task is... MSP It can be represented as:

[0128]

[0129] Where, x i p(x) represents the i-th character in the first training text x that is replaced by a mask. r represents the number of characters in the first training text x that are masked, where r is a positive integer greater than or equal to 1, and i is a positive integer not greater than r. i ) represents the i-th character x in the first training text x that is replaced by a mask. i The probability distribution of the actual characters corresponding to the text in the character set. q(x) i ) represents the i-th character x in the first training text x output by the language model that is replaced by a mask. i The predicted probability distribution corresponding to the text library. This second loss value L MSP It can be the sum of the loss values ​​of the language model within a training batch.

[0130] In an embodiment of the present application, assuming that there are J characters in the character library, the i-th masked and replaced character x in the first training text x i The corresponding probability distribution p(x i ) in the character library can be [p1,..., p j ,..., p J . Among them, p j represents the probability that the character x i is the character Wj, and j is a positive integer not greater than J. The predicted probability distribution q(x i ) in the character library for the i-th masked and replaced character x in the first training text x output by the language model can be [q1,..., q i ,..., q j ,..., q J . Among them, q j represents the probability that the character x i output by the language model is the j-th character in the character library.

[0131] Figure 6 is a schematic diagram of the process of pre-training a language model in an embodiment of the present application, where the training tasks of the first stage are the SOP task and the MSP task. Referring to Figure 6 , assuming that for two sentence samples d n in the first document d n,m-1 and d n,m , after granularity division and masking processing, the first training sample obtained is "[CLS]上火上火[SEP][MASK][CLS]". During the process of using the first training text to train the language model in the first stage, the server can determine the first loss value determined by the language model in the SOP task and the second loss value determined in the MSP task as the overall loss value of the language model in the first stage. And the server can pre-train the language model based on the overall loss value of the first stage. Correspondingly, the loss function L of the first stage can be expressed as: L = L SOP + L MSP .

[0132] Step 206: Obtain a first input text from a database in the target domain and an identification text of a second document associated with the first input text to obtain a second training text.

[0133] In this embodiment, the server can obtain a second training text based on the target domain to which the language model obtained in the first stage (i.e., the pre-training stage) is to be applied, and the specific downstream task (i.e., text matching task) to be performed by the language model, by retrieving multiple first input texts related to the downstream task from the database of the target domain, and the identifier text of a second document related to the first input text. The identifier text can be the title of the second document. Optionally, in addition to the title of the second document, the identifier text may also include a summary and / or keywords of the second document.

[0134] For example, assuming the specific downstream task to be performed by this language model is a text matching task in the medical field, the server can obtain multiple second training texts from a medical database. The set G of these multiple second training texts can be represented as:

[0135] G = {(q1,T1),(q2,T2),…,(q n ,T n ),…,(q m-1 ,T m-1 ),(q m ,T m )};

[0136] Where m is the total number of second training texts obtained by the server, and q n T is the first input text in the nth second training text. n ={t1,t2,…,t o} is the first input text q n The relevant set of identifier text for the second document, t o This refers to the identifier text of the Oth second document in the identifier text set, where O is a positive integer greater than 1. For example... Figure 7 As shown, the server can use [SEP] to process the first input text q. n and the set of identifier text T n The identifier text t of the second document n The text is then concatenated to obtain a second training text. Where t... n It can identify the text set T n The identifier text of any second document in the document.

[0137] Optionally, for the set of identifier text T n For each second document in the input text, the server can calculate the identifier text of that second document and compare it with the first input text q. n The sum of the matching scores. The server can select at least one second document's identifier text with a high matching score from the first input text q. nThe text is then spliced ​​together to obtain at least one second training text.

[0138] Step 207: Calculate the matching degree between the first input text and the second document to obtain the matching degree label of the second training text.

[0139] For each second training text, the server can calculate the matching degree between the first input text and the second document based on the relevance between the first input text and the second document in the second training text. Based on the matching degree between the first input text and the second document in the second training, the server can determine the matching degree label of the second training text.

[0140] Optionally, the server can pre-store multiple matching degree tags, each corresponding to a matching degree range. For each second training text, after calculating the matching degree between the first input text and the second document in that second training text, the server can determine the matching degree tag corresponding to the first input text and the second document based on the matching degree range in which that matching degree falls. The server can use methods such as cosine similarity algorithm, Gaussian distance, or similarity matrix to calculate the matching degree between the first input text and the second document.

[0141] In this embodiment, the matching degree label can be represented numerically. Furthermore, the larger the matching degree label of the second training text, the higher the relevance between the first input text and the second document.

[0142] For example, assume the server is pre-configured with the following 5 matching degree labels: f1, f2, f3, f4, and f5. The matching degree range for label "f1" is 0 ≤ f < 0.2. That is, when the matching degree f between the first input text and the second document in the second training text is greater than or equal to 0 and less than 0.2, the matching degree label for the second training text can be set to f1. The matching degree range for label "f2" is 0.2 ≤ f < 0.4. The matching degree range for label "f3" is 0.4 ≤ f < 0.6. The matching degree range for label "f4" is 0.6 ≤ f < 0.8. The matching degree range for label "f5" is 0.8 ≤ f ≤ 1.

[0143] If the server calculates that the matching degree f between the first input text and the second document is 0.9, then since the matching degree range of 0.9 is 0.8≤f≤1, the matching degree label of the second training text can be configured as f5.

[0144] Step 208: Using the text matching task as the training task for the second stage, train the language model using the second training text.

[0145] After completing the first stage of training, the server can use a text matching task as the second stage of training and continue training the language model using a second training text. This text matching task is used to predict the matching degree between the second document and the first input text. Optionally, this text matching task can be a text entailment recognition task, a dialogue matching task, or a question-answering search task, etc.

[0146] Based on steps 207 and 208 above, it is known that the server uses text matching as the training task in the second stage, and uses the first input text and the identifier text of the second document as the second training text to train the language model. This allows the language model to learn the correlation between the first input text and the identifier text of the second document during the training process. Furthermore, the language model can predict the matching degree between the first input text and the second document based on the correlation between the identifier text of the first input text and the second document.

[0147] It is understandable that directly using the first input text and the second document as the second training text would result in an excessively large amount of data for the second training text, significantly reducing training efficiency. Therefore, in this embodiment, the first input text and the identifier text of the second document can be used as the second training text for the text matching task. Accordingly, after the second stage of training, the language model can learn the matching degree between the identifier text of the second document and the first input text, and this matching degree can be used as the matching degree between the second document and the first input text.

[0148] In the second stage of training, the server can input the second training text into the language model and obtain the predicted matching score of the language model for that second training text. For each second training text, the server can determine the third loss value of the language model in this text matching task based on the predicted matching score output by the language model and the matching score label between the second document and the first input text in the second training text, and train the language model based on the third loss value. It can be understood that this second stage of training task is the initial downstream task. By using the second training text to train the language model, the server can achieve preliminary fine-tuning of the language model.

[0149] The third loss value of the language model in the text matching task can also be calculated using the cross-entropy loss function. That is, the loss function for this text matching task can be the cross-entropy loss function. The loss function L for this text matching task... TM It can be represented as:

[0150]

[0151] In the above formula, s represents the second training text, p(s) represents the probability distribution of the matching labels of the second training text s, and q(s) represents the probability distribution of the predicted matching scores of the second training text s output by the language model. The third loss value L... TM It can be the sum of the loss values ​​of the language model within a training batch.

[0152] Understandably, if the server pre-sets K (K is an integer greater than or equal to 1) matching labels, then the probability distribution p(s) of the matching labels of the second training text s on the aforementioned K matching labels can be represented as [c1, ..., ck, ..., cK]. Here, ck represents the probability that the matching label is the k-th matching label, and k is a positive integer not greater than K. Furthermore, the probability distribution q(s) of the predicted matching degree of the second training text s on the aforementioned K matching labels can be represented as [e1, ..., ek, ..., eK]. Here, ek represents the probability that the predicted matching degree is the k-th matching label among the K matching labels.

[0153] For example, suppose K=5, meaning the server has 5 preset matching labels: L1, L2, L3, L4, L5, and the matching label of the second training text s is L3, i.e., c1=c2=c4=c5=0, and c3=1. Then the probability distribution p(s) of the matching label of the second training text s on the above 5 matching labels can be [0, 0, 1, 0, 0]. If the predicted matching degree of the second training text s output by the language model has probabilities of 0.005, 0.02, 0.95, 0.015, and 0.01 respectively on the above 5 matching degree labels, i.e., e1 = 0.005, e2 = 0.02, e3 = 0.95, e4 = 0.015, and e5 = 0.01, then the probability distribution q(s) of the second training text s on the above 5 matching degree labels can be [0.005, 0.02, 0.95, 0.015, 0.01].

[0154] Continue to refer to Figure 7 For the second training text "[CLS]qn[SEP]t" input to the language model n The language model can aggregate the overall semantic information of the second training text into the first [CLS] in the second training text. After obtaining the [CLS], the classifier of the language model can determine the distribution probability of the predicted matching degree of the second training text on the preset K matching degree labels based on the overall semantic information of the second training text aggregated by the [CLS].

[0155] Through the second stage of training described above, the language model learns some basic knowledge of the target domain, thereby increasing its generalization ability. This allows the language model to learn target domain knowledge using limited, precisely labeled data (i.e., manually annotated training text) when training on specific downstream tasks. Thus, while ensuring the training effectiveness of the language model, the training cost is effectively reduced. Furthermore, by using text matching as the training task in this second stage, and by adding training tasks closely related to the downstream task (i.e., text matching), the language model can converge quickly in the downstream task and effectively improve its performance in text relevance matching.

[0156] Step 209: Using the text matching task as the third stage training task, train the language model using the third training text.

[0157] After completing the second phase of training, the server can continue with text matching as the training task and use third training text to train the language model in the third phase. This third training text includes the second input text from the target domain and the identifier text of the third document in the target domain. The matching degree between the first input text and the second document in the second training text is machine-annotated, while the matching degree between the second input text and the third document in the third training text is manually annotated. In this embodiment, the second training text can also be referred to as recall data, and the third training text can also be referred to as precision-annotated data.

[0158] It is understood that the training process in the third stage is the same as that in the second stage, and the output of the language model is also the same. Therefore, the language model can use the same loss function as in the second training stage to determine the loss value of the language model in the third stage, and train the language model based on the loss value. This will not be elaborated further in the embodiments of this application.

[0159] It's also understandable that, since the second training text is randomly obtained by the server and machine-annotated, the server can obtain a large number of second training texts for large-scale training of the language model closely related to downstream tasks. However, the machine struggles to ensure the relevance between the first input text and the second document in the second training text, and it also struggles to accurately annotate the matching degree between the first input text and the second document. Therefore, the second stage of training is insufficient for achieving precise fine-tuning of the language model.

[0160] Because the third training text is manually annotated, and the accuracy of manually annotated training text is far greater than that of machine-annotated training text, the server can fine-tune the language model using a smaller amount of third training text (i.e., finely labeled data) when training the language model for specific downstream tasks.

[0161] In this embodiment, if the text matching task is a text entailment task, then the first input text in the second training text and the second input text in the third training text can both be premise texts in the text entailment task. The identifier text of the second document in the second training text and the third document in the third training text can both be hypothesis texts that match the premise text.

[0162] If the text matching task is a dialogue matching task, then the first input text in the second training text and the second input text in the third training text can both be dialogue texts for the dialogue matching task. The second document in the second training text and the third document in the third training text can both be response texts to the dialogue text.

[0163] If the text matching task is a question-and-answer search task, then the first input text in the second training text and the second input text in the third training text can both be question texts for the question-and-answer search task. The second document in the second training text and the third document in the third training text can both be answer texts for that question text.

[0164] Figure 8 This is a schematic flowchart illustrating the training process of a language model provided in an embodiment of this application. (Reference) Figure 8 As can be seen, the training process of the language model provided in this application embodiment can be divided into three stages. The first stage can be regarded as the pre-training process of the language model, and the third stage can be regarded as the fine-tuning stage of the language model. Correspondingly, the second stage can be regarded as the transition stage between pre-training and fine-tuning of the language model.

[0165] Among them, in the first stage, the "[CLS] Suffering from excessive internal heat Suffering from excessive internal heat [SEP] [MASK] [CLS]" input into the language model is the first training text obtained after granularity division and masking processing. In the second stage, for the second training text "[CLS] Q1 [SEP] T1 [CLS]" input into the language model, Q1 represents the first input text, and T1 represents the identification text of the second document associated with the first input text Q1. In the third stage, for the third training text "[CLS] Q2 [SEP] T2 [CLS]" input into the language model, Q2 represents the second input text, and T2 represents the identification text of the third document in the target domain associated with the second input text Q2.

[0166] It can also be understood that after being trained using the training method provided in the embodiments of the present application, the obtained language model can be used as a teacher model to transfer the knowledge learned by itself to a student model with weaker learning ability through knowledge distillation. Alternatively, this language model can be used as an auxiliary module for the semantic matching module and the search module in the target domain to assist in screening out highly relevant samples for related tasks.

[0167] Table 1 and Table 2 respectively show the performance of the language model (hereinafter referred to as the target language model) obtained by using the method provided in the embodiments of the present application and the language model in the related art on the long-tail dataset and the global dataset in the medical field. Among them, the long-tail dataset refers to the text that contains semantically similar texts in the medical field and is difficult to distinguish or differentiate. The global dataset refers to the text that contains all aspects of the medical field. The embodiments of the present application use the relevance matching task as the test task, and use the basic BERT model and the model obtained by only performing the training in the first stage and the second stage (i.e., the two-stage language model) as the reference models of the target language model. In addition, the 5-grade accuracy rate in the embodiments of the present application refers to the accuracy rate of the predicted matching degree output by the model after dividing the matching degree of the text into five grades. The 3-grade accuracy rate refers to the accuracy rate of the predicted matching degree output by the model after dividing the matching degree of the text into three grades. Furthermore, the embodiments of the present application also use the Normalized Discounted Cumulative Gain (NDCG) and the inversion pair accuracy rate as the evaluation indicators of the model on the dataset. Among them, when using NDCG as the evaluation indicator and calculating the NDCG of the model during the test process, the predicted matching degrees output by the model will be sorted in descending order. NDCG@1 means that during the process of evaluating the performance of the model, only the first (i.e., the highest) matching result of the predicted matching degree is concerned. NDCG@5 means that during the process of evaluating the performance of the model, only the first five matching results of the predicted matching degree are concerned.

[0168] Table 1

[0169]

[0170] Table 1 shows the test results of the three models on the global test set. As can be seen from Table 1, compared to using the base BERT model alone for semantic matching tasks, the target language model improved the accuracy of the 5-point scale by 1.7%, NDCG@5 by 0.004%, and inversion pair accuracy by 1.1% on the global test dataset.

[0171] Table 2

[0172]

[0173] Table 2 shows the test results of the three models on the long-tail test set. Referring to Table 2, compared to using the basic BERT model alone for semantic matching tasks, the target language model improved the accuracy of the 5-point scale by 2.6%, NDCG@5 by 0.009, and inversion pair accuracy by 1.8% on the long-tail test dataset.

[0174] Figure 9 , Figure 10 , Figure 11 as well as Figure 12 The performance of the language model (hereinafter referred to as the target language model) obtained after training using the method provided in the embodiments of this application, and the basic reference model in the related art are shown respectively by visual attention.

[0175] like Figure 9 As shown, the target input text is "Can I still eat an egg if it's moldy?", and the identifier text of the document associated with the target input text is "Can I still eat an egg if its shell is moldy?". Figure 9 The left side shows a visualization of the attention mechanism of the base reference model, while the right side shows a visualization of the attention mechanism of the target language model. (Reference) Figure 9 It can be seen that "egg" and "egg shell" are semantically related but have different meanings. The base reference model identifies "moldy" as a modifier of "egg." However, the target language model focuses on "moldy" within the document identifier "egg shell," paying particular attention to the "shell." Therefore, the target language model can more accurately predict the relevance between the target input text and its corresponding document identifier.

[0176] like Figure 10 As shown, the target input text is "right shoulder pain, about half a year", and the document associated with the target input text is "right shoulder pain and arm pain for more than half a year". Figure 10On the left is the visualization of the attention of the base reference model, and on the right is the visualization of the attention of the target language model. From the alignment relationship, the attention of the "right" character in the "right shoulder" at the bottom layer should pay more attention to "shoulder" rather than "left". In the base reference model, because the word "left and right" is a high-frequency word, the "right" character in "right shoulder" pays too much attention to the "left" character. However, the target language model can determine that "left and right" in "about half a year" is already a word, and can make the "right" character in "right shoulder" pay more attention to the word "shoulder", thus eliminating the incorrect alignment relationship.

[0177] As Figure 11 shown, on the left is the visualization of the attention of the base reference model, and on the right is the visualization of the attention of the target language model. For the target input text "Is it effective to take AB on an empty stomach or after a meal?", and the identification text of the document associated with this target input text "Is it better to take cold medicine on an empty stomach or after a meal?". Here, AB refers to a certain cold medicine. Refer Figure 11 to know that the base reference model cannot recognize that "AB" is a type of "cold medicine", so the information exchanged between the two is very little. However, the target language model can notice the hypernym and hyponym relationship between "AB" and "cold medicine", and can accurately notice that "AB" is a type of "cold medicine".

[0178] As Figure 12 shown, on the left is the visualization of the attention of the base reference model, and on the right is the visualization of the attention of the target language model. For the target input text "Illustration of applying motion sickness patch on the ear", and the identification text of the document associated with this target input text "Schematic diagram of where to apply motion sickness patch". The base reference model fails to learn the high-frequency medical term "motion sickness patch", so it misidentifies this term as "car sticker". For the target language model, since high-frequency words in the medical field are introduced during the training of this target language, this target language model can accurately identify this high-frequency word, enhancing the ability to recognize ambiguous texts, and thus accurately predicting the relevance between the target input text and the identification text.

[0179] Optionally, the server can stop the pre-training in the first stage and start the pre-training in the second stage when the loss function L of the language model converges in the first stage, or when the accuracy of the language model in the first stage reaches the preset accuracy, or when the number of training rounds of this language model in the first training stage reaches the preset number of rounds. Similarly, the server can stop the pre-training in the second stage when the loss function L TMUpon convergence, or when the language model's accuracy in the second stage reaches a preset accuracy, or when the language model has completed a preset number of training epochs in the second training stage, the second stage of pre-training is stopped, and the third stage of pre-training begins. Furthermore, the server can stop the third stage of pre-training when the loss function in the third stage converges, or when the language model's accuracy in the third stage reaches a preset accuracy, or when the language model has completed a preset number of training epochs in the third training stage, thus obtaining a language model capable of performing text matching tasks.

[0180] It is understood that the order of steps in the pre-training method for the language model provided in this application embodiment can be appropriately adjusted, and steps can be added or removed as needed. For example, step 201 can be deleted as needed, that is, the sentence sample can be obtained by the server in advance. Any variations that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application, and therefore will not be elaborated further.

[0181] In summary, this application provides a method for training a language model. This method integrates high-frequency words from the target domain into a first training text through granular partitioning, and uses the SOP and MSP tasks as the first training tasks, employing this first training text to pre-train the language model. Thus, in the pre-training stage, this method enables the language model to learn not only general domain knowledge but also target domain knowledge. Furthermore, this method can use a text matching task as the second training task, employing a second training text obtained from the target domain to train the language model, thereby enabling the language model to learn a wider range of fundamental knowledge from the target domain. This increases the generalization ability of the language model, thus ensuring the training effect of the model.

[0182] Figure 13 This is a flowchart illustrating a text matching method provided in this application. This text matching method can be applied to computer devices, for example, it can be applied to… Figure 1 Server 110 or terminal 120 in the scenario shown. See also Figure 6 The method may include the following steps.

[0183] Step 301: Obtain the target input text.

[0184] The target input text can be input text specific to the target domain.

[0185] As a first possible example, if the text matching method is applied to a server, the target input text can be sent from the terminal to the server.

[0186] As a second possible example, if the text matching method is applied to a terminal, the target input text can be entered into the terminal by the terminal user, or the search text can be sent to the terminal from another terminal.

[0187] Step 302: Input the target input text into the text matching model to obtain the identifier text of the recommended document that matches the target input text.

[0188] The text matching model can be adopted by the server, such as... Figure 2 and Figure 3 The language model shown is obtained through pre-training and fine-tuning. The text matching model outputs the identifier text of recommended documents that match the target input text. This identifier text can be the title of the recommended document.

[0189] In the first possible example above, the server pre-stores a text matching model. After obtaining the target input text, the server can input the target input text into the text matching model for processing and obtain the identifier text of the recommended documents that match the target input text, output by the text matching model. Then, the server can use the identifier text of the recommended documents output by the text matching model.

[0190] In the second possible example above, the server can send the trained text matching model to the terminal. After receiving the target input text, the terminal can directly input the target input text into the text matching model for processing and obtain the identifier text of the recommended document output by the text matching model.

[0191] In summary, this application provides a text matching method that can acquire target input text and input it into a text matching model for processing, thereby obtaining the identifier text of recommended documents output by the text matching model. Since the text matching model is pre-trained using the language model training method provided in the above embodiments and then fine-tuned, the identifier text of recommended documents output by the text matching model is relatively comprehensive and accurate, effectively ensuring the reliability of the text matching results.

[0192] Figure 14 This is a structural block diagram of a language model training device provided in this application. This language model training device can be applied to a computer device, which can be... Figure 1 Server 110 in the scenario shown. (For example...) Figure 14 As shown, the device includes:

[0193] The acquisition module 401 is used to acquire two sentence samples from the first document in the document library, which contains documents in the target domain.

[0194] The segmentation module 402 is used to perform granular segmentation on each sentence sample. The granular segmented sentence sample includes multiple sentence segments, at least one sentence segment includes multiple characters, and at least one word belonging to the high-frequency word library composed of multiple characters. The high-frequency word library contains high-frequency words in the target domain.

[0195] The masking module 403 is used to mask at least one sentence segment in two sentence samples after granular division to obtain the first training text.

[0196] Training module 404 is used for training tasks in the first stage, with the SOP task and MSP task as the first stage. The language model is pre-trained using the first training text. The SOP task is used to predict the order of two sentence samples in the first document, and the MSP task is used to predict at least one sentence segment that has been masked.

[0197] The training module 404 is also used to train the language model with a text matching task as the second stage training task. The second training text includes the first input text in the target domain and the identifier text of the second document in the target domain. The text matching task is used to predict the matching degree between the second text and the first input text.

[0198] Optionally, the masking module 403 is used to: obtain at least one sentence segment from two sentence samples after granular division; for each sentence segment, replace the first sentence component in the sentence segment with a mask, and replace the second sentence component in the sentence segment with other sentence components; wherein the first sentence component and the second sentence component are both single characters or words.

[0199] Optionally, the training module 404 is configured to: input the first training text into the language model to obtain the predicted order and predicted sentence segments output by the language model; determine a first loss value based on the predicted order and the sequence labels of the two sentence samples, wherein the sequence labels are used to indicate the order of the two sentence samples in the first document; determine a second loss value based on the predicted sentence segments and the masked sentence segments in the two sentence samples after granular segmentation; and pre-train the language model based on the first loss value and the second loss value.

[0200] Optionally, refer to Figure 14 The training device for the language model further includes a sequence labeling module 405, which is used to: set the sequence label of the two sentence samples as a first label if the order of the two sentence samples in the first training text is the same as the order in the first document; and set the sequence label of the two sentence samples as a second label if the order of the two sentence samples in the first training text is different from the order in the first document.

[0201] Optionally, the training module 404 is used to: input the second training text into the language model to obtain the predicted matching degree output by the language model; determine a third loss value based on the predicted matching degree and the matching degree label between the second document and the first input text; and train the language model based on the third loss value.

[0202] Optionally, the acquisition module 401 is configured to: acquire a first input text and the identifier text of a second document associated with the first input text from a database of the target domain to obtain a second training text; calculate the matching degree between the first input text and the second document to obtain a matching degree label.

[0203] Alternatively, the training module 404 can also be used for:

[0204] The third training task is text matching. The language model is trained using third training text, which includes the second input text in the target domain and the identifier text of the third document in the target domain. The matching degree between the first input text and the second document in the second training text is machine-annotated, while the matching degree between the second input text and the third document in the third training text is manually annotated.

[0205] In summary, this application provides a language model training apparatus. This apparatus can integrate high-frequency words from the target domain into a first training text through granular partitioning, and pre-train the language model using the first training text with SOP and MSP tasks as the first training tasks. Therefore, in the pre-training stage, this method enables the language model to learn not only general domain knowledge but also target domain knowledge. Furthermore, this method can also train the language model using a second training text obtained from the target domain with a text matching task as the second training task, thereby enabling the language model to learn a wider range of fundamental knowledge from the target domain. This increases the generalization ability of the language model, thus ensuring the training effect of the model.

[0206] Figure 16 This is a structural block diagram of a text matching device provided in an embodiment of this application. This text matching device can be applied to a computer device, which may be... Figure 1 Terminal 120 in the scenario shown. Figure 16 As shown, the device includes:

[0207] The acquisition module 501 is used to acquire the target input text.

[0208] Matching module 502 is used to input the target input text into the text matching model and obtain the identifier text of the recommended documents that match the target input text, output by the text matching model; wherein, the text matching model adopts, for example, Figure 14 or Figure 15The language model shown was trained using the training device.

[0209] In summary, this application provides a text matching device that can acquire target input text and input it into a text matching model for processing, thereby obtaining the identifier text of recommended documents that match the target input text, output by the text matching model. Since the text matching model is pre-trained using the language model training device provided in the above embodiments and then fine-tuned, the identifier text of recommended documents output by the text matching model is relatively comprehensive and accurate, effectively ensuring the reliability of the text matching results.

[0210] It is understood that the language model training device and text matching device provided in the above embodiments are only illustrative examples of the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0211] Furthermore, the language model training device and language model training method embodiments provided in the above embodiments belong to the same concept, and the text matching device and text matching method embodiments belong to the same concept. For details of their specific implementation process, please refer to the method embodiments, which will not be repeated here.

[0212] Figure 17 This is a schematic diagram of the structure of a server provided in an embodiment of this application. The computer device can be... Figure 1 The server 110 in the scenario shown will be referred to as server 110 in the following description.

[0213] Specifically, server 110 includes a central processing unit (CPU) 111, a system memory 112 including random access memory (RAM) 1121 and read-only memory (ROM) 1122, and a system bus 113 connecting the system memory 112 and the CPU 111. Server 110 also includes a mass storage device 114 for storing the operating system 1141, application programs 1142, and other program modules 1143.

[0214] Mass storage device 114 is connected to central processing unit 111 via a mass storage controller (not shown) connected to system bus 113. Mass storage device 114 and its associated computer-readable media provide non-volatile storage for server 110. That is, mass storage device 114 may include computer-readable media (not shown) such as hard disk or compact disc read-only memory (CD-ROM) drives.

[0215] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The system memory 112 and mass storage device 114 described above can be collectively referred to as memory.

[0216] According to various embodiments of this application, server 110 can also be connected to a remote computer on a network, such as the Internet. That is, server 110 can be connected to network 116 via network interface unit 115 connected to system bus 113, or it can also use network interface unit 115 to connect to other types of networks or remote computer systems (not shown).

[0217] The aforementioned memory also includes one or more programs, which are stored in the memory and configured to be executed by the CPU to implement the language model training method or text matching method provided in the above method embodiments.

[0218] Embodiments of this application also provide a computer device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, code set, or instruction set. The at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to implement the language model training method or text matching method provided in the above-described method embodiments.

[0219] Embodiments of this application also provide a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the language model training method or text matching method provided in the above-described method embodiments.

[0220] Embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the language model training methods or text matching methods described in the above embodiments.

[0221] Optionally, the computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. The random access memory may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM). The sequence numbers of the embodiments in this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0222] It is understood that the term "multiple" in this application means two or more. The terms "first," "second," etc., in this application are used to distinguish identical or similar items with essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor is there any limitation on the quantity or execution order.

[0223] It is also understood that the information and data involved in this application (including but not limited to data used for training the language model and the target input text) have all been authorized by the user or by all parties, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the first training text, the second training text, the third training text and the target input text involved in this application were all obtained with full authorization.

[0224] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0225] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for training a language model, characterized in that, The method comprises: obtaining two sentence samples from a first document in a document library, the document library containing documents of a target field, the two sentence samples being two consecutive sentences in the first document; performing granularity division on each of the sentence samples respectively, the sentence samples after granularity division comprising a plurality of sentence segments, at least one of the sentence segments comprising a plurality of characters, and at least one word belonging to a high-frequency word library composed of the plurality of characters, the high-frequency word library containing high-frequency words of the target field, if a plurality of words can be composed of a plurality of consecutive characters in the sentence sample, the plurality of consecutive characters and the plurality of words belong to one sentence segment, or if a character in the sentence sample cannot form a word with its adjacent character, the character is a sentence segment alone; performing mask processing on at least one of the sentence segments in the two sentence samples after granularity division to obtain a first training text; using the first training text to pre-train a language model as a first-stage training task of a sentence order prediction (SOP) task and a mask segment prediction (MSP) task, the SOP task being used to predict the order of the two sentence samples in the first document, and the MSP task being used to predict at least one masked sentence segment; using a text matching task as a second-stage training task, and using a second training text to train the language model, the second training text comprising a first input text of the target field and an identification text of a second document of the target field, the text matching task being used to predict the matching degree between the second document and the first input text.

2. The method of claim 1, wherein, The mask processing on at least one of the sentence segments in the two sentence samples after granularity division comprises: obtaining at least one of the sentence segments from the two sentence samples after granularity division; for each of the sentence segments, replacing a first sentence component in the sentence segment with a mask, and replacing a second sentence component in the sentence segment with another sentence component; wherein the first sentence component and the second sentence component are single characters or words.

3. The method of claim 1, wherein, The using the first training text to pre-train the language model as the first-stage training task of the SOP task and the MSP task comprises: inputting the first training text into the language model to obtain a predicted order and a predicted sentence segment output by the language model; determining a first loss value according to the predicted order and an order label of the two sentence samples, the order label being used to indicate the order of the two sentence samples in the first document; determining a second loss value according to the predicted sentence segment and the masked sentence segment in the two sentence samples after granularity division; pre-training the language model according to the first loss value and the second loss value.

4. The method of claim 3, wherein, Before the using the first training text to pre-train the language model as the first-stage training task of the SOP task and the MSP task, the method further comprises: if the order of the two sentence samples in the first training text is the same as the order in the first document, setting an order label of the two sentence samples as a first label; if the order of the two sentence samples in the first training text is different from the order in the first document, setting an order label of the two sentence samples as a second label.

5. The method according to any one of claims 1 to 4, characterized in that, The training task with the text matching task as the second stage trains the language model by using a second training text, including: inputting the second training text into the language model to obtain a predicted matching degree output by the language model; determining a third loss value according to the predicted matching degree and a matching degree label of the second document and the first input text; training the language model according to the third loss value.

6. The method of claim 5, wherein, Before the training task with the text matching task as the second stage trains the language model by using a second training text, the method further includes: obtaining the first input text and an identification text of a second document associated with the first input text from a database of the target field to obtain the second training text; calculating a matching degree of the first input text and the second document to obtain the matching degree label.

7. The method according to any one of claims 1 to 4, characterized in that, The method further includes: training the language model by using a third training text with the text matching task as a third stage, the third training text including a second input text of the target field and an identification text of a third document of the target field; wherein the matching degree of the first input text and the second document in the second training text is machine-labeled, and the matching degree of the second input text and the third document in the third training text is manually labeled.

8. A method of text matching, characterized by, The method includes: obtaining a target input text; inputting the target input text into a text matching model to obtain identification text of a recommended document matched with the target input text output by the text matching model; wherein the text matching model is obtained by training a language model by using the method of any one of claims 1 to 7.

9. A language model training apparatus, characterized by comprising: The apparatus includes: an obtaining module configured to obtain two sentence samples from a first document in a document library, the document library containing documents of a target field, the two sentence samples being two consecutive sentences in the first document; a dividing module configured to divide each of the sentence samples in granularity, the sentence samples divided in granularity including a plurality of sentence segments, at least one of the sentence segments including a plurality of characters, and at least one word belonging to a high-frequency word library and composed of the plurality of characters, the high-frequency word library containing high-frequency words of the target field, if a plurality of words can be composed of a plurality of consecutive characters in the sentence sample, the plurality of consecutive characters and the plurality of words belong to one sentence segment, or if a character in the sentence sample cannot form a word with its adjacent character, the character is a sentence segment alone; a mask module configured to mask at least one of the sentence segments in the two sentence samples divided in granularity to obtain a first training text; The training module is configured to perform, as a first-stage training task, a sentence order prediction (SOP) task and a mask segment prediction (MSP) task in a sentence order, and pre-train a language model using first training text, the SOP task being configured to predict an order of the two sentence samples in the first document, and the MSP task being configured to predict at least one sentence segment that is processed by masking. The training module is further configured to perform, as a second-stage training task, a text matching task, and train the language model using second training text, the second training text including first input text of the target domain and identification text of a second document of the target domain, and the text matching task being configured to predict a matching degree between the second document and the first input text.

10. The apparatus of claim 9, wherein, The masking module is configured to: obtain at least one sentence segment from the two sentence samples after granularity division; for each of the sentence segments, replace a first sentence component in the sentence segment with a mask, and replace a second sentence component in the sentence segment with another sentence component; wherein the first sentence component and the second sentence component are each a single character or word.

11. The apparatus of claim 9, wherein, The training module is configured to: input the first training text into the language model to obtain a predicted order and a predicted sentence segment output by the language model; determine a first loss value according to the predicted order and an order label of the two sentence samples, the order label being configured to indicate the order of the two sentence samples in the first document; determine a second loss value according to the predicted sentence segment and a sentence segment that is processed by masking in the two sentence samples after granularity division; pre-train the language model according to the first loss value and the second loss value.

12. The apparatus of claim 11, wherein, The device further includes: an order labeling module configured to set an order label of the two sentence samples as a first label if an order of the two sentence samples in the first training text is the same as an order of the two sentence samples in the first document, and set the order label of the two sentence samples as a second label if the order of the two sentence samples in the first training text is different from the order of the two sentence samples in the first document.

13. The apparatus of any one of claims 9 to 12, wherein, The training module is configured to: input the second training text into the language model to obtain a predicted matching degree output by the language model; determine a third loss value according to the predicted matching degree and a matching degree label of the second document and the first input text; train the language model according to the third loss value.

14. The apparatus of claim 13, wherein, The obtaining module is configured to: obtain the first input text and identification text of the second document associated with the first input text from a database of the target domain to obtain the second training text; calculate a matching degree between the first input text and the second document to obtain the matching degree label.

15. The apparatus of any one of claims 9 to 12, wherein, The training module is further configured to: perform, as a third-stage training task, the text matching task, and train the language model using third training text, the third training text including second input text of the target domain and identification text of a third document of the target domain. The matching degree of the first input text with the second document in the second training text is machine-labeled, and the matching degree of the second input text with the third document in the third training text is manually labeled.

16. A text matching apparatus, characterized by comprising: The device comprises: An acquisition module is configured to acquire a target input text; A matching module is configured to input the target input text into a text matching model to obtain identification text of a recommended document matched with the target input text output by the text matching model. The text matching model is trained by the language model training device of claim 9.

17. A computer device, comprising: The computer device comprises a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the language model training method of any one of claims 1 to 7 or the text matching method of claim 8.

18. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the language model training method of any one of claims 1 to 7 or the text matching method of claim 8.

Images