Language model pre-training method, result recommendation method and related device

Abstract

The present application discloses a language model pre-training method, a result recommendation method and related devices. The method can pre-train the language model with the MLM task and the semantic original prediction task as the first-stage training tasks, so that the language model learns the semantic originals of the words in the training text. After completing the first-stage pre-training, the method can use contrastive learning as the second-stage training task, and use the second training text determined based on the knowledge graph of the target domain to pre-train the language model, thereby integrating the knowledge graph of the target domain into the pre-training process of the language model, so that the language model can learn the knowledge in the knowledge graph of the target domain. Since the method provided by the present application can introduce different training tasks in the pre-training process of the language model, it can ensure that the pre-training effect of the language model is better.

Description

Technical Field

[0001] The present application relates to the field of Natural Language Processing (NLP), and in particular to a language model pre-training method, a result recommendation method, and related devices. Background Art

[0002] In natural language processing, a large amount of unlabeled language text can be used to pre-train the initial model to obtain a language model that is independent of the specific task. This language model can then be fine-tuned based on a specific task (such as reading comprehension or entity recognition) to obtain a target language model for performing the specific task.

[0003] The initial model used in related technologies is generally the Bidirectional Encoder Representations from Transformers (BERT) model. When pre-training this BERT model, the text in the training text is converted into embedding vectors. The BERT model can then be pre-trained using the masked language model (MLM) task.

[0004] However, the above pre-training methods have poor effects. Summary of the Invention

[0005] The present invention provides a language model pre-training method, result recommendation method and related devices, which can effectively improve the pre-training effect of the language model. The technical solution is as follows.

[0006] In one aspect, a method for pre-training a language model is provided, the method comprising:

[0007] Performing masking processing on the first training text, where some words in the first training text after the masking processing are replaced with masks, the words including one or more characters;

[0008] Obtaining an embedding vector corresponding to each character in the first training text after masking, the embedding vector comprising: a character vector, a sememe vector, a segment vector, and a position vector, wherein the sememe vector is a vector representation of the sememe of the character;

[0009] The first phase of training involves a masked language model task and a sememe prediction task, and pre-training the language model based on the embedding vector. The sememe prediction task is used to predict the sememes of words replaced with masks in the first training text, where the sememes of the words are the intersection of the sememes of the various characters included in the word.

[0010] The second stage training task is contrastive learning, and the language model is pre-trained using a second training text. The second training text includes: the original text, a positive sample text obtained by replacing the first word in the original text based on the knowledge graph of the target field, and a negative sample text obtained by replacing the second word in the original text based on the knowledge graph of the target field.

[0011] In another aspect, a result recommendation method is provided, the method comprising:

[0012] Get the search text;

[0013] The search text is input into a search model to obtain a recommendation result output by the search model; wherein the search model is obtained based on a pre-trained language model, and the language model is pre-trained using the method described in the above aspects.

[0014] In another aspect, a language model pre-training device is provided, the device comprising:

[0015] a masking module, configured to perform masking on the first training text, wherein some words in the first training text after the masking are replaced with masks, wherein the words include one or more characters;

[0016] An embedding module, configured to obtain an embedding vector corresponding to each character in the first training text after masking, wherein the embedding vector includes: a character vector, a sememe vector, a segment vector, and a position vector, wherein the sememe vector is a vector representation of the sememe of the character;

[0017] A first training module is configured to use a masked language model task and a sememe prediction task as first-stage training tasks, and to pre-train the language model based on the embedding vector, wherein the sememe prediction task is configured to predict the sememes of words replaced with masks in the first training text, where the sememes of the words are the intersection of the sememes of the various characters included in the words;

[0018] A second training module is used to pre-train the language model using the second-year training text, using contrastive learning as the second-stage training task;

[0019] The second training text includes: the original text, a positive sample text obtained by replacing the first word in the original text based on the knowledge graph of the target field, and a negative sample text obtained by replacing the second word in the original text based on the knowledge graph of the target field.

[0020] In another aspect, a result recommendation device is provided, comprising:

[0021] Acquisition module, used to obtain search text;

[0022] A search module, configured to input the search text into a search model and obtain recommendation results output by the search model;

[0023] The search model is trained using the language model pre-training device described in the above aspects.

[0024] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the language model pre-training method as described in the above aspects, or the result recommendation method as described in the above aspects.

[0025] On the other hand, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the language model pre-training method as described in the above aspects, or the result recommendation method as described in the above aspects.

[0026] In another aspect, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the language model pre-training method or the result recommendation method described in the above aspects.

[0027] The beneficial effects of the technical solution provided by this application include at least:

[0028] The present application discloses a language model pre-training method, a result recommendation method and related devices. The method can pre-train the language model with the MLM task and the semantic original prediction task as the first-stage training tasks, so that the language model learns the semantic originals of the words in the training text. After completing the first-stage pre-training, the method can use contrastive learning as the second-stage training task, and use the second training text determined based on the knowledge graph of the target domain to pre-train the language model, thereby integrating the knowledge graph of the target domain into the pre-training process of the language model, so that the language model can learn the knowledge in the knowledge graph of the target domain. Since the method provided by the present application can introduce different training tasks in the pre-training process of the language model, it can ensure that the pre-training effect of the language model is better. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0030] Figure 1 Schematic diagram of the structure of a language model pre-training system provided in an embodiment of the present application;

[0031] Figure 2 This is a flowchart of a language model pre-training method provided in an embodiment of the present application;

[0032] Figure 3 This is a flowchart of another language model pre-training method provided in an embodiment of the present application;

[0033] Figure 4 This is a schematic diagram of a server determining a semantic matrix of a character provided by an embodiment of the present application;

[0034] Figure 5 This is a schematic diagram of the structure of a language model provided in an embodiment of the present application;

[0035] Figure 6 This is a schematic diagram of generating positive sample text and negative sample text provided by an embodiment of the present application;

[0036] Figure 7 This is a flowchart of a result recommendation method provided in an embodiment of the present application;

[0037] Figure 8 This is a structural block diagram of a language model pre-training device provided in an embodiment of the present application;

[0038] Figure 9This is a structural block diagram of another language model pre-training device provided in an embodiment of the present application;

[0039] Figure 10 This is a structural block diagram of a result recommendation device provided in an embodiment of the present application;

[0040] Figure 11 This is a structural block diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0042] The solution provided in the embodiments of this application can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, etc. First, a brief introduction is given to the terms involved in the embodiments of this application.

[0043] Artificial Intelligence (AI) is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0044] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0045] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0046] Machine Learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0047] Natural language processing (NLP) is a key area of research in computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. Natural language processing integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language we use in everyday life—and is closely linked to linguistics. NLP technologies typically include text processing, semantic understanding, machine translation, robotic question answering, and knowledge graphs.

[0048] Language Model (LM): A model used in the field of NLP to analyze and process language texts. It can generally be divided into grammar rule language models, statistical language models, and neural network language models.

[0049] A knowledge graph (KG) is a semantic network consisting of interconnected entity concepts, consisting of nodes (entities / attribute values) and edges (relationships / attributes). Entity concepts in a knowledge graph are stored as triples, such as (entity, relationship, entity) or (attribute value, attribute, attribute value). Entities are the most basic elements in a knowledge graph, and different entities have different relationships. Attributes primarily refer to the characteristics, properties, features, and parameters that an object may have, such as nationality or birthday. Attribute values primarily refer to the values of a specified attribute of an object, such as China or 1988-09-08.

[0050] Traditional language model pre-training methods use various unsupervised training methods to pre-train language models using a large amount of general knowledge from open domains. This allows the pre-trained language models to learn some general knowledge. However, this pre-training method does not enable the pre-trained language models to learn knowledge from specific domains (such as finance) and thus handle tasks in those domains.

[0051] In the related art, the knowledge graph of a specific field can be integrated into the pre-training process of the language model, so that the language model can learn the knowledge of the specific field during the pre-training process. Correspondingly, the pre-trained language model can also better handle the tasks of the specific field after fine-tuning. Among them, the knowledge graph of a specific field can be integrated into the pre-training process of the language model through an embedding-based method and a pre-training task-based method. The embedding-based method is to convert the entity description of the entity in the original sentence, or the entity triple corresponding to the entity in the original sentence in the knowledge graph of a specific field into text and add it to the original sentence, and use the original sentence with the added entity description or entity triple to pre-train the language model, thereby completing the integration of the knowledge graph of the specific field. The pre-training task-based method is to introduce different pre-training tasks into the pre-training process of the language model, so that the language model can learn the knowledge of the specific field during the pre-training process.

[0052] For embedding-based methods, the related art generally adopts the BERT (K-BERT) pre-training method based on the knowledge graph. The K-BERT pre-training method includes: for some words (i.e., entities) in the original sentence used for pre-training, the triple knowledge corresponding to the word in the knowledge graph is added to the original sentence to expand the original sentence into a sentence tree. Then, the embedding layer of the BERT model can first convert the sentence tree of the original sentence into a one-dimensional sequence, and then convert the one-dimensional sequence into an embedding vector and input it into the encoding layer of the BERT model.

[0053] Among them, the process of injecting triples into the original sentence can include two steps: knowledge query and knowledge injection. In knowledge query, it is necessary to query the triples corresponding to some entities in the original sentence from the knowledge graph. Knowledge injection generates a sentence tree by adding triples to the positions corresponding to the entities in the original sentence. Since there may be multiple triples corresponding to the entities in the original sentence, in order to avoid the sentence tree of the original sentence being too large, for each entity in the original sentence, only one triple among the multiple triples corresponding to it will be selected to generate the sentence tree. In the process of converting the sentence tree of the original sentence into a one-dimensional sequence, the embedding layer of the K-BERT model will insert the triples corresponding to each entity in the original sentence after the entity. Correspondingly, the content after the entity in the original sentence will move backward in sequence.

[0054] However, the above method only focuses on the conceptual information corresponding to the entity in the original sentence in the knowledge graph, and pays less attention to the attributes (or categories) of the entity. Moreover, when the entity in the original sentence corresponds to multiple triples, only one triple can be selected to generate the sentence tree, resulting in the loss of information about the other triples of the entity in the knowledge graph. In addition, in the process of converting the sentence tree of the original sentence into a one-dimensional sequence, the length of the original sentence changes significantly, and the additional triples inserted also destroy the structure of the original sentence.

[0055] Based on the above analysis, it can be seen that the pre-training effect of the pre-training method based on the K-BERT model is poor.

[0056] Figure 1 This is a structural diagram of a language model pre-training system provided in an embodiment of the present application. Figure 1 The system includes: a server 110 and a terminal 120. A wired or wireless communication connection is established between the server 110 and the terminal 120. Optionally, the server 110 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal 120 can be a personal computer (PC), an in-vehicle terminal, a tablet computer, a smart phone, a wearable device, an intelligent robot, or other terminal with data computing, processing, and storage capabilities.

[0057] In an embodiment of the present application, the terminal 120 in the system can be used to obtain training texts (e.g., first training texts and second training texts) and send the training texts to the server 110. The server 110 can then use the training texts to pre-train a language model. The server can then fine-tune the pre-trained language model to obtain a language model capable of performing specific tasks (e.g., a search model recommended based on search text feedback results).

[0058] Optionally, the training text used for pre-training the language model may also be pre-stored in the server 110. Accordingly, the language model pre-training system may also not include the terminal 120.

[0059] Alternatively, the system can also perform specific tasks, such as result recommendation tasks. Accordingly, the terminal 120 in the system can be used to obtain input text (such as search text) and send the input text to the server 110 for search. The server 110 pre-stores a language model that has been pre-trained and fine-tuned, such as a search model. After the server obtains the input text, it can input the input text into the language model, which can then process the input text and output the result text. For example, the result text output by the language model can be a recommendation result for the search text. Afterwards, the server 110 can send the result text to the terminal 120.

[0060] Optionally, the terminal 120 may also store a pre-trained and fine-tuned language model. After receiving input text, the terminal 120 may directly input the input text into the language model, which may then process the input text and output a result text. Accordingly, the language model pre-training system may also not include the server 110.

[0061] Combined with the above-mentioned noun introduction and application scenarios, the pre-training method of the language model provided in the embodiment of the present application is described. The method can be applied to a computer device, which can be Figure 1 The server 110 in the scenario shown is shown in FIG. Figure 2 As shown, the method includes the following steps.

[0062] Step 101: Perform mask processing on the first training text.

[0063] In an embodiment of the present application, the server can obtain multiple first training texts for pre-training and input each first training text into the language model for masking. Part of the words in the masked first training text are replaced with a mask [MASK], and the words replaced with the mask [MASK] can include one or more characters. It is understood that masking the first training text is the beginning of pre-training the language model.

[0064] Optionally, the server can randomly replace a certain percentage of words (e.g., 15% of the words) in each first training text with a mask [MASK]. Furthermore, the words replaced with the mask [MASK] in the first training text are the words that the language model needs to predict during pre-training. Optionally, the server can obtain the multiple first training texts from a corpus, which can include: a knowledge graph of the target domain, an encyclopedia knowledge base, relevant forums, social platforms, and journal websites.

[0065] Step 102: Obtain embedding vectors corresponding to the characters in the first training text after masking.

[0066] In the embodiment of the present application, for each character in the first training text, the server can obtain the embedding vector corresponding to the character. The embedding vector corresponding to each character includes: a character vector, a semantic vector, a segment vector, and a position vector.

[0067] It can be understood that the word (Token) vector E in the embedding vector corresponding to each word tok Represents the vector corresponding to the word itself. The Sememe vector E in the embedding vector corresponding to each word sem It is the vector representation of the semantic primitive (i.e. category or attribute) of the word. The position vector E in the embedding vector corresponding to each word pos It is a vector representation of the order of the characters in the training text. The segment vector E in the embedding vector corresponding to each character seg It is a vector representation used to assist the language model in distinguishing between two training texts. If there is only one training text input to the language model, then the mid-segment vector E corresponding to each word in the training text is seg The values are all 0.

[0068] It can also be understood that the first training text in the embodiment of the present application may be a Chinese text, and accordingly, the text in the first training text may refer to Chinese characters.

[0069] Optionally, the embedding vector E corresponding to each character in the first training text is c It can be expressed as: E c =E tok +E sem +E seg +E pos That is, the embedding vector E c It can be the word vector E for the text tok , semantic vector E sem , segment vector E seg and position vector E pos Obtained by splicing.

[0070] Step 103: Using the masked language model task and the sememe prediction task as the first-stage training tasks, pre-train the language model based on the embedding vector.

[0071] The masked language model task is used to predict words replaced with the mask [MASK] in the first training text. The sememe prediction task is used to predict the sememes of words replaced with the mask in the first training text. The sememe of a word is the intersection of the sememes of the various characters included in the word. In other words, the sememe prediction task can predict the sememe of each character replaced with the mask and determine the intersection of the sememes of the various characters included in the word replaced with the mask as the sememe of the word.

[0072] Step 104: Using contrastive learning as the training task of the second stage, the language model is pre-trained using the second training text.

[0073] The second training text includes: the original text, a positive sample text obtained by replacing the first word in the original text based on the knowledge graph of the target domain, and a negative sample text obtained by replacing the second word in the original text based on the knowledge graph of the target domain. After completing the first phase of training, the server can use contrastive learning (CL) as the second phase of training for the language model, using the second training text to pre-train the language model.

[0074] It is understandable that since both the positive and negative sample texts in the second training text are obtained based on the knowledge graph of the target domain, using the second training text to pre-train the language model enables the language model to learn the knowledge of the target domain. In other words, the embodiment of the present application can generate the second training text based on the knowledge graph of the target domain to integrate the knowledge graph of the target domain into the pre-training process of the language model.

[0075] In summary, an embodiment of the present application provides a method for pre-training a language model. The method can pre-train the language model with the MLM task and the semantic original prediction task as the training tasks of the first stage, so that the language model learns the semantic originals of the words in the training text. After completing the pre-training of the first stage, the method can use contrastive learning as the training task of the second stage, and use the second training text determined based on the knowledge graph of the target domain to pre-train the language model, thereby integrating the knowledge graph of the target domain into the pre-training process of the language model, so that the language model can learn the knowledge in the knowledge graph of the target domain. Since the method provided by the embodiment of the present application can introduce different training tasks in the pre-training process of the language model, it can ensure that the pre-training effect of the language model is better.

[0076] Figure 3 This is a flow chart of another language model pre-training method provided in an embodiment of the present application. This embodiment uses the method applied to a server as an example. Figure 3 , the method comprising:

[0077] Step 201: Obtain the semantic origin of each reference word in the reference word library.

[0078] In the embodiment of the present application, a reference word library is pre-stored in the server. The reference word library can be obtained based on the knowledge graph of the target domain and the Chinese concept knowledge base (such as HowNet). The reference word library can include multiple reference words and the semantic origins of the reference words.

[0079] It is understood that for a reference term in a knowledge graph derived from a target domain, the reference term may be an entity in the knowledge graph of the target domain, and the semantic primitive of the reference term may be the attribute (also known as a category) corresponding to the entity in the knowledge graph of the target domain. For a reference term derived from a Chinese concept knowledge base, the reference term is annotated with semantic information (i.e., semantic primitive). Furthermore, the semantic information of the reference term is the smallest semantic unit that describes the reference term.

[0080] Step 202: Based on the characters included in each reference word in the reference word library and the semantic origin of each reference word, obtain the semantic origin of each character in the reference word library.

[0081] For each character included in each reference term in the reference term library, the server can retrieve each reference term containing the character from the reference term library. The server can then determine the sememe of the character by taking the union of the sememes of each reference term. In other words, the server can perform an OR operation on the sememes of each reference term to obtain the sememe of each character in the reference term library.

[0082] Step 203: Generate a character sememe matrix based on the sememes of each character in the reference word library.

[0083] After determining the sememe of each character in the reference vocabulary, the server can obtain a character sememe matrix, which includes the sememe vector for each character in the reference vocabulary. In this embodiment of the present application, the character sememe matrix can be a matrix with Dw rows and Ds columns. Where Dw and Ds are both positive integers greater than 0, Dw represents the number of characters included in the reference vocabulary, and Ds represents the total number of sememes included in each character in the reference vocabulary.

[0084] The values of the elements in the character sememe matrix can be 0 or 1, and the one-dimensional vector of length Ds consisting of the Ds elements in the i-th row is the sememe vector of the i-th character. Furthermore, if the value of the element in the i-th row, j-th column is 1, it means that the i-th character has the j-th sememe. If the value of the element in the i-th row, j-th column is 0, it means that the i-th character does not have the j-th sememe.

[0085] For example, Figure 4 As shown in , it is assumed that the reference word library includes 3 reference words: w0, w1 and w2, and the reference word library includes 4 semantic primitives: s1, s2, s3 and s4. Figure 4 As shown in the figure, a black circle indicates that a word or character has the sememe, and the corresponding element value in the sememe matrix can be 1. A white circle indicates that a word or character does not have the sememe, and the corresponding element value in the sememe matrix can be 0. Figure 4 , word w0 consists of 3 characters c0, c1, c2, and the semantic origin of word w0 is {s1}. Word w1 consists of 2 characters c0, c2, and word w1 has the semantic origin {s2, s4}. Word w2 consists of 2 characters c1, c3, and word w2 has the semantic origin {s3}.

[0086] For a character c0, the server can obtain the words w0 and w1 that contain the character c0, and determine the sememe {s1, s2, s4} of the character c0 as the union of the sememe {s1} of the word w0 and the sememe {s1, s4} of the word w1. Similarly, the server can determine the sememe {s1, s3} of the character c1. The sememe {s1, s2, s4} of the character c2 is {s1, s2, s4}. The sememe {s3} of the character c3 is

[0087] Assume that in the semantic matrix, the four semantic elements are arranged in the order of s1, s2, s3 and s4. Figure 4 It can be seen that the semantic vector of the text c0 can be [1 1 0 1], the semantic vector of the text c1 can be [1 0 1 0], the semantic vector of the text c2 can be [1 1 0 1], and the semantic vector of the text c3 can be [0 0 1 0].

[0088] Step 204: Perform mask processing on the first training text.

[0089] In an embodiment of the present application, the server can obtain multiple first training texts for pre-training, and can input each first training text into the language model for masking. Part of the words in the masked first training text are replaced with a mask [MASK], and the words replaced with a mask [MASK] may include one or more characters. The words replaced with a mask [MASK] may be entities that exist in the knowledge graph of the target domain or words that are not entities in the knowledge graph of the target domain.

[0090] Optionally, the server can randomly replace a certain percentage of words (e.g., 15% of the words) in each first training text with a mask [MASK]. Furthermore, the words replaced with the mask [MASK] in the first training text are the words that the language model needs to predict during pre-training. Optionally, the server can obtain the multiple first training texts from a corpus, which can include: a knowledge graph of the target domain, as well as encyclopedias, relevant forums, social platforms, journal websites, etc.

[0091] For example, suppose the first training text is "Hummingbird is the smallest bird in the world." The server can replace the two characters in the entity "Hummingbird" in the first training text with the mask [MASK]. Accordingly, after masking, the first training text can be represented as "[MASK][MASK] is the smallest bird in the world." Alternatively, the server can replace the word "small" in the first training text with the mask [MASK]. After masking, the first training text can be represented as "Hummingbird is the smallest [MASK] bird in the world."

[0092] Step 205: Based on the character sememe matrix, obtain the sememe vectors of the characters in the first training text after masking.

[0093] After performing masking on the characters in the first training text, the server can obtain the sememe of each character in the first training text from a predetermined character sememe matrix, thereby determining the sememe vector of each character in the first training text.

[0094] In the embodiment of the present application, the first training text can be represented as X=(c0, c1, ..., c n ), c i is the i-th character included in the first training text X, and the number of characters included in the first training text is n+1. Where n is a positive integer greater than or equal to 1, i is an integer not greater than n, and That is, the text c in the first training text i Belongs to the reference word library

[0095] For the semantic vector E of each word in the first training text X sem , we can use two-layer mapping E sem =SMX. That is, M represents the mapping from the characters in the reference vocabulary to the semantic primitives of the characters (i.e., the union of the semantic primitives of the various words containing the characters). Assume that the total number of characters included in the reference vocabulary is D w , the length of the semantic origin of each word is D SOne-dimensional vector representation (i.e., the total number of semantic primitives of all characters in the reference word library is D S ), then this M can be D w Dimension to D S Dimension mapping. Represents the mapping from the semantic primitive of the character (i.e., D S Dimension) to the encoding dimension (i.e., d dimension) of the encoding layer of the language model, where d is the length of the input vector of the encoding layer.

[0096] For example, assume that the first training text after masking is "[MASK] is the smallest bird in the world", then the server can obtain the semantic primitive vectors of "is", "world", "on", "body", "type", "most", "small", "bird", respectively.

[0097] Step 206: Obtain the embedding vector corresponding to the character in the first training text after masking.

[0098] In the embodiment of the present application, for each character in the first training text, the server can obtain the embedding vector corresponding to the character. Among them, the embedding vector corresponding to each character includes: character vector, semantic primitive vector, segment vector, and position vector.

[0099] It can be understood that the character vector E in the embedding vector corresponding to each character tok Represents the vector corresponding to the character itself. The semantic primitive vector E in the embedding vector corresponding to each character sem Is the vector representation of the semantic primitive (i.e., category or attribute) of the character. Among them, the server can determine the semantic primitive vector E of each character based on the pre-determined character semantic primitive matrix sem . The position vector E in the embedding vector corresponding to each character pos Is the vector representation of the arrangement order of the character in the training text. The segment vector E in the embedding vector corresponding to each character seg Is the vector representation used to assist the language model to distinguish the two training texts in the training text pair. If there is only one training text input to the language model, the value of the segment vector E in the embedding vector corresponding to each character in this training text seg Is 0. [[ID=�6]]

[0100] It can also be understood that the first training text in the embodiment of the present application can be a Chinese text. Correspondingly, the characters in this first training text can refer to Chinese characters.

[0101] Optionally, the embedding vector E corresponding to each character in the first training text c Can be expressed as: E c = E tok + E sem + E seg+E pos That is, the embedding vector E c It can be the word vector E for the text tok , semantic vector E sem , segment vector E seg and position vector E pos Obtained by splicing.

[0102] Step 207: Input the embedding vector into the language model to obtain the predicted word and the predicted semantic primitive of the predicted word output by the language model.

[0103] The server can embed the corresponding vector E of each character in the first training text c Input to the language model. The embedding vector E c After sequentially passing through the embedding layer, encoding layer, and linear layer of the language model, the predicted words for the words replaced by the mask in the first training text and the predicted semantic primitives of the words replaced by the mask can be output.

[0104] Optionally, the embodiment of the present application may adopt a BERT model as the initial language model and pre-train it using the first training text.

[0105] Step 208: Determine a first loss value based on the predicted words and the words replaced with masks in the first training text.

[0106] For each first training text, the server may determine a first loss value of the language model in the MLM task based on the predicted words output by the language model for the first training text and the words replaced with masks in the first training text.

[0107] The first loss value can be calculated by the loss function of the MLM task. MLM It can be expressed as:

[0108] L MLM = -logp(c|context);

[0109] Here, c represents the text that the language model needs to predict, and context represents the embedded representation of all words in the first training text, excluding the words replaced by the mask. p(c│context) represents the probability that the predicted text output by the language model is the text replaced by the mask in the first training text. It is understood that for the words replaced by the mask in the first training text, the language model predicts each word as a unit. After the language model predicts each masked word in the first training text, it can group the predicted words that are consecutive in position into a predicted word.

[0110] Step 209: Determine a second loss value based on the predicted sememe of the predicted word and the sememe of the word replaced by the mask in the first training text.

[0111] After step 207, for each first training text, the server may determine a second loss value of the loss function of the language model in the semantic origin prediction task based on the predicted semantic origin of the predicted word output by the language model and the semantic origin of the word replaced by the mask in the first training text.

[0112] like Figure 5 As shown, during the pre-training process, after the embedding vector of the first training text is processed by the language model's embedding layer, encoding layer, and linear layer, the language model can output a predicted sememe vector for the characters replaced by the mask. Based on the predicted sememe vector, the server can determine the predicted sememe for each character replaced by the mask. Subsequently, the server can determine the intersection of the predicted sememes of the characters included in the predicted word as the predicted sememe for the predicted word. That is, the server can perform an AND operation on the predicted sememes of the characters included in the predicted word to obtain the predicted sememe for the predicted word.

[0113] For example, for a first training text T = {w0, w1, w2, w3}, after the server replaces the words w0 and w3 in the first training text T with masks, the sememe vector of each word in the first training text T can be determined based on the sememe matrix. Figure 5 The figure schematically illustrates the semantic vectors of each character in words w0 and w3 replaced by the mask [MASK]. After the semantic vectors in the embedding vectors of each character in the first training text T are processed by the embedding layer, encoding layer, and linear layer of the language model, the linear layer of the language model can output the predicted semantic vectors for each character in the words replaced by the mask in the first training text T.

[0114] Continue to refer Figure 5 The linear layer of the language model can output a predicted sememe vector for each character from c0 to c4. This predicted sememe vector is a one-dimensional vector of length Ds. The value of the jth element in the predicted sememe vector represents the probability that the character has the jth sememe, and the value range of this probability is [0, 1].

[0115] For example, Figure 5 The grayscale of the circle's fill color represents the probability. The darker the circle, the closer the element (i.e., probability) corresponding to the circle is to 1, and the greater the probability that the text has the corresponding sememe. The lighter the circle, the closer the element (i.e., probability) corresponding to the circle is to 0, and the smaller the probability that the text has the corresponding sememe.

[0116] For each character's predicted sememe vector output by the language model embedding layer, the server can determine the sememes indicated by elements in the predicted sememe vector with a probability greater than a probability threshold (e.g., 0.5) as the sememes possessed by the character. Correspondingly, for sememes indicated by elements in the predicted sememe vector with a probability not greater than the probability threshold, the server can determine that the character does not possess the sememe. In this way, the server can determine the predicted sememe for each character.

[0117] Example, reference Figure 5 , assuming that the predicted semantic vector of the text c0 is [1 0.8 0.2 0.9], the server can determine that the predicted semantic vector of the text c0 is {s1, s2, s4}.

[0118] Furthermore, the server may determine the intersection of the predicted sememes of various characters belonging to the same predicted word as the predicted sememe of the predicted word.

[0119] After the server determines the predicted sememe of the predicted word replaced by the mask [MASK] in the first training text, it can determine a second loss value of the language model in the sememe prediction task based on the predicted sememe of the predicted word and the true sememe of the predicted word.

[0120] The second loss value can be calculated by the loss function of the sememe prediction task, where the loss function L CSP It can be expressed as:

[0121]

[0122] Where s represents the predicted sememe of the predicted word that the language model is trying to predict, and w represents the predicted word containing multiple characters c. p(s│context) represents the probability that the predicted sememe of the predicted word output by the language model is the true sememe of the masked word in the first training text, that is, the probability that the language model correctly predicts the sememe of the masked word. PS(c) represents the set of all possible semes of character c.

[0123] In the above formula, The intersection of the sememes of each character c in the predicted word w, i.e., the set of true sememes of the predicted word w, is hereinafter referred to as the target set. It is understood that for a predicted word w, the sememe of the predicted word w can be determined by taking the intersection of the sememes of each character c in the predicted word w. This indicates that the predicted sememe s of the predicted word w output by the language model belongs to the target set. It is understood that if the predicted sememe s of the predicted word w output by the language model belongs to the target set, then the predicted sememe s can be determined as the correct sememe of the predicted word w. The server can then perform a summation process on the correct sememes of the predicted word w.

[0124] In the embodiment of the present application, since the number of semantic primitives of each word is relatively unevenly distributed, in order to avoid overfitting of the language model during the pre-training process, a regularization term can be added to the loss function of the semantic primitive prediction task. The regularization term can be expressed as:

[0125]

[0126] in, It indicates the number of semantic primitives of the predicted word w.

[0127] Optionally, in the embodiment of the present application, the sememe prediction task may also be referred to as a Chinese word sememe prediction (CSP) task.

[0128] Step 210: Pre-train a language model according to the first loss value and the second loss value.

[0129] In an embodiment of the present application, the server may determine the first loss value determined by the language model in the MLM task and the second loss value determined in the semantic original prediction task as the overall loss value of the language model in the first stage. Furthermore, the server may pre-train the language model based on the overall loss value of the first stage. Accordingly, the loss function L of the first stage can be expressed as: L = L MLM +L CSP .

[0130] Step 211: Obtain the original text.

[0131] In an embodiment of the present application, the server can identify sentences in the corpus that contain entities in the knowledge graph of the target domain based on the target domain to which the pre-trained language model is to be applied, and use the sentences as the original text. It is understandable that some technical fields will construct relevant knowledge in the field into a knowledge graph, using a visual knowledge graph to describe and display the knowledge in the field and the relationships between the various entities included in the knowledge.

[0132] Step 212: Based on the knowledge graph of the target domain, perform synonym replacement on the first word in the original text to obtain a positive sample text.

[0133] After the server obtains the original text based on the corpus and the target domain knowledge graph, it can use the entity linking tool to link the entities in the original text that exist in the target domain to the target domain knowledge graph. Based on this entity linking tool, the server can obtain the entity relationship corresponding to each entity in the original text in the target domain knowledge graph and use the words in the target domain knowledge graph that have a synonymous relationship with the first word in the original text to synonymously replace the first word to obtain positive sample text.

[0134] It is understood that the relationships between entities in the target domain knowledge graph can be semantic relationships, such as synonymy, hyponymy, and specific relationships between entities. By performing synonymous substitution on the first word in the original text and using the replaced sentences as positive sample text, the language model can better learn the semantic relationships between the various words in the original text during pre-training. In other words, it can better learn the relationships between the various entities in the target domain knowledge graph, thereby better learning the knowledge of the target domain.

[0135] Example, reference Figure 6 For the original text S1, "Zhang San applied to the Academy of Fine Arts because he likes painting," in the target domain's knowledge graph, "painting" and "drawing" are synonymous. Therefore, the server can replace the first word "painting" in the original text with "drawing," obtaining the positive sample text P1, "Zhang San applied to the Academy of Fine Arts because he likes painting."

[0136] Step 213: Based on the knowledge graph of the target domain, perform non-synonymous replacement on the second word in the original text to obtain a negative sample text.

[0137] In an embodiment of the present application, after the server obtains the entity relationship corresponding to each entity in the original text in the target domain knowledge graph through the entity linking tool, it can use a word in the target domain knowledge graph that has a non-synonymous relationship with the second word in the original text to perform a non-synonymous replacement on the second word to obtain a negative sample text.

[0138] Among them, non-synonymous replacement can refer to other replacement methods besides synonymous replacement. For example, the non-synonymous replacement can include at least one of: superordinate replacement, subordinate replacement and relational replacement. Among them, superordinate replacement refers to replacing the second word with the superordinate (i.e., hypernym) word of the second word, subordinate replacement refers to replacing the second word with the subordinate (i.e., hyponym) word of the second word, and relational replacement refers to replacing one of the entities based on the relationship between the two entities when there are two or more entities in the original text.

[0139] For example, suppose the original text includes entity A and entity B, and the relationship between entity A and entity B is R. If the relationship R between entity A and entity B is unique, the server can replace entity A with entity C in the knowledge graph that has the same category (i.e., attribute) as entity A, thereby obtaining a negative sample text. If the relationship R between entity A and entity B is not unique, the server can replace entity A with entity D in the knowledge graph that also has a relationship R with entity B, thereby obtaining a negative sample text.

[0140] For example, continue to refer to Figure 6 For the original text S1, "Zhang San applied to the Academy of Fine Arts because he likes painting," if the server performs non-synonymous replacement on the original text S1 based on the knowledge graph of the target domain, it can replace the entity (i.e., the name) "Zhang San" in the original text S1 with another entity "Li Si" with the same attributes as the entity, resulting in a negative sample text N1, "Li Si applied to the Academy of Fine Arts because he likes painting." Alternatively, the server can replace another entity (i.e., the school) "Academy of Fine Arts" in the original text S1 with its superordinate entity "Professional School," resulting in a negative sample text, "Zhang San applied to the Professional School because he likes painting."

[0141] Step 214: Input the second training text into the language model to obtain the first predicted similarity and the second predicted similarity output by the language model.

[0142] In an embodiment of the present application, the server can splice an original text, a positive sample text and a negative sample text determined based on the original text to obtain a second training text, and input the second training text into the language model for training. For the second training text input into the language model, the language model can calculate and output a first predicted similarity between the original text and the positive sample text in the second training text. In addition, the language model can calculate and output a second predicted similarity between the original text and the negative sample text in the second training text. Optionally, the language model can use cosine similarity, Euclidean distance or Gaussian distance to calculate the first predicted similarity and the second predicted similarity.

[0143] It is understood that for each original text, the server can generate at least one positive sample text and at least one negative sample text. Accordingly, the server can obtain one or more second training texts based on the original text, the at least one positive sample text, and the at least one negative sample text. Furthermore, the original texts in the multiple second training texts generated by the server based on one original text are the same, and the positive sample texts and / or negative sample texts are different.

[0144] Step 215: Determine a third loss value according to the first prediction similarity and the second prediction similarity.

[0145] During the pre-training of the language model, the server may determine a third loss value of the language model in the training phase based on the first prediction similarity and the second prediction similarity output by the language model.

[0146] It is understandable that, for the multiple second training texts obtained, the server can divide the multiple second training texts into multiple batches (Batch) for training. The size of each batch (Batch Size) is N, that is, the number of second training texts included in the batch N, where N is a positive integer greater than 1. In the process of pre-training the language model, for a second training text, the server can calculate the third loss value corresponding to the second training text based on the first predicted similarity and the second predicted similarity within a batch to which the second training text belongs.

[0147] It is also understood that within a training batch, to improve the accuracy of the language model during pre-training, for each of the multiple original texts obtained by the server, other original texts from the multiple original texts, excluding the original text, can be used as negative sample texts for the original text to obtain multiple second training texts corresponding to the original text. Accordingly, pre-training a language model based on a larger number of second training texts can effectively improve the accuracy of the language model and ensure the pre-training effect of the language model.

[0148] The third loss value can be calculated by the loss function of the CL task. When the CL task pre-trains the i-th second training text, the cross entropy loss function can be used as the loss function of the CL task. The cross entropy loss function L i It can be expressed as:

[0149]

[0150] Among them, h i represents the original text in the i-th second training text, h j Represents the original text in the j-th second training text. Represents the original text h i The positive sample text, Represents the original text h i Negative sample text. Represents the original text h i With the original text h i Positive sample text The similarity of is the first prediction similarity, and τ represents the temperature hyperparameter. Represents the original text h i With the original text h j Positive sample text similarity. Represents the original text h i With the original text h j Negative sample text When pre-training a language model with batches as the training cycle, since negative sample texts are introduced in the contrastive learning phase, and the above loss function also introduces the second predicted similarity determined based on the negative sample texts, the above loss function can also be called a cross-entropy loss function with in-batch negative examples.

[0151] For each original text h i , the original text h output by the language model during pre-training i and positive sample text The greater the corresponding first prediction similarity, the greater the third loss value L i The smaller it is, the higher the accuracy of the language model during training and the better the training effect.

[0152] Step 216: Pre-train the language model according to the third loss value.

[0153] After determining the third loss value for each second training text, the server may pre-train the language model based on the third loss value. During the pre-training of the language model, the server may adjust relevant parameters of the language model based on the third loss value corresponding to the second training text to optimize the accuracy of the language model.

[0154] Optionally, the server can use the loss function L in the second stage of the language model i When convergence occurs, or when the accuracy of the language model reaches a preset accuracy, or when the number of training rounds of the language model in the second training phase reaches a preset number, the second phase pre-training is stopped. Furthermore, in step 210, the server may stop the first phase pre-training and start the second phase pre-training when the loss function L of the first phase converges, or when the accuracy of the language model reaches a preset accuracy, or when the number of training rounds of the language model in the first training phase reaches a preset number.

[0155] In an embodiment of the present application, the language model obtained by pre-training the language model provided by the embodiment of the present application can be called a domain-adaptive language model pre-training framework with knowledge graph (K-DLM).

[0156] Tables 1, 2, and 3 respectively show the performance of the language model (i.e., K-DLM) obtained after pre-training using an embodiment of the present application and the language model in the related art on multiple public Chinese natural language datasets.

[0157] Table 1 shows the performance of each model on the corpus of five open fields. Among them, the open corpus can include a book review corpus, a shopping platform corpus, a cross-language evaluation dataset (Cross-Lingual Natural Language Inference, XNLI), a large-scale Chinese question matching corpus (Large-scale Chinese Question Matching Corpus, LCQMC) and a named entity recognition (Microsoft Research-Named Entity Recognition, MSRA-NER) corpus released by Microsoft. Among them, Google's BERT (Google BERT), K-BERT, and robustly optimized BERT (RoBERT) can be used as reference models for the language model (i.e., K-DLM) provided in the embodiment of the present application.

[0158] Optionally, the performance of the models on the five corpora can be characterized by comparing their accuracy (Acc) or the comprehensive evaluation metric F1 (F-Measure) on the validation set (Dev) and test set (Test). The book review corpus, shopping platform corpus, XNLI, and LCQMC are compared based on accuracy, while MSRA-NER is compared based on F1.

[0159] Table 1

[0160]

[0161] As shown in Table 1, for the five corpora mentioned above, the K-DLM trained using the pre-training method provided in the present embodiment achieves higher accuracy and F1 scores on both the validation and test sets than the reference model. This means that the language model trained using the pre-training method provided in the present embodiment outperforms existing language model pre-training methods.

[0162] Table 2 shows the performance results of different models on the classification task based on the Chinese Language Understanding Evaluation (CLUE). Among them, the classification task may include: Toutiao Chinese news short text classification task TNEWS, Chinese model-based natural language inference (CMNLI) task, CLUB-based international cognitive intelligence test 2020 version (CLUB Winograd Schema Challenge 2020, CLUEWSC2020) and Chinese Scientific Literature (CSL) task. Among them, Google BERT, Chinese version RoBERT (ChineseRoBERT) and RoBERT can be used as reference models of the language model (K-DLM) provided in the embodiment of the present application, and the performance of the multiple models on each classification task can be characterized by the accuracy (%) of the model.

[0163] Table 2

[0164] Model TNEWS CMNLI CLUEWSC2020 CSL Google BERT 56.09 79.47 59.60 79.63 Chinese Robert 57.51 80.70 67.20 80.67 RoBERT 57.26 80.31 80.92 81.07 K-DLM 57.76 81.21 85.20 81.30

[0165] Referring to Table 2, we can see that the accuracy of K-DLM in the above four classification tasks is higher than that of the reference model.

[0166] Table 3 shows the comparison results of the accuracy (Precision), recall (Recall) and comprehensive evaluation index F1 value of each model when performing tasks in a specific field. Among them, Google BERT, K-BERT obtained based on HowNet pre-training, K-BERT obtained based on assisted knowledge graph (Service-KG) pre-training and RoBERT can be used as reference models of the language model (K-DLM) provided in the embodiment of this application. The tasks in this specific field may include: named entity recognition (Finance-Named Entity Recognition, Finance-NER) tasks in the financial field, NER (Medicine-NER) tasks in the medical field, text classification tasks (Affair-Classify, Affair-CLS) in the government field and text classification tasks (Service-CLS) in the service industry.

[0167] Table 3 (P:%, R:%, F1:%)

[0168]

[0169] As shown in Table 3, for tasks in specific fields, the accuracy, recall rate and comprehensive evaluation index F1 value of the language model (K-DLM) trained using the pre-training method provided in the embodiment of the present application are basically higher than those of the reference model.

[0170] Based on the data shown in the above three tables, it can be seen that the language model trained using the pre-training method provided in the embodiments of the present application performs better on multiple public corpora, in performing classification tasks, and in performing specific domain tasks than the reference model trained using the pre-training method in related technologies.

[0171] It can be understood that the pre-training method of the language model provided in the embodiment of the present application allows the language model to learn the knowledge contained in each sentence in the corpus and the relevant semantics of each word in the sentence through the MLM task and the CSP task, which can effectively improve the performance of the language model on various public corpora and improve the performance of the language model on classification tasks. Moreover, by pre-training the language model with the CL task, it is possible to integrate the knowledge graph of the target domain into the pre-training process of the language model while ensuring that the language model learns the relationship between each entity in the target knowledge graph. Accordingly, when the pre-trained language model is applied to the target domain, the language model also performs better in executing tasks in the target domain.

[0172] It is understandable that the order of the steps of the language model pre-training method provided in the embodiment of the present application can be appropriately adjusted, and the steps can also be increased or decreased according to the circumstances. For example, the above steps 201 to 203 can be deleted according to the circumstances, that is, the server can pre-store a text semantic matrix, and the server can directly determine the semantic vector of each character in the first training text based on the text semantic matrix. Any technician familiar with this technical field can easily think of a method of change within the technical scope disclosed in this application, which should be covered by the protection scope of this application, so it will not be repeated.

[0173] In summary, an embodiment of the present application provides a method for pre-training a language model. The method can pre-train the language model with the MLM task and the semantic original prediction task as the training tasks of the first stage, so that the language model learns the semantic originals of the words in the training text. After completing the pre-training of the first stage, the method can use contrastive learning as the training task of the second stage, and use the second training text determined based on the knowledge graph of the target domain to pre-train the language model, thereby integrating the knowledge graph of the target domain into the pre-training process of the language model, so that the language model can learn the knowledge in the knowledge graph of the target domain. Since the method provided by the embodiment of the present application can introduce different training tasks in the pre-training process of the language model, it can ensure that the pre-training effect of the language model is better.

[0174] Figure 7 This is a flowchart of a result recommendation method provided by the present application. The result recommendation method can be applied to computer devices, for example, Figure 1 The server 110 or the terminal 120 in the illustrated scenario. Figure 6 , the method may include the following steps.

[0175] Step 301: Get the search text.

[0176] The search text may be a question text or a professional term for the target field.

[0177] As a first possible example, if the result recommendation method is applied to a server, the search text may be sent by the terminal to the server.

[0178] As a second possible example, if the result recommendation method is applied to a terminal, the search text may be input into the terminal by a terminal user, or the search text may be sent to the terminal by another terminal.

[0179] Step 302: Input the search text into the search model to obtain the recommended results output by the search model.

[0180] The search model can be a server using Figure 2 and Figure 3 The language model is pre-trained using the pre-training method shown in the figure and then fine-tuned. The recommendation results output by the search model can be web pages, documents, or images related to the search text.

[0181] In the first possible example described above, the server pre-stores a search model. After receiving the search text, the server can input the search text into the search model for processing and obtain recommendation results output by the search model. The server can then send the recommendation results output by the search model to the terminal.

[0182] In the second possible example, the server can send the pre-trained search model to the terminal. After the terminal obtains the search text, it can directly input the search text into the search model for processing and obtain the recommendation results output by the search model.

[0183] In summary, the embodiments of the present application provide a result recommendation method that can obtain search text, input the search text into a search model for processing, and obtain recommendation results output by the search model. Because the search model is pre-trained using the language model pre-training method provided in the above embodiments and obtained after fine-tuning, the recommendation results output by the search model for the search text are also relatively comprehensive and accurate, effectively ensuring the reliability of the search results.

[0184] Figure 8 This is a structural block diagram of a language model pre-training device provided by the present application. The language model pre-training device can be applied to a computer device, which can be Figure 1 The server 110 in the illustrated scenario. Figure 8 As shown, the device includes:

[0185] The masking module 401 is configured to perform masking processing on the first training text. After the masking processing, some words in the first training text are replaced with masks, and the words include one or more characters.

[0186] The embedding module 402 is used to obtain the embedding vector corresponding to the characters in the first training text after masking. The embedding vector includes: a character vector, a semantic primitive vector, a segment vector and a position vector, wherein the semantic primitive vector is a vector representation of the semantic primitive of the character.

[0187] The first training module 403 is used to pre-train the language model based on the embedded vector using the masked language model task and the semantic origin prediction task as the first stage training tasks, wherein the semantic origin prediction task is used to predict the semantic origin of the words replaced with masks in the first training text, and the semantic origin of the word is the intersection of the semantic origins of the various characters included in the word.

[0188] The second training module 404 is used to use contrastive learning as the second-stage training task and pre-train the language model using a second training text; wherein the second training text includes: the original text, a positive sample text obtained by replacing the first word in the original text based on the knowledge graph of the target domain, and a negative sample text obtained by replacing the second word in the original text based on the knowledge graph of the target domain.

[0189] Optionally, the first training module 403 is used to: input the embedding vector into the language model to obtain the predicted words and the predicted semantics of the predicted words output by the language model; determine a first loss value based on the predicted words and the words replaced with masks in the first training text; determine a second loss value based on the predicted semantics of the predicted words and the semantics of the words replaced with masks in the first training text; and pre-train the language model based on the first loss value and the second loss value.

[0190] Alternatively, as Figure 9As shown, the language model pre-training device may further include a sememe determination module 405. The sememe determination module 405 is used to:

[0191] Obtain the semantics of each reference word in the reference word library; obtain the semantics of each word in the reference word library based on the words included in each reference word in the reference word library and the semantics of each reference word, and the semantics of each word is determined based on the union of the semantics of each reference word containing the word; generate a word semantic matrix based on the semantics of each word in the reference word library, and obtain the semantic vectors of the words in the first training text after masking based on the word semantic matrix.

[0192] Optionally, the reference vocabulary is obtained based on a knowledge graph of the target domain and a Chinese concept knowledge base.

[0193] Optionally, continue to refer to Figure 9 The language model pre-training device further includes a sample determination module 406, which is used to:

[0194] Obtain the original text; based on the knowledge graph of the target domain, perform synonymous replacement on the first word in the original text to obtain a positive sample text; based on the knowledge graph of the target domain, perform non-synonymous replacement on the second word in the original text to obtain a negative sample text.

[0195] Optionally, the non-synonymous replacement includes at least one of: hyponymous replacement and relational replacement.

[0196] Optionally, the second training module 404 is used to: input the second training text into the language model to obtain a first predicted similarity and a second predicted similarity output by the language model; determine a third loss value based on the first predicted similarity and the second predicted similarity; and pre-train the language model based on the third loss value; wherein the first predicted similarity is the similarity between the original text and the positive sample text, and the second predicted similarity is the similarity between the original text and the negative sample text.

[0197] In summary, an embodiment of the present application provides a pre-training device for a language model. The device can pre-train the language model with the MLM task and the semantic original prediction task as the first-stage training tasks, so that the language model learns the semantic originals of the words in the training text. After completing the pre-training of the first stage, the method can use contrastive learning as the second-stage training task, and use the knowledge graph based on the target domain to determine the second training text to pre-train the language model, thereby integrating the knowledge graph of the target domain into the pre-training process of the language model, so that the language model can learn the knowledge in the knowledge graph of the target domain. Since the device provided by the embodiment of the present application can introduce different training tasks in the pre-training process of the language model, it can ensure that the pre-training effect of the language model is better.

[0198] Figure 10 This is a structural block diagram of a result recommendation device provided by an embodiment of the present application. The result recommendation device can be applied to a computer device. The computer device can be Figure 1 The terminal 120 in the scene shown. Figure 10 As shown, the device includes:

[0199] The acquisition module 501 is used to acquire the search text.

[0200] The search module 502 is used to input the search text into the search model to obtain the recommended results output by the search model; wherein the search model adopts Figure 8 or Figure 9 The language model shown is trained using a pre-training device.

[0201] In summary, the embodiments of the present application provide a result recommendation device that can obtain search text, input the search text into a search model for processing, and obtain recommendation results output by the search model. Because the search model is pre-trained using the language model pre-training device provided in the above embodiments and obtained after fine-tuning, the recommendation results output by the search model for the search text are also relatively comprehensive and accurate, effectively ensuring the reliability of the search results.

[0202] It can be understood that the language model pre-training device and result recommendation device provided in the above embodiments are only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0203] In addition, the language model pre-training device and language model pre-training method embodiments provided in the above embodiments belong to the same concept, and the result recommendation device and result recommendation method embodiments belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0204] Figure 11 This is a schematic diagram of the structure of a server provided in an embodiment of the present application. The computer device may be Figure 1 The server 110 in the illustrated scenario is described below with the computer device being the server 110 .

[0205] Specifically, the server 110 includes a central processing unit (CPU) 111, a system memory 112 including a random access memory (RAM) 1121 and a read-only memory (ROM) 1122, and a system bus 113 connecting the system memory 112 and the CPU 111. The server 110 also includes a mass storage device 114 for storing an operating system 1141, application programs 1142, and other program modules 1143.

[0206] The mass storage device 114 is connected to the central processing unit 111 through a mass storage controller (not shown) connected to the system bus 113. The mass storage device 114 and its associated computer-readable media provide non-volatile storage for the server 110. That is, the mass storage device 114 may include a computer-readable medium (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0207] Without loss of generality, computer-readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other solid-state storage technologies, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, tape cassettes, magnetic tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the above-mentioned ones. The above-mentioned system memory 112 and mass storage device 114 can be collectively referred to as memory.

[0208] According to various embodiments of the present application, the server 110 may also be connected to a remote computer on a network such as the Internet for operation. That is, the server 110 may be connected to the network 116 via a network interface unit 115 connected to the system bus 113, or the network interface unit 115 may be used to connect to other types of networks or remote computer systems (not shown).

[0209] The above-mentioned memory also includes one or more programs, which are stored in the memory and configured to be executed by the CPU to implement the language model pre-training method or result recommendation method provided by the above-mentioned method embodiment.

[0210] An embodiment of the present application also provides a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the language model pre-training method or result recommendation method provided in the above-mentioned method embodiments.

[0211] An embodiment of the present application also provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the language model pre-training method or result recommendation method provided in the above-mentioned method embodiments.

[0212] Embodiments of the present application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the language model pre-training method or result recommendation method described in any of the above embodiments.

[0213] Optionally, the computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid-state drive (SSD), or an optical disk. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0214] It is understood that the term "at least one" in this application means one or more, and "plurality" means two or more. The terms "first," "second," and the like are used in this application to distinguish between identical or similar items with substantially the same role or function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor does it limit the quantity or order of execution.

[0215] It is also understood that the information and data involved in this application (including but not limited to data used to train the model and data used to provide search text) are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the first training text, second training text and search text involved in this application were all obtained with full authorization.

[0216] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0217] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A language model pre-training method, characterized in that: The method comprises: Performing masking processing on the first training text, where some words in the first training text after the masking processing are replaced with masks, the words including one or more characters; Obtaining an embedding vector corresponding to each character in the first training text after masking, the embedding vector comprising: a character vector, a sememe vector, a segment vector, and a position vector, wherein the sememe vector is a vector representation of the sememe of the character and is determined based on a union of the sememes of each reference word containing the character; The first phase of training involves a masked language model task and a sememe prediction task, and pre-training the language model based on the embedding vector. The sememe prediction task is used to predict the sememes of words replaced with masks in the first training text, where the sememes of the words are the intersection of the sememes of the various characters included in the word. The second stage training task is contrastive learning, and the language model is pre-trained using a second training text. The second training text includes: the original text, a positive sample text obtained by replacing the first word in the original text based on the knowledge graph of the target field, and a negative sample text obtained by replacing the second word in the original text based on the knowledge graph of the target field.

2. The method according to claim 1, characterized in that The training task of the first stage, which uses the masked language model task and the sememe prediction task, is based on the embedded vector pre-trained language model and includes: Inputting the embedding vector into the language model to obtain a predicted word output by the language model and a predicted semantic primitive of the predicted word; Determining a first loss value based on the predicted word and the word replaced by the mask in the first training text; determining a second loss value according to the predicted sememe of the predicted word and the sememe of the word replaced with the mask in the first training text; The language model is pre-trained according to the first loss value and the second loss value.

3. The method according to claim 2, characterized in that Before obtaining the embedding vectors corresponding to the characters in the first training text after masking, the method further includes: Obtaining the semantic origin of each reference word in the reference word database; Based on the characters included in each reference word in the reference word library and the sememe of each reference word, the sememe of each character in the reference word library is obtained, wherein the sememe of each character is determined based on the union of the sememes of each reference word containing the character; generating a character sememe matrix based on the sememes of each character in the reference word library, wherein the character sememe matrix includes a sememe vector of each character in the reference word library; Based on the character sememe matrix, the sememe vectors of the characters in the first training text after masking are obtained.

4. The method according to any one of claims 1 to 3, characterized in that: Before pre-training the language model using a second training text using contrastive learning as the training task of the second stage, the method further includes: Obtaining the original text; Based on the knowledge graph of the target domain, performing synonym replacement on the first word in the original text to obtain the positive sample text; Based on the knowledge graph of the target domain, a non-synonymous replacement is performed on the second word in the original text to obtain the negative sample text.

5. The method according to claim 4, characterized in that The non-synonymous substitution includes at least one of a broad sense substitution, a narrow sense substitution and a relative substitution.

6. The method according to any one of claims 1 to 3, characterized in that: The second-stage training task of contrastive learning uses a second training text to pre-train the language model, including: Inputting the second training text into the language model to obtain a first predicted similarity and a second predicted similarity output by the language model; determining a third loss value according to the first prediction similarity and the second prediction similarity; pre-training the language model according to the third loss value; The first predicted similarity is the similarity between the original text and the positive sample text, and the second predicted similarity is the similarity between the original text and the negative sample text.

7. A result recommendation method, characterized in that: The method comprises: Get the search text; Inputting the search text into a search model to obtain a recommendation result output by the search model; The search model is obtained based on a pre-trained language model, and the language model is pre-trained using the method according to any one of claims 1 to 6.

8. A language model pre-training device, characterized in that: The device comprises: a masking module, configured to perform masking on the first training text, wherein some words in the first training text after the masking are replaced with masks, wherein the words include one or more characters; an embedding module, configured to obtain an embedding vector corresponding to each character in the first training text after masking, the embedding vector comprising: a character vector, a sememe vector, a segment vector, and a position vector, wherein the sememe vector is a vector representation of the sememe of the character and is determined based on a union of the sememes of each reference word containing the character; A first training module is configured to use a masked language model task and a sememe prediction task as first-stage training tasks, and to pre-train the language model based on the embedding vector, wherein the sememe prediction task is configured to predict the sememes of words replaced with masks in the first training text, where the sememes of the words are the intersection of the sememes of the various characters included in the words; A second training module is configured to use contrastive learning as a second-stage training task and to pre-train the language model using a second training text; The second training text includes: the original text, a positive sample text obtained by replacing the first word in the original text based on the knowledge graph of the target field, and a negative sample text obtained by replacing the second word in the original text based on the knowledge graph of the target field.

9. The device according to claim 8, characterized in that The first training module is configured to input the embedding vector into the language model to obtain a predicted word output by the language model and a predicted sememe of the predicted word; determine a first loss value based on the predicted word and the word replaced with the mask in the first training text; and determine a second loss value based on the predicted sememe of the predicted word and the sememe of the word replaced with the mask in the first training text; The language model is pre-trained according to the first loss value and the second loss value.

10. The device according to claim 9, characterized in that The device further includes a sememe determination module; The sememe determination module is used to obtain the sememe of each reference word in the reference word library; Based on the characters included in each reference word in the reference word library and the semantic origin of each reference word, the semantic origin of each character in the reference word library is obtained, wherein the semantic origin of each character is determined based on the union of the semantic origins of each reference word containing the character; based on the semantic origin of each character in the reference word library, a character semantic origin matrix is generated, and the character semantic origin matrix includes the semantic origin vector of each character in the reference word library; based on the character semantic origin matrix, the semantic origin vectors of the characters in the first training text after masking are obtained.

11. The device according to any one of claims 8 to 10, characterized in that The apparatus further comprises a sample determination module; The sample determination module is configured to obtain the original text; perform synonymous replacement on the first word in the original text based on the knowledge graph of the target domain to obtain the positive sample text; Based on the knowledge graph of the target domain, a non-synonymous replacement is performed on the second word in the original text to obtain the negative sample text.

12. The device according to claim 11, characterized in that The non-synonymous substitution includes at least one of a broad sense substitution, a narrow sense substitution and a relative substitution.

13. The device according to any one of claims 8 to 10, characterized in that The second training module is used to input the second training text into the language model to obtain a first predicted similarity and a second predicted similarity output by the language model; determine a third loss value based on the first predicted similarity and the second predicted similarity; and pre-train the language model based on the third loss value; wherein the first predicted similarity is the similarity between the original text and the positive sample text, and the second predicted similarity is the similarity between the original text and the negative sample text.

14. A result recommendation device, characterized in that: The device comprises: Acquisition module, used to obtain search text; A search module, configured to input the search text into a search model and obtain recommendation results output by the search model; The search model is trained using a language model pre-training device as described in any one of claims 8 to 13.

15. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the language model pre-training method according to any one of claims 1 to 6, or the result recommendation method according to claim 7.

16. A computer-readable storage medium, characterized in that The storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by the processor to implement the language model pre-training method as described in any one of claims 1 to 6, or the result recommendation method as described in claim 7.

17. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. The processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the language model pre-training method as described in any one of claims 1 to 6, or the result recommendation method as described in claim 7.

Images