Language model pre-training method, device, equipment and storage medium

Abstract

The present application discloses a language model pre-training method, device, equipment and storage medium, which relates to the field of computer technology, and in particular to the field of deep learning and natural language processing. The specific implementation scheme is: construct a semantic graph based on search terms, search results and historical behavior logs, and sample the semantic graph to generate a first text sequence; mask the first text in the first text sequence to obtain a second text sequence, and encode the second text in the second text sequence through the encoder in the initial model to generate a first semantic representation of the second text; input the first semantic representation into the graph aggregation module of the initial model to obtain a second semantic representation of the second text; perform mask prediction on the second semantic representation to obtain the predicted value of the masked word in the second text, and determine the loss function based on the predicted value and true value of the masked word; pre-train the initial model according to the loss function to obtain a language model.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and further to the field of deep learning and natural language processing, and in particular to a method, apparatus, device and storage medium for pre-training a language model. Background Art

[0002] In scenarios such as search, ad recall, and news push, it is often necessary to display more relevant search results or news push that users are more interested in to users. For example, a language model is used to semantically represent the search terms input by the user to obtain a semantic representation vector of the search terms, and the similarity between the semantic representation vector of the search terms and the feature vector of the search results is calculated, and search results that meet preset conditions are displayed to users. It can be seen that in order to obtain more relevant search results or news push that users are more interested in, the key is to improve the semantic representation ability of the language model. Therefore, how to improve the semantic representation ability of the language model has become an important research direction. Summary of the invention

[0003] The present application provides a language model pre-training method, apparatus, device and storage medium.

[0004] According to a first aspect of the present application, a method for training a language model is provided, comprising:

[0005] Construct a semantic graph according to the search terms, search results, and historical behavior logs, and sample the semantic graph to generate a first text sequence;

[0006] Masking the first text in the first text sequence to obtain a second text sequence, and encoding the second text in the second text sequence through the encoder in the initial model to generate a first semantic representation of the second text;

[0007] Inputting the first semantic representation into a graph aggregation module of the initial model to obtain a second semantic representation of the second text;

[0008] Performing mask prediction on the second semantic representation to obtain a predicted value of the masked word in the second text, and determining a loss function according to the predicted value and the true value of the masked word;

[0009] According to the loss function, the initial model is pre-trained to obtain a language model.

[0010] According to a second aspect of the present application, a neural network modeling method in a target task scenario is provided, comprising:

[0011] Obtain training samples of target task scenarios;

[0012] The neural network is trained according to the training samples to obtain a neural network model suitable for the target task.

[0013] According to a third aspect of the present application, a language model pre-training device is provided, comprising:

[0014] A sampling module, used to construct a semantic graph according to the search terms, search results and historical behavior logs, and to sample the semantic graph to generate a first text sequence;

[0015] A first generation module is used to perform mask processing on the first text in the first text sequence to obtain a second text sequence, and encode the second text in the second text sequence through the encoder in the initial model to generate a first semantic representation of the second text;

[0016] A second generation module is used to input the first semantic representation into the graph aggregation module of the initial model to obtain a second semantic representation of the second text;

[0017] A prediction module, configured to perform mask prediction on the second semantic representation, obtain a predicted value of a masked word in the second text, and determine a loss function according to the predicted value and a true value of the masked word;

[0018] The pre-training module is used to pre-train the initial model according to the loss function to obtain a language model.

[0019] According to a fourth aspect of the present application, a neural network modeling device for a target task is provided, comprising:

[0020] An acquisition module is used to obtain training samples of target task scenarios;

[0021] The training module is used to train the neural network according to the training samples to obtain a neural network model suitable for the target task.

[0022] According to a fifth aspect of the present application, there is provided an electronic device, including:

[0023] at least one processor; and

[0024] a memory communicatively connected to at least one processor; wherein,

[0025] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor so that the at least one processor can execute the language model pre-training method of the first aspect mentioned above.

[0026] According to a sixth aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the language model pre-training method of the aforementioned first aspect.

[0027] According to a seventh aspect of the present application, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the language model pre-training method according to the aforementioned first aspect.

[0028] According to the technical solution of the present application, by sampling the semantic graph, a first text sequence containing semantics and semantic edge relations is obtained, and the initial model is trained by the encoder and graph aggregation module in the initial model, and the initial model is pre-trained into a language model. The encoder and graph aggregation module are pre-trained while pre-training the language model. Not only is the encoder pre-trained in terms of semantics, but the graph aggregation module is also pre-trained in terms of semantic edge relations, so that the graph aggregation module in the language model is hot-started, improving the semantic understanding ability of the language model, and thus improving performance in downstream tasks.

[0029] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.

[0031] Figure 1 It is a flowchart of a method for pre-training a language model provided in an embodiment of the present application;

[0032] Figure 2 is a schematic diagram of a semantic graph provided according to an embodiment of the present application;

[0033] Figure 3 is an example diagram of a language pre-training method provided according to an embodiment of the present application;

[0034] Figure 4 It is a flowchart of a neural network modeling method in a target task scenario provided by an embodiment of the present application;

[0035] Figure 5 is a schematic diagram of an explicit modeling method of a neural network in a target task scenario provided according to an embodiment of the present application;

[0036] Figure 6 It is a schematic diagram of an implicit modeling method of a neural network in a target task scenario provided according to an embodiment of the present application;

[0037] Figure 7 It is a structural block diagram of a language model pre-training device provided according to an embodiment of the present application;

[0038] Figure 8It is a structural block diagram of a neural network modeling device in a target task scenario provided according to an embodiment of the present application;

[0039] Fig. 9 It is a block diagram of an electronic device for implementing a pre-training method of a language model according to the implementation of the present application. DETAILED DESCRIPTION

[0040] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0041] In the prior art, a pre-trained language model is generally used to encode the text information of nodes in the semantic graph and produce node representations. The graph aggregation module is then used to aggregate the node representations to improve the node representation capabilities. Aggregation methods include neighbor summation, averaging, and maximum value. Currently, most graph aggregation modules are cold-started, and the aggregation method is too simple. Only semantic-related pre-training is performed in the model, and there is a lack of pre-training modeling for semantic edge relationships on the semantic graph.

[0042] To this end, the present application provides a language model pre-training method, apparatus, device and storage medium. Specifically, the language model pre-training method, apparatus, device and storage medium of the present application embodiment are described below with reference to the accompanying drawings.

[0043] Figure 1 It is a flow chart of a language model pre-training method provided according to an embodiment of the present application. It should be noted that the language model pre-training method of the embodiment of the present application can be applied to the language model pre-training device of the embodiment of the present application, and the language model pre-training device can be configured on an electronic device.

[0044] It should also be noted that the language model of the embodiment of the present application can be used in any one or more scenarios such as search scenarios, advertisement recall scenarios, news recommendation scenarios, etc., and the language model is used to obtain the semantic representation vector of the text content in the scenario.

[0045] like Figure 1 As shown, the pre-training method of the language model may include the following steps:

[0046] Step 101: construct a semantic graph based on search terms, search results, and historical behavior logs, and sample the semantic graph to generate a first text sequence.

[0047] Optionally, in some embodiments of the present application, search terms and search results are used as nodes, and the relationship between the nodes is determined based on historical behavior logs, and the relationship is determined as an edge between the nodes to construct a semantic graph.

[0048] It should be noted that search terms, search results, and historical behavior logs can be obtained from the historical database to construct a semantic graph. The search terms and search results can be a word or a sentence, and the historical behavior log records the relationship between nodes. Optionally, the relationship between the nodes can be the relationship between the search terms and the search results, the relationship between the search terms and the search terms, or the relationship between the search results and the search results, and the relationship between the nodes is determined as the edge between the nodes. As an example, Figure 2 As shown, Figure 2 A schematic diagram of a semantic graph provided in an embodiment of the present application, wherein node q is a search term (query), t is a search result (title), and 201 is an edge between node q1 and node t1.

[0049] Optionally, in some embodiments of the present application, a deep walk may be performed on the constructed semantic graph to sample and generate the first text sequence. Figure 2 Taking the semantic graph shown in the embodiment as an example, a deep walk of the semantic graph can collect a text sequence of four nodes q1-t1-q2-t2 and the relationship between the nodes, and the text sequence is used as the first text sequence. It should be noted that the number of nodes and the relationship between the nodes in the first text sequence is only an example given to facilitate the understanding of the present application by those skilled in the art. That is to say, the text sequences of other numbers of nodes and the relationship between the nodes in the semantic graph can also be sampled, such as t3-q1-t1, t3-q1-t1-q4 and other text sequences, which are not specifically limited in the present application.

[0050] Step 102, masking is performed on the first text in the first text sequence to obtain a second text sequence, and the second text in the second text sequence is encoded by the encoder in the initial model to generate a first semantic representation of the second text.

[0051] It should be noted that in some embodiments of the present application, the first text in the first text sequence can be masked by randomly masking the words in the first text to obtain a second text sequence after masking, and the second text in the second text sequence is encoded by the encoder in the initial model to generate a first semantic representation of the second text. As an example, the encoder in the initial model can use ERNIE (Enhanced Representation through Knowledge Integration, a knowledge-enhanced pre-trained language model).

[0052] Step 103: input the first semantic representation into a graph aggregation module of the initial model to obtain a second semantic representation of the second text.

[0053] Optionally, in some embodiments of the present application, the first semantic representation may be input into the graph aggregation module of the initial model to perform self-attention calculation to obtain a second semantic representation of the second text, wherein the second semantic representation at least includes a graph representation of the second text in the second text sequence.

[0054] Optionally, in some embodiments of the present application, the graph aggregation module may adopt Transformersage (graph aggregation module based on self-attention). As an example, the first semantic representation is input into transformer sage for self-attention calculation to obtain a second semantic representation of the second text, which at least includes a graph representation of the second text in the second text sequence. The graph representation of the second text in the second text sequence includes, in addition to the semantic representation of the current second text, a semantic representation of the text adjacent to the current second text in the second text sequence.

[0055] Step 104 , performing mask prediction on the second semantic representation to obtain the predicted value of the masked word in the second text, and determining the loss function according to the predicted value and the true value of the masked word.

[0056] As an example, the second semantic representation obtained after aggregation can be input into a masked language model (mlm), and the masked words in the second text are predicted based on the graph representation in the second semantic representation to obtain the predicted value of the masked words, and the loss function is determined based on the predicted value and true value of the masked words.

[0057] Step 105: pre-train the initial model according to the loss function to obtain a language model.

[0058] Optionally, in some embodiments of the present application, the model parameters of the encoder and the graph aggregation module can be updated by backpropagation according to the loss function to obtain a language model.

[0059] In order to better understand the pre-training method of the language model proposed in the embodiment of the present application, Figure 3 This is an example diagram of the pre-training method of the language model provided in the embodiment of the present application. Figure 3As shown, according to the search terms, search results and historical behavior logs, a semantic graph is constructed (S301), and the semantic graph is sampled to generate a first text sequence; the first text in the first text sequence is masked to obtain a second text sequence (S302), and the second text in the second text sequence is encoded by the encoder in the initial model (S303) to generate a first semantic representation of the second text (S304); the first semantic representation is input into the graph aggregation module in the initial model (S305) to obtain a second semantic representation of the second text (S306), wherein the second semantic representation of the second text is a graph representation that aggregates the first semantic representation of the second text and the second text in the second text sequence. Mask prediction is performed on the second semantic representation (S307), and a loss function is determined according to the predicted mask word prediction value and the true value, and the initial model is pre-trained based on the function to obtain a language model.

[0060] According to the method for pre-training a language model in an embodiment of the present application, a first text sequence containing semantics and semantic edge relations is obtained by sampling a semantic graph, and the initial graph is trained by the encoder and graph aggregation module in the initial model, and the initial model is pre-trained into a language model. The encoder and graph aggregation module are pre-trained while pre-training the language model. Not only is the encoder pre-trained in terms of semantics, but the graph aggregation module is also pre-trained in terms of semantic edge relations, so that the graph aggregation module in the language model is hot-started, the semantic understanding ability of the language model is improved, and the performance in downstream tasks is improved.

[0061] In some embodiments of the present application, the language model is pre-trained to improve the semantic understanding ability of the language model, and the neural network can continue to be modeled, used, and trained to obtain a neural network model suitable for the target task scenario. The target scenario can be a search scenario, an advertisement recall scenario, a news recommendation scenario, etc. Based on the pre-trained language model, the neural network is continued to be trained to obtain a neural network model with better performance in downstream tasks, such as recommending search results that are highly relevant to the search term or matching advertisement texts that the user is interested in. As an example, Figure 4 As shown, the neural network modeling method in the target task scenario provided by the embodiment of the present application may include the following steps:

[0062] Step 401, obtaining training samples of the target task scenario.

[0063] Step 402: Train the neural network based on the training samples to obtain a graph neural network model suitable for the target task.

[0064] Optionally, in some embodiments of the present application, the training samples in the neural network modeling method in the target task scenario may include search word samples and graph structure information of the search word samples, and the neural network may include Figure 1 As an example, the training sample includes the search word sample and the graph structure information of the search word sample. Figure 1 The language model in the illustrated embodiment constructs a neural network suitable for the target task scenario. It can be seen that since the neural network learns the graph structure information between samples, the neural network modeling method can be called an explicit graph modeling method.

[0065] like Figure 5 As shown, Figure 5 A schematic diagram of an explicit modeling method for a neural network in a target task scenario provided in an embodiment of the present application. The explicit graph modeling method utilizes both a pre-trained encoder and a graph aggregation module to input graph structure information into the graph neural network. Taking the target task scenario as a search scenario as an example, in the search scenario, the graph structure information center is the search term entered by the user recorded in the historical behavior log, and neig is the search result clicked by the user after entering the search term. The search term and the search result clicked by the user after entering the search term are input into the neural network, encoded by the encoder, and aggregated by the graph aggregation module to obtain representation information that aggregates the search result neig information, so that the neural network can be trained using the representation information so that the trained neural network is suitable for the target task scenario.

[0066] Optionally, in some embodiments of the present application, the training samples in the neural network modeling method in the target task scenario may include search word samples, and the neural network may include Figure 1 The encoder in the language model in the embodiment shown. As an example, the training sample includes a search word sample, and the training sample and Figure 1 The encoder in the language model in the illustrated embodiment constructs a neural network suitable for the target task scenario. It can be seen that since the neural network has learned semantic representation capabilities but has not learned graph structure information, the neural network modeling method can be called an implicit graph modeling method.

[0067] like Figure 6 As shown, Figure 6 A schematic diagram of an implicit modeling method of a neural network in a target task scenario provided by an embodiment of the present application. The implicit graph modeling method only needs to use an encoder in a pre-trained language model to input a search sample word into the neural network so as to train the neural network using the semantic representation information of the search sample word so that the trained neural network is suitable for the target task scenario.

[0068] It should be noted that the explicit graph modeling method or the implicit graph modeling method can be selected according to the actual business scenario. Since the explicit graph modeling method uses the encoder and graph aggregation module, the recommendation results are more accurate, but the response speed is slower than that of the implicit graph modeling method, and it is more suitable for business scenarios that do not require response speed, such as offline recommendation scenarios. The implicit graph modeling method only uses the encoder, and the response speed is fast, but the accuracy of the recommendation results is lower than that of the explicit graph modeling method, and it is more suitable for business scenarios that require response speed, such as online recommendations.

[0069] According to the neural network modeling method in the target task scenario of the embodiment of the present application, the pre-trained neural network is further modeled, used, and trained to obtain a neural network model suitable for the target task scenario, thereby obtaining search results with high relevance to the search terms or matching the advertising text that the user is interested in. The explicit graph modeling method or the implicit graph modeling method can be selected according to actual business needs, which is more flexible in the use of downstream tasks.

[0070] Figure 7 is a structural block diagram of a language model pre-training device provided according to an embodiment of the present application. Figure 7 As shown, the pre-training device of the language model may include a sampling module 701, a first generation module 702, a second generation module 703, a prediction module 704 and a pre-training module 705.

[0071] Specifically, the sampling module 701 is used to construct a semantic graph according to the search terms, search results and historical behavior logs, and to sample the semantic graph to generate a first text sequence;

[0072] The first generating module 702 is used to perform mask processing on the first text in the first text sequence to obtain a second text sequence, and encode the second text in the second text sequence through the encoder in the initial model to generate a first semantic representation of the second text;

[0073] The second generation module 703 is used to input the first semantic representation into the graph aggregation module of the initial model to obtain a second semantic representation of the second text;

[0074] A prediction module 704 is used to perform mask prediction on the second semantic representation, obtain a predicted value of the masked word in the second text, and determine a loss function according to the predicted value and the true value of the masked word;

[0075] The pre-training module 705 is used to pre-train the initial model according to the loss function to obtain a language model.

[0076] In some embodiments of the present application, the sampling module 701 is specifically used to: use search terms and search results as nodes, determine the relationship between nodes based on historical behavior logs, and determine the relationship as the edge of the node to construct a semantic graph.

[0077] In some embodiments of the present application, the second generation module 703 is specifically used to: input the first semantic representation into the graph aggregation module of the initial model to perform self-attention calculation to obtain a second semantic representation of the second text; wherein the second semantic representation at least includes a graph representation of the second text in the second text sequence.

[0078] In some embodiments of the present application, the pre-training module 705 is specifically used to: perform back-propagation according to the loss function to update the model parameters of the encoder and the graph aggregation module to obtain a language model.

[0079] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0080] According to the pre-training device of the language model of the embodiment of the present application, by sampling the semantic graph, a first text sequence containing semantics and semantic edge relations is obtained, and the initial graph is trained by the encoder and graph aggregation module in the initial model, and the initial model is pre-trained into a language model. The encoder and graph aggregation module are pre-trained while pre-training the language model. Not only is the encoder pre-trained in terms of semantics, but the graph aggregation module is also pre-trained in terms of semantic edge relations, so that the graph aggregation module in the language model is hot-started, the semantic understanding ability of the language model is improved, and the performance in downstream tasks is improved.

[0081] Figure 8 1 is a block diagram of a neural network modeling device in a target task scenario provided by an embodiment of the present application. Figure 8 As shown, the neural network modeling device in the target task scenario may include an acquisition module 801 and a training module 802.

[0082] Specifically, the acquisition module 801 is used to acquire training samples of the target task scenario;

[0083] The training module 802 is used to train the neural network according to the training samples to obtain a neural network model suitable for the target task.

[0084] In some embodiments of the present application, the training sample may include a search word sample and graph structure information of the search word sample; the neural network may include Figure 1 Language model in the illustrated embodiment.

[0085] In some embodiments of the present application, the training sample may include a search word sample; the neural network may include Figure 1 In the illustrated embodiment, an encoder within a language model.

[0086] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0087] According to the neural network modeling device in the target task scenario of the embodiment of the present application, the pre-trained neural network continues to be modeled, used, and trained to obtain a neural network model suitable for the target task scenario, thereby obtaining search results with high relevance to the search terms or matching the advertising text that the user is interested in. The explicit graph modeling method or the implicit graph modeling method can be selected according to actual business needs, which is more flexible in the use of downstream tasks.

[0088] According to an embodiment of the present application, the present application also provides an electronic device, a readable storage medium and a computer program product.

[0089] like Fig. 9 , is a block diagram of an electronic device for implementing a pre-training method of a language model according to an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present application described herein and / or required.

[0090] like Fig. 9 As shown, the electronic device includes: one or more processors 901, a memory 902, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig. 9 A processor 901 is taken as an example.

[0091] The memory 902 is a non-transitory computer-readable storage medium provided in the present application. The memory stores instructions executable by at least one processor to enable the at least one processor to perform the pre-training method of the language model provided in the present application. The non-transitory computer-readable storage medium of the present application stores computer instructions, which are used to enable a computer to perform the pre-training method of the language model provided in the present application.

[0092] The memory 902 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the program instructions / modules corresponding to the pre-training method of the language model in the embodiment of the present application (for example, the attached Figure 7 The processor 901 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 902, that is, the pre-training method of the language model in the above method embodiment is implemented.

[0093] The memory 902 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of an electronic device for implementing a pre-training method for a language model, etc. In addition, the memory 902 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 902 may optionally include a memory remotely disposed relative to the processor 901, and these remote memories may be connected to an electronic device for implementing a pre-training method for a language model via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0094] The electronic device for implementing the pre-training method of the language model may further include: an input device 903 and an output device 904. The processor 901, the memory 902, the input device 903 and the output device 904 may be connected via a bus or other means. Fig. 9 The example of connecting through bus is taken in the following.

[0095] The input device 903 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device used to implement the pre-training method of the language model, such as a touch screen, a keypad, a mouse, a trackpad, a touchpad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 904 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0096] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs, that is, the present application also proposes a computer program, which, when executed by a processor, implements the pre-training method of the language model described in the above embodiment, and the one or more computer programs can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a dedicated or general-purpose programmable processor, which can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0097] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0098] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0099] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0100] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0101] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0102] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A language model pre-training method, comprising: Constructing a semantic graph according to the search terms, search results, and historical behavior logs, and sampling the semantic graph to generate a first text sequence; Performing masking processing on a first text in a first text sequence to obtain a second text sequence, and encoding a second text in the second text sequence through an encoder in an initial model to generate a first semantic representation of the second text; Inputting the first semantic representation into a graph aggregation module of the initial model to obtain a second semantic representation of the second text; Performing mask prediction on the second semantic representation to obtain a predicted value of a masked word in the second text, and determining a loss function according to the predicted value and a true value of the masked word; The initial model is pre-trained according to the loss function to obtain the language model.

2. The method according to claim 1, wherein: The step of constructing a semantic graph based on search terms, search results, and historical behavior logs includes: The search terms and the search results are taken as nodes, and the relationship between the nodes is determined according to the historical behavior log, and the relationship is determined as the edge between the nodes to construct the semantic graph.

3. The method according to claim 1, wherein: The step of inputting the first semantic representation into a graph aggregation module of the initial model to obtain a second semantic representation of the second text includes: The first semantic representation is input into the graph aggregation module of the initial model to perform self-attention calculation to obtain a second semantic representation of the second text; wherein the second semantic representation at least includes a graph representation of the second text in the second text sequence.

4. The method according to claim 1, wherein: The pre-training of the language model according to the loss function to obtain the language model includes: Backward conduction is performed according to the loss function to update the model parameters of the encoder and the graph aggregation module to obtain the language model.

5. A neural network modeling method in a target task scenario, comprising: Obtaining a training sample of the target task scenario; Training the neural network according to the training samples to obtain a neural network model suitable for the target task; The training samples include search term samples and graph structure information of the search term samples; and the neural network includes a language model as described in any one of claims 1 to 4.

6. The method according to claim 5, wherein: The training samples include search word samples; the neural network includes an encoder in the language model as described in any one of claims 1 to 4.

7. A language model pre-training device, comprising: A sampling module, used to construct a semantic graph according to the search terms, search results and historical behavior logs, and to sample the semantic graph to generate a first text sequence; A first generation module is used to perform mask processing on the first text in the first text sequence to obtain a second text sequence, and encode the second text in the second text sequence through an encoder in the initial model to generate a first semantic representation of the second text; A second generation module, configured to input the first semantic representation into a graph aggregation module of the initial model to obtain a second semantic representation of the second text; A prediction module, configured to perform mask prediction on the second semantic representation, obtain a predicted value of a masked word in the second text, and determine a loss function according to the predicted value and a true value of the masked word; A pre-training module is used to pre-train the initial model according to the loss function to obtain the language model.

8. The device according to claim 7, wherein: The sampling module is specifically used for: The search terms and the search results are taken as nodes, and the relationship between the nodes is determined according to the historical behavior log, and the relationship is determined as the edge of the node to construct the semantic graph.

9. The device according to claim 7, wherein: The second generating module is specifically used for: The first semantic representation is input into the graph aggregation module of the initial model to perform self-attention calculation to obtain a second semantic representation of the second text; wherein the second semantic representation at least includes a graph representation of the second text in the second text sequence.

10. The device according to claim 7, wherein: The pre-training module is specifically used for: Backward conduction is performed according to the loss function to update the model parameters of the encoder and the graph aggregation module to obtain the language model.

11. A neural network modeling device in a target task scenario, comprising: An acquisition module, used to acquire training samples of the target task scenario; A training module, used to train the neural network according to the training samples to obtain a neural network model suitable for the target task; The training samples include search term samples and graph structure information of the search term samples; and the neural network includes a language model as described in any one of claims 1 to 4.

12. The device according to claim 11, wherein The training samples include search word samples; the neural network includes an encoder in the language model as described in any one of claims 1 to 4.

13. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 4.

14. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 4.

15. A computer program product comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 4.

Images