Method for training natural language processing model, and method for generating subsequent text of dialogue

Abstract

Provided in the present invention are a method for training a natural language processing model, and a method for generating subsequent text of a dialogue. The method for training a natural language processing model comprises: acquiring multiple types of heterogeneous sample data and a pre-constructed natural language processing model, wherein the heterogeneous sample data includes structured data, unstructured data, a knowledge graph and expert experience data; performing data encoding fusion on the structured data, the unstructured data and the knowledge graph, so as to obtain encoding fused data; and on the basis of the encoding fused data and the expert experience data, training the pre-constructed natural language processing model, so as to obtain a trained natural language processing model. The present invention enables a knowledge graph to be placed in a model, not as an independent retrieval corpus, but as a method for knowledge enhancement, thereby improving the efficiency of a natural language processing model.

Description

Training methods for natural language processing models and methods for generating conversational context Technical Field

[0001] The present invention relates to the field of natural language processing technology, and in particular to a method for training a natural language processing model and a method for generating a conversation context. Background Art

[0002] Question answering (QA) systems are an advanced form of information retrieval system that accurately and concisely answer questions posed by users in natural language. Based on the domain of knowledge, QA systems can be categorized as "domain-specific" and "open-domain." Domain-specific systems focus on answering questions in specific areas, such as medicine, sports, and government affairs. Open-domain systems aim to answer questions on a wide range of topics, encompassing everything from astronomy to geography.

[0003] The basic process of a commonly used question-answering system is shown in Figure 1. It first extracts keywords from the question, then searches for information in a corpus based on the extracted keywords. Based on the retrieved relevant documents, the most relevant answer is output. Therefore, the core of existing question-answering systems is information retrieval. In recent years, as the concept of knowledge graphs has permeated various fields, using knowledge graphs as corpora for retrieval has become a mainstream approach. Related research has been conducted in vertical fields such as finance, healthcare, tourism, agriculture, and e-commerce. These knowledge graph-based question-answering systems either utilize the knowledge data in them combined with deep learning to build question-answering systems, leverage the graph's reasoning capabilities to understand questions, or integrate the information of questions and triples into a vector space, performing similarity calculations on the questions within the vector space to obtain the approximate answer required by the user. In these knowledge graph-based question-answering systems, different question-answering tasks (including simple and complex questions) require different knowledge graphs. During each round of dialogue, the dialogue system must search the knowledge graph corpus, which reduces computational efficiency and has significant limitations, including poor context understanding and poor multi-round dialogue performance. Summary of the Invention

[0004] The present invention provides a method for training a natural language processing model and a method for generating a dialogue context to solve the above problems.

[0005] The present invention provides a training method for a natural language processing model, comprising:

[0006] Acquire a variety of heterogeneous sample data and a pre-built natural language processing model, wherein the heterogeneous sample data includes structured data, unstructured data, knowledge graphs, and expert experience data;

[0007] Performing data encoding fusion on the structured data, unstructured data, and knowledge graph to obtain encoded fusion data;

[0008] The pre-built natural language processing model is trained based on the encoded fusion data and the expert experience data to obtain a trained natural language processing model.

[0009] According to a natural language processing model training method provided by the present invention, the data encoding and fusion of the structured data, unstructured data, and knowledge graph is performed to obtain encoded fusion data, including:

[0010] Performing vector conversion on the structured data using a preset first encoding algorithm to obtain first encoded data;

[0011] Performing vector conversion on the unstructured data using a preset first encoding algorithm and / or dependency syntax analysis or a second encoding algorithm to obtain second encoded data;

[0012] A preset graph convolutional neural network is used to extract features from the knowledge graph, and a graph attention mechanism is used to enhance the extracted features to obtain third encoded data; wherein the encoded fusion data includes first encoded data, second encoded data, and third encoded data, the first encoding algorithm is at least one of TF-IDF, Word2vec, or One-hot, and the second encoding algorithm is a BPE-based encoding algorithm.

[0013] According to a natural language processing model training method provided by the present invention, a second encoding algorithm is used to perform vector conversion on the unstructured data to obtain second encoded data, including:

[0014] Performing named entity recognition on the unstructured data to obtain entity label information;

[0015] Performing dependency syntax analysis on the unstructured data to obtain annotation relationships;

[0016] Based on the entity tag information and the annotation relationship, the words in the unstructured data are encoded using the BPF algorithm to obtain second encoded data.

[0017] According to a training method for a natural language processing model provided by the present invention, the pre-built natural language processing model is a Transformer-based neural network model, including a Transformer layer of a multi-layer fusion RNN; the pre-built natural language processing model is trained based on the encoded fusion data and expert experience data to obtain a trained natural language processing model, including:

[0018] Encoding the expert experience data to obtain encoded professional experience data;

[0019] Using the third encoded data as a query vector and a key vector, and the first encoded data and the second encoded data as a value vector;

[0020] The query vector, the key vector, the value vector and the encoded professional experience data are input into the Transformer layer of the multi-layer fusion RNN, so as to train and update the network parameters of the Transformer layer until a preset update stop condition is reached, thereby obtaining a trained natural language processing model.

[0021] The present invention also provides a method for generating a conversation context, comprising:

[0022] Obtain the conversation context provided by the user and the trained natural language processing model;

[0023] The preceding conversation context is input into the trained natural language processing model to obtain a following conversation context corresponding to the preceding conversation context; wherein the trained natural language processing model is obtained based on the training method of the natural language processing model as described above.

[0024] According to a method for generating a conversation subtext provided by the present invention, inputting the conversation preface into the trained natural language processing model to obtain a conversation subtext corresponding to the conversation preface includes:

[0025] Perform keyword extraction and expansion on the conversation text to obtain a question keyword vector;

[0026] The question keyword vector is input into the trained natural language processing model, and a database vector corresponding to the question keyword vector is obtained through multi-layer Transformer layer reasoning in the trained natural language processing model, and the conversation context is obtained based on the database vector.

[0027] The present invention also provides a training device for a natural language processing model, comprising:

[0028] A data and model acquisition module is used to acquire a variety of heterogeneous sample data and pre-built natural language processing models. The heterogeneous sample data includes structured data, unstructured data, knowledge graphs, and expert experience data;

[0029] A data fusion module is used to perform data encoding fusion on the structured data, unstructured data and knowledge graph to obtain encoded fusion data;

[0030] The model training module is used to train the pre-built natural language processing model based on the encoded fusion data and expert experience data to obtain a trained natural language processing model.

[0031] The present invention also provides a device for generating a conversation context, comprising:

[0032] The context and model acquisition module is used to obtain the conversation context provided by the user and the trained natural language processing model;

[0033] A context output module is used to input the conversation context into the trained natural language processing model to obtain a conversation context corresponding to the conversation context; wherein the trained natural language processing model is obtained based on the training device of the natural language processing model as described above.

[0034] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the program, the method for training a natural language processing model as described in any one of the above or the method for generating a conversation context as described in any one of the above is implemented.

[0035] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for training a natural language processing model as described in any one of the above or a method for generating a dialogue context as described in any one of the above.

[0036] The present invention provides a method for training a natural language processing model and a method for generating conversational context. The method integrates structured data, unstructured data, a knowledge graph, and expert experience as sample data, enabling better model training, enhancing context-based text reasoning, and improving the accuracy of multi-round conversations. Furthermore, the knowledge graph is no longer used as an independent retrieval corpus, but rather as a knowledge enhancement within the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] FIG1 is a schematic diagram of the workflow of a question-answering system in the prior art;

[0039] FIG2 is a flow chart of a method for training a natural language processing model according to an embodiment of the present invention;

[0040] FIG3 is a second flow chart of a method for training a natural language processing model according to an embodiment of the present invention;

[0041] FIG4 is a schematic diagram of a flow chart of a BPF algorithm provided by an embodiment of the present invention;

[0042] 5 is a flow chart of a method for generating a conversation subtitle according to an embodiment of the present invention;

[0043] FIG6 illustrates a schematic diagram of the physical structure of an electronic device. [0043.1][Corrected 14.03.2024 according to Rule 91] Figure 7 is the existing technical underlying model used in the present invention - Transformer model DETAILED DESCRIPTION

[0044] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0045] The present invention proposes a training method for a natural language processing model. Through knowledge fusion, the knowledge graph is no longer used as an independent retrieval corpus, but is placed within the natural language processing model as a knowledge enhancement method. In addition, for heterogeneous data sets, unified training is performed through different encoding methods, so that the model is better trained, the context-based text reasoning ability is enhanced, and the accuracy of multi-round dialogue is improved. The present invention adopts a combination of multiple data sets such as knowledge graphs, expert knowledge, structured data and unstructured data for training. The training method of the natural language processing model provided by the present invention is specifically described below with reference to the accompanying drawings.

[0046] FIG2 is a flowchart of a method for training a natural language processing model according to an embodiment of the present invention; FIG3 is a flowchart of a method for training a natural language processing model according to an embodiment of the present invention; as shown in FIG2 and FIG3 , the method for training a natural language processing model includes the following steps:

[0047] S101, obtaining a variety of heterogeneous sample data and a pre-built natural language processing model, wherein the heterogeneous sample data includes structured data, unstructured data, knowledge graphs, and expert experience data.

[0048] Structured data is highly organized and neatly formatted data. Also known as quantitative data, structured data is information that can be represented by numbers or a uniform structure, such as numbers or symbols. In data processing, this data is typically stored and managed in a relational database. When using Structured Query Language, or SQL, computer programs can easily search for these terms.

[0049] Unstructured data is essentially any data other than structured data. It does not conform to any predefined model. It may be textual or non-textual, and may be human-generated or machine-generated. Simply put, unstructured data is data with variable fields.

[0050] Knowledge graph is a form of expressing entities through relationships and attributes. Google Knowledge Graph's slogan "things not strings" gives the essence of knowledge graph, namely: the object or thing hidden behind the string, this object is supported by relationships and attributes to determine that it is this entity and not another entity.

[0051] S102, performing data encoding fusion on the structured data, unstructured data and knowledge graph to obtain encoded fusion data.

[0052] In this step, different data encodings are performed for different types of heterogeneous data. The specific process is described below.

[0053] S103: Training the pre-built natural language processing model based on the encoded fusion data and the expert experience data to obtain a trained natural language processing model.

[0054] The natural language processing model training method provided by the embodiments of the present invention integrates structured data, unstructured data, knowledge graphs, and expert experience as sample data, enabling better model training, enhancing context-based text reasoning capabilities, and improving the accuracy of multi-round conversations. Furthermore, the knowledge graph is no longer used as an independent retrieval corpus, but rather as a knowledge enhancement within the model.

[0055] Furthermore, based on the above embodiment, the data encoding and fusion of the structured data, the unstructured data and the knowledge graph are performed to obtain the encoded fusion data, including:

[0056] The structured data is vector-converted using a preset first encoding algorithm to obtain first encoded data.

[0057] The unstructured data is vector-converted using a preset first encoding algorithm and / or dependency syntax analysis or a second encoding algorithm to obtain second encoded data.

[0058] A preset graph convolutional neural network is used to extract features from the knowledge graph, and a graph attention mechanism is used to enhance the extracted features to obtain third encoded data; wherein the encoded fusion data includes first encoded data, second encoded data, and third encoded data, the first encoding algorithm is at least one of TF-IDF, Word2vec, or One-hot, and the second encoding algorithm is a BPE-based encoding algorithm.

[0059] Specifically, for the knowledge graph, the convolutional neural network (R-Graph Convolutional Networks, RGCN) integrated with the knowledge graph is used to perform preliminary feature extraction on the multi-attribute and directed edges of the knowledge graph, which is then enhanced twice through the graph attention mechanism (Graph Attention Network, GAT) and used as the Q value (query value) and K value (key value) input of the Transformer to represent the model for training.

[0060] For structured data, due to its high degree of alignment, a general encoding method (i.e., the first encoding algorithm) can be used, such as TF-IDF, Word2vec, One-hot, etc.

[0061] For unstructured data, encoding can be performed using general encoding, a combination of general encoding and dependency syntax analysis, or an improved BPE algorithm (i.e., the second encoding algorithm).

[0062] Furthermore, based on the above embodiment, the unstructured data is vectorized using a second encoding algorithm to obtain second encoded data, including:

[0063] Perform named entity recognition on the unstructured data to obtain entity label information.

[0064] Dependency syntactic analysis is performed on the unstructured data to obtain tagging relationships.

[0065] Based on the entity tag information and the annotation relationship, the words in the unstructured data are encoded using the BPF algorithm to obtain second encoded data.

[0066] Specifically, the BPE algorithm, short for Byte Pair Encoding, is a bottom-up compression algorithm. It is primarily used in English text. It treats an English word as a combination of multiple letters. Certain common letter combinations, such as es and ing, are replaced with new, custom symbols and merged layer by layer until a satisfactory result is achieved. In the Chinese context, unstructured data consists of "characters," which cannot be broken down into finer granularity. Combinations of characters form words or sentences. Therefore, common words are replaced with other symbols, called subwords, enabling the application of BPE in Chinese text. However, using only the BPE algorithm can easily result in different subword sequences within the same sentence, and different subword sequences can produce completely different ID sequence representations. Therefore, the present invention proposes an improved BPE algorithm, which combines the BPE algorithm with dependency syntax analysis and named entity recognition to screen different subword sequences of the same sentence and find the most suitable sequence. As shown in Figure 4, named entity recognition is first performed on the unstructured data to obtain entity label information. For example, named entity recognition is performed on "Wuhan Yangtze River Bridge Welcomes You", the entity label corresponding to Wuhan is "B-LOC", the entity label corresponding to the Yangtze River Bridge is "I-LOC", the entity label corresponding to Welcome is "O", and the entity label corresponding to You is "O".

[0067] At the same time, dependency syntax analysis is performed on the unstructured data to obtain labeled relationships. For example, the labeled relationship between the Yangtze River Bridge and Wuhan City is ATT, and the labeled relationship between Welcome and the Yangtze River Bridge is SBV, and so on.

[0068] Then, based on the entity tag information and the annotation relationship, the words in the unstructured data are encoded using the BPF algorithm to obtain second encoded data.

[0069] The training method of the natural language processing model provided by the present invention processes unstructured data through an improved BPE algorithm, and has better performance in Chinese.

[0070] Furthermore, based on the above embodiment, the pre-built natural language processing model is a Transformer-based neural network model, including a multi-layer fusion RNN Transformer layer; the pre-built natural language processing model is trained based on the encoded fusion data and expert experience data to obtain a trained natural language processing model, including:

[0071] The expert experience data is encoded to obtain encoded professional experience data.

[0072] The third encoded data is used as a query vector and a key vector, and the first encoded data and the second encoded data are used as a value vector.

[0073] The query vector, the key vector, the value vector and the encoded professional experience data are input into the Transformer layer of the multi-layer fusion RNN, so as to train and update the network parameters of the Transformer layer until a preset update stop condition is reached, thereby obtaining a trained natural language processing model.

[0074] In this embodiment, expert knowledge is treated independently from other heterogeneous data and integrated into the n-layer Transformer training structure as external data for training, enhancing the effectiveness of downstream tasks. To further enhance the memory capabilities of multi-turn dialogues, this invention improves the original Transformer structure by integrating recurrent neural networks (RNNs) into the Transformer training layer. This ensures that the encoded question also contains relevant information from the previous round, enabling an effective multi-turn dialogue model.

[0075] The training method for the natural language processing model provided by the present invention utilizes a multi-layer improved Transformer structure for feature extraction training and integrates expert knowledge into the model to further enhance knowledge and improve the effect of context connection in multi-round dialogues.

[0076] FIG5 is a flow chart of a method for generating a conversation subtitle according to an embodiment of the present invention. As shown in FIG5 , a method for generating a conversation subtitle includes the following steps:

[0077] S401, obtaining the conversation context provided by the user and the trained natural language processing model.

[0078] S402: Input the preceding conversation into the trained natural language processing model to obtain a following conversation corresponding to the preceding conversation.

[0079] The trained natural language processing model is obtained based on the aforementioned natural language processing model training method.

[0080] Specifically, keywords are extracted and expanded from the conversation text to obtain a question keyword vector.

[0081] The question keyword vector is input into the trained natural language processing model, and a database vector corresponding to the question keyword vector is obtained through multi-layer Transformer layer reasoning in the trained natural language processing model, and the conversation context is obtained based on the database vector.

[0082] The method for generating a conversation subtext provided by an embodiment of the present invention directly obtains the corresponding conversation subtext by inputting the conversation subtext into a trained natural language processing model, rather than retrieving relevant information from a corpus like existing question-answering systems. This improves computational efficiency and enhances the effectiveness of multi-round conversations.

[0083] The following describes the training device for the natural language processing model and the device for generating the continuation of a conversation provided by the present invention. The training device for the natural language processing model and the device for generating the continuation of a conversation described below can be referenced to the training method for the natural language processing model and the method for generating the continuation of a conversation described above.

[0084] The training device of the natural language processing model includes a data and model acquisition module, a data fusion module and a model training module.

[0085] The data and model acquisition module acquires a variety of heterogeneous sample data and pre-built natural language processing models. The heterogeneous sample data includes structured data, unstructured data, knowledge graphs, and expert experience data.

[0086] Structured data is highly organized and neatly formatted data. Also known as quantitative data, structured data is information that can be represented by numbers or a uniform structure, such as numbers or symbols. In data processing, this data is typically stored and managed in a relational database. When using Structured Query Language, or SQL, computer programs can easily search for these terms.

[0087] Unstructured data is essentially any data other than structured data. It does not conform to any predefined model. It may be textual or non-textual, and may be human-generated or machine-generated. Simply put, unstructured data is data with variable fields.

[0088] Knowledge graph is a form of expressing entities through relationships and attributes. Google Knowledge Graph's slogan "things not strings" gives the essence of knowledge graph, namely: the object or thing hidden behind the string, this object is supported by relationships and attributes to determine that it is this entity and not another entity.

[0089] The data fusion module performs data encoding fusion on the structured data, unstructured data and knowledge graph to obtain encoded fusion data.

[0090] In this module, different data encodings are performed for different types of heterogeneous data. The specific process is described below.

[0091] The model training module trains the pre-built natural language processing model based on the encoded fusion data and expert experience data to obtain a trained natural language processing model.

[0092] The natural language processing model training device provided by the embodiments of the present invention integrates structured data, unstructured data, knowledge graphs, and expert experience as sample data, enabling better model training, enhancing context-based text reasoning capabilities, and improving the accuracy of multi-round conversations. Furthermore, the knowledge graph no longer serves as an independent retrieval corpus, but rather as a knowledge enhancement within the model.

[0093] The device for generating a dialogue context includes a context and model acquisition module and a context output module.

[0094] Among them, the context and model acquisition module obtains the conversation context provided by the user and the trained natural language processing model.

[0095] The following text output module inputs the above conversation context into the trained natural language processing model to obtain the following conversation context corresponding to the above conversation context.

[0096] The trained natural language processing model is obtained based on the aforementioned natural language processing model training method.

[0097] Specifically, keywords are extracted and expanded from the conversation text to obtain a question keyword vector.

[0098] The question keyword vector is input into the trained natural language processing model, and a database vector corresponding to the question keyword vector is obtained through multi-layer Transformer layer reasoning in the trained natural language processing model, and the conversation context is obtained based on the database vector.

[0099] The device for generating a conversation context provided by an embodiment of the present invention directly obtains the corresponding conversation context by inputting the conversation context into a trained natural language processing model, rather than retrieving relevant information from a corpus like existing question-answering systems, thereby improving computing efficiency and enhancing the effectiveness of multi-round conversations.

[0100] FIG6 illustrates a schematic diagram of the physical structure of an electronic device. As shown in FIG6 , the electronic device may include: a processor 610 (processor), a communication interface 620 (Communications Interface), a memory 630 (memory), and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the above-mentioned training method for the natural language processing model, which includes: obtaining a plurality of heterogeneous sample data and a pre-built natural language processing model, wherein the heterogeneous sample data includes structured data, unstructured data, a knowledge graph, and expert experience data; performing data encoding and fusion on the structured data, unstructured data, and knowledge graph to obtain encoded fusion data; and training the pre-built natural language processing model based on the encoded fusion data and expert experience data to obtain a trained natural language processing model.

[0101] Or a method for generating a conversation context, the method comprising: obtaining a conversation context provided by a user and a trained natural language processing model; inputting the conversation context into the trained natural language processing model to obtain a conversation context corresponding to the conversation context; wherein the trained natural language processing model is obtained based on the above-mentioned natural language processing model training method.

[0102] Furthermore, the logic instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0103] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training method for the natural language processing model provided above, the method comprising: obtaining a variety of heterogeneous sample data and a pre-built natural language processing model, the heterogeneous sample data comprising structured data, unstructured data, a knowledge graph, and expert experience data; performing data encoding and fusion on the structured data, unstructured data, and knowledge graph to obtain encoded fusion data; and training the pre-built natural language processing model based on the encoded fusion data and expert experience data to obtain a trained natural language processing model.

[0104] Or a method for generating a conversation context, the method comprising: obtaining a conversation context provided by a user and a trained natural language processing model; inputting the conversation context into the trained natural language processing model to obtain a conversation context corresponding to the conversation context; wherein the trained natural language processing model is obtained based on the above-mentioned natural language processing model training method.

[0105] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0106] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for training a natural language processing model, characterized in that: include: Acquire a variety of heterogeneous sample data and a pre-built natural language processing model, wherein the heterogeneous sample data includes structured data, unstructured data, knowledge graphs, and expert experience data; Performing data encoding fusion on the structured data, unstructured data and knowledge graph to obtain encoded fusion data; The pre-built natural language processing model is trained based on the encoded fusion data and the expert experience data to obtain a trained natural language processing model.

2. The method for training a natural language processing model according to claim 1, characterized in that: The step of performing data encoding fusion on the structured data, the unstructured data and the knowledge graph to obtain encoded fusion data includes: Performing vector conversion on the structured data using a preset first encoding algorithm to obtain first encoded data; Performing vector conversion on the unstructured data using a preset first encoding algorithm and / or dependency syntax analysis or a second encoding algorithm to obtain second encoded data; A preset graph convolutional neural network is used to extract features from the knowledge graph, and a graph attention mechanism is used to enhance the extracted features to obtain third encoded data; wherein the encoded fusion data includes first encoded data, second encoded data, and third encoded data, the first encoding algorithm is at least one of TF-IDF, Word2vec or One-hot, and the second encoding algorithm is a BPE-based encoding algorithm.

3. The method for training a natural language processing model according to claim 2, characterized in that: Performing vector conversion on the unstructured data using a second encoding algorithm to obtain second encoded data includes: Performing named entity recognition on the unstructured data to obtain entity label information; Performing dependency syntactic analysis on the unstructured data to obtain a labeling relationship; Based on the entity tag information and the annotation relationship, the words in the unstructured data are encoded using the BPF algorithm to obtain second encoded data.

4. The method for training a natural language processing model according to claim 2, characterized in that: The pre-built natural language processing model is a Transformer-based neural network model, including a Transformer layer of a multi-layer fusion RNN; the pre-built natural language processing model is trained based on the encoded fusion data and the expert experience data to obtain a trained natural language processing model, including: Encoding the expert experience data to obtain encoded professional experience data; Using the third encoded data as a query vector and a key vector, and the first encoded data and the second encoded data as a value vector; The query vector, the key vector, the value vector and the encoded professional experience data are input into the Transformer layer of the multi-layer fusion RNN, so as to train and update the network parameters of the Transformer layer until a preset update stop condition is reached, thereby obtaining a trained natural language processing model.

5. A method for generating a conversation context, characterized in that: include: Obtain the conversation context provided by the user and the trained natural language processing model; The conversation context is input into the trained natural language processing model to obtain a conversation context corresponding to the conversation context; wherein the trained natural language processing model is obtained based on the training method of the natural language processing model as described in any one of claims 1-4.

6. The method for generating a dialogue context according to claim 5, characterized in that: The step of inputting the conversation context into the trained natural language processing model to obtain a conversation context corresponding to the conversation context includes: Extracting and expanding keywords from the conversation context to obtain a question keyword vector; The question keyword vector is input into the trained natural language processing model, a database vector corresponding to the question keyword vector is obtained through multi-layer Transformer layer reasoning in the trained natural language processing model, and the conversation context is obtained based on the database vector.

7. A training device for a natural language processing model, characterized in that: include: A data and model acquisition module, used to acquire a variety of heterogeneous sample data and pre-built natural language processing models, wherein the heterogeneous sample data includes structured data, unstructured data, knowledge graphs, and expert experience data; A data fusion module, used to perform data encoding fusion on the structured data, unstructured data and knowledge graph to obtain encoded fusion data; The model training module is used to train the pre-built natural language processing model based on the encoded fusion data and the expert experience data to obtain a trained natural language processing model.

8. A device for generating a dialogue context, characterized in that: include: The context and model acquisition module is used to obtain the conversation context provided by the user and the trained natural language processing model; A context output module is used to input the conversation context into the trained natural language processing model to obtain a conversation context corresponding to the conversation context; wherein the trained natural language processing model is obtained based on the training device of the natural language processing model as described in claim 7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor executes the computer program to implement the training method of the natural language processing model as described in any one of claims 1-4 or the method for generating dialogue context as described in claim 5 or 6.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it is the method for training a natural language processing model as described in any one of claims 1 to 4 or the method for generating a dialogue context as described in claim 5 or 6.

Images