Named entity recognition method and equipment based on large language model

Abstract

The invention aims to provide a named entity recognition method and device based on a large language model, and the method comprises the steps: carrying out the data cleaning of a to-be-trained text, carrying out the data enhancement of the cleaned data, and obtaining target data; the lora structure is fused into a large language model LLaMA model, and a new large language model is obtained; training the new large language model by using target data to obtain a target model; and extracting a target entity from the text by using the target model to complete identification of the named entity. Therefore, the problem of insufficient effect caused by less parameter quantity and knowledge in the traditional method is solved, the training data is increased, the effect of the model is improved, the training speed of the model is improved by adopting lora, and the problem of insufficient training resources is solved.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a method and device for named entity recognition based on a large language model. Background Art

[0002] Named Entity Recognition (NER) is an important task in the field of natural language processing (NLP). Its goal is to identify named entities with specific meanings from text, such as names of people, places, and organizations. NER is of great significance for applications such as information extraction, question-answering systems, and machine translation, because it can help computers understand the contextual relationships and semantic information of entities in text.

[0003] At present, the mainstream named entity recognition methods mainly include three technical routes, namely rule-based methods, statistical model-based methods, and deep learning-based methods.

[0004] Among them, rule-based methods identify entities through pre-defined rules, dictionaries or patterns. This method is suitable for some specific fields or languages, especially when the entities have obvious patterns and regularities, it can achieve higher accuracy. However, this method is usually manually designed for specific fields or tasks and lacks generalization. When faced with new fields or diverse contexts, the rules need to be constantly updated and expanded, and the maintenance cost is high.

[0005] Statistical model-based methods use statistical principles to model text sequences in named entity recognition to capture the probability distribution and contextual relationship of entity occurrence; among them, hidden Markov model (HMM), conditional random field (CRF) and maximum entropy model (MaxEnt) are representative statistical models. HMM labels the sequence by considering the state transition probability, CRF optimizes the labeling sequence globally, and MaxEnt learns the entity recognition model by maximizing entropy. These methods have better versatility than rule-based methods. However, statistical model-based methods have difficulties in handling long-distance dependencies and complex feature engineering. Statistical models may perform poorly when handling long-distance dependencies, especially in sequence labeling tasks, where the model may find it difficult to capture long-distance contextual information. When using statistical models, features need to be carefully designed and selected, which usually requires domain expertise and experience.

[0006] The named entity recognition method based on traditional deep learning uses neural network models for feature learning. These models include traditional recurrent neural networks (RNNs), improved long short-term memory networks (LSTMs), bidirectional long short-term memory networks (BiLSTMs), and pre-trained language models. These networks can automatically capture the semantic and contextual information in the input text, thereby improving the recognition accuracy of named entities. Among them, RNN processes sequence data recursively, and LSTM introduces a gating mechanism to more effectively handle long-distance dependencies. BiLSTM makes up for the shortcomings of traditional RNN in modeling long-distance relationships in sequences by considering both forward and backward contextual information. The core idea of ​​the named entity recognition method based on pre-trained language models is to use pre-trained models on large-scale unsupervised corpora to learn rich language representations, including semantic relationships and contextual information between words. BERT is one of the successful pre-trained models. Its bidirectional attention mechanism allows the model to use its previous and next contexts when considering a word, so as to better capture the association between words. When applied to named entity recognition tasks, pre-trained language models can be customized for specific fields or tasks through a fine-tuning phase. During the fine-tuning process, the model receives labeled named entity recognition data and adjusts its parameters to adapt it to the requirements of a specific task. Since the pre-trained model has learned a general language representation, this method can achieve good performance on a relatively small labeled data set, especially in areas where there is a lack of a large amount of labeled data. Although traditional deep learning has made some progress in named entity recognition, it still has the following difficult-to-solve problems:

[0007] 1) Requirement for large amounts of annotated data: Deep learning methods usually require a large amount of annotated data for training, and in some fields, obtaining large-scale annotated data can be expensive and time-consuming.

[0008] 2) The model is relatively simple: Traditional deep learning methods generally have fewer parameters and can contain limited information. Therefore, their performance is average for slightly more complex tasks.

[0009] 3) Unable to process long texts: The input of pre-trained language models usually has a length limit, which may lead to certain limitations in the processing of long texts. When processing texts that exceed the maximum input length of the model, truncation, segmentation, etc. are required, which may affect the performance of the model. Summary of the invention

[0010] One purpose of the present application is to provide a method and device for named entity recognition based on a large language model to solve the problems in the prior art such as lack of generalization, difficulty in handling long-distance dependencies, complex feature engineering, requirement for a large amount of annotated data, and inability to handle long texts.

[0011] According to one aspect of the present application, a method for named entity recognition based on a large language model is provided, the method comprising:

[0012] Perform data cleaning on the training text, and perform data enhancement on the cleaned data to obtain the target data;

[0013] Integrate the lora structure into the large language model LLaMA model to obtain a new large language model;

[0014] Using the target data to train the new large language model to obtain a target model;

[0015] The target model is used to extract the target entity from the text.

[0016] Optionally, perform data cleaning on the training text, including:

[0017] Based on Python, the noise data in the text to be trained is removed, and the text after noise removal is processed based on regular expressions and rules to obtain the cleaned data.

[0018] Optionally, perform data enhancement on the cleaned data, including:

[0019] Step a, performing synonym replacement on the cleaned data;

[0020] Step b, converting the cleaned data into multiple languages;

[0021] Step c, performing GPT4 conversion on the cleaned data;

[0022] Repeat the above steps a to c multiple times to generate multiple different corpora and obtain the target data.

[0023] Optionally, the lora structure is integrated into the large language model LLaMA model, including:

[0024] The values ​​of all parameters of LLaMA 7B were kept unchanged during the training process;

[0025] Add two LoRa adapters to each of the original transformer blocks;

[0026] Initialize the parameters in each lora adapter, activate all initialized parameters to accept gradient updates, and save the LLaMA model with the lora structure added.

[0027] Optionally, initialize parameters in each LoRa adapter, including:

[0028] The parameters in each lora adapter are initialized by random initialization, where the parameters in the adapter include a matrix of dimension h×r and a matrix of r×h, where h is the dimension of the input and r is a pre-configured hyperparameter.

[0029] Optionally, each of the original transformer blocks includes a self-attention layer and a feed-forward neural network layer, and two LoRa adapters are added to each of the original transformer blocks, including:

[0030] One LoRa adapter is added after the self-attention layer and another LoRa adapter is added after the feed-forward neural network layer.

[0031] Optionally, training the new large language model using the target data includes:

[0032] Use Python to process the target data into data that conforms to the training format and obtain the target format data;

[0033] Inputting the target format data into the new large language model, and calculating the loss of the new large language model under the target format data using a cross entropy loss function;

[0034] The obtained loss is back-propagated through the new large language model and all activated parameters are updated.

[0035] According to another aspect of the present application, a named entity recognition device based on a large language model is also provided, the device comprising:

[0036] one or more processors; and

[0037] A memory storing computer readable instructions, which, when executed, cause the processor to perform the operations of the method as described above.

[0038] According to another aspect of the present application, a computer-readable medium is provided, on which computer-readable instructions are stored. The computer-readable instructions can be executed by a processor to implement the method described above.

[0039] Compared with the prior art, this application performs data cleaning on the training text, performs data enhancement on the cleaned data, obtains the target data; integrates the lora structure into the large language model LLaMA model to obtain a new large language model; uses the target data to train the new large language model to obtain the target model; uses the target model to extract the target entity from the text. This solves the problem of insufficient effect of the traditional method due to the small number of parameters and knowledge, increases the training data, improves the effect of the model, uses lora to improve the training speed of the model, and solves the problem of insufficient training resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0041] Figure 1 A schematic diagram of a process flow of a named entity recognition method based on a large language model provided according to one aspect of the present application is shown;

[0042] Figure 2 A schematic diagram showing the principle of adding lora architecture to LLaMA in one embodiment of the present application;

[0043] Figure 3 A schematic diagram showing data enhancement in an embodiment of the present application is shown;

[0044] Figure 4 A schematic diagram showing the principle of the LLaMA 7B model in one embodiment of the present application is shown;

[0045] Figure 5 A schematic diagram showing how lora is integrated into the LLaMA model in one embodiment of the present application;

[0046] Figure 6 A schematic diagram showing the training data input format in an embodiment of the present application.

[0047] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0048] The present application is described in further detail below in conjunction with the accompanying drawings.

[0049] In a typical configuration of the present application, the terminal, the device of the service network and the trusted party all include one or more processors (eg, a central processing unit (CPU)), an input / output interface, a network interface and a memory.

[0050] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0051] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, Phase-Change RAM (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of random access memory (RAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0052] Figure 1 A schematic flow chart of a method for named entity recognition based on a large language model according to one aspect of the present application is shown, the method comprising: steps S11 to S14, wherein:

[0053] Step S11, data cleaning is performed on the text to be trained, and data enhancement is performed on the cleaned data to obtain target data; here, given the text to be trained, data cleaning is performed on the text to be trained, including denoising and removing useless strings, and data enhancement is performed to obtain the required enhanced data, i.e., target data, for subsequent model training.

[0054] Step S12, integrate the lora structure into the large language model LLaMA model to obtain a new large language model; here, construct the model to be trained, transform the large language model LLaMA model, and integrate the lora structure into the LLaMA model, such as Figure 2As shown, a schematic diagram of the principle of adding lora architecture to LLaMA. Among them, LLaMA is a large basic language model, which is an open and efficient large basic language model, with four versions of 7B (7 billion parameters), 13B (13 billion parameters), 33B (33 billion parameters), and 65B (65 billion parameters). LLaMA adopts a deep neural network model based on the Transformer architecture; LLaMA 7B is used as the basic model in the embodiment of the present application, and lora is integrated into it. Lora (Low-Rank Adaptation of Large Language Models) is a low-rank adaptation model, a training method that can accelerate the training of large language models and has lower memory consumption.

[0055] Step S13, the target data is used to train the new large language model to obtain a target model; here, the data after data enhancement, i.e., the target data, is used to train the improved new large language model to obtain a model for named entity recognition. Thus, in step S14, the target model is used to extract the target entity from the text, complete the recognition of the named entity, and recognize the target entities such as names of people, places, and organizational structures from the text.

[0056] In one embodiment of the present application, in step S11, noise data in the text to be trained is removed based on Python, and the text after noise removal is processed based on regular expressions and rules to obtain cleaned data. Here, noise data in the data is removed based on Python, wherein the noise data includes repeated text, empty text, garbled text, too long text or too short text. For the text after noise removal, useless strings such as html tags, @, and emoticons are removed based on regular expressions and rules.

[0057] Next, the cleaned data is enhanced. The data enhancement methods include synonym conversion, multilingual conversion, and GPT4 conversion. Specifically, step a replaces the cleaned data with synonyms; step b converts the cleaned data into multiple languages; step c converts the cleaned data into GPT4; repeat the above steps a to c multiple times to generate multiple different corpora to obtain the target data. Figure 3 As shown in the figure, synonym conversion, for the cleaned text, a word is randomly selected after word segmentation, and it is replaced with a synonym using the modern Chinese dictionary to obtain a new data, such as Figure 3In the example, the word "love" in "I love Beijing XXX" is converted to "like", and a new piece of data is obtained, namely "I like Beijing XXX"; when performing multi-language conversion, the cleaned Chinese text is translated into English using translation software, and then translated back into Chinese using translation software to obtain a new piece of data. For example, Beijing XXX is a square-like place. After translation and then translation back, "I love Beijing XXX" is converted into "I like Beijing's XXX Square"; when performing GPT4 transcription, the cleaned text is transcribed into a synonymous sentence containing the same entity using GPT4, and a new annotated data is generated, such as "I love Beijing XXX" is transcribed into "I have a deep love for Beijing XXX". Among them, GPT4 is a first-generation natural language processing model, which can generate high-quality text content based on deep learning technology, simulate human dialogue, and support a variety of natural language tasks, such as language translation, sentiment analysis, question answering, etc. Repeat the above steps of synonym replacement, multilingual conversion and GPT4 transcription N times to generate multiple different corpora for subsequent model training, improve the performance and generalization ability of the model, and improve the performance of the model by expanding the training data set. Even when the amount of labeled data is small, a large amount of labeled corpora can be quickly obtained. Use a variety of data enhancement methods to increase training data and improve the effect of the model.

[0058] In one embodiment of the present application, in step S12, the values ​​of all parameters of LLaMA 7B are kept unchanged during the training process; two lora adapters are added to each original transformer block; the parameters in each lora adapter are initialized, and all initialized parameters are activated to accept gradient updates, and the LLaMA model with the lora structure added is saved. The parameters in each lora adapter are initialized by random initialization, wherein the parameters in the adapter include a matrix of dimension h×r and a matrix of r×h, where h is the input dimension and r is a pre-configured hyperparameter. Here, LLaMA 7B is used as the basic model and LoRa is integrated into it. The specific steps include: 1) In the deep learning framework PyTorch, all parameters of LLaMA 7B are frozen, and the values ​​of all parameters of LLaMA 7B are kept unchanged during training so that they do not accept gradient updates; 2) Two LoRa adapters are added to each original transformer block; 3) The parameters of the two added adapters are initialized by random initialization. The parameters of each adapter contain a matrix of dimension h×r and a matrix of dimension r×h, where h is the input dimension and r is a pre-set hyperparameter, and all initialized parameters are activated to accept gradient updates; d) Save LLaMA 7B with LoRa added.

[0059] Continuing with the above embodiment, each original transformer block includes a self-attention layer and a feedforward neural network layer. When two lora adapters are added to each original transformer block, one lora adapter is added after the self-attention layer, and the other lora adapter is added after the feedforward neural network layer. Figure 4 As shown in Figure 5, the LLaMA 7B model includes 32 transformer blocks, which are connected in series in sequence, and then connected to the normalization layer, linear layer, and SoftMax layer in sequence. Among them, a layer of the transformer model, such as the self-attention layer, residual connection, and other structures, is collectively referred to as a transformer block. Each transformer block includes a self-attention layer and a feedforward neural network layer. One adapter is placed after the self-attention layer and the other is placed after the feedforward neural network layer. Specifically, as shown in Figure 5, the self-attention layer is followed by a linear layer, and the linear layer is followed by an adapter. The adapter is followed by a normalization layer and a linear layer, and then another adapter is connected, and finally to the normalization layer. Therefore, when using this structure for training, while ensuring the effect, by reducing the number of trainable parameters, the training cost and training time can be greatly reduced.

[0060] In one embodiment of the present application, in step S13, the target data is processed into data in a training format using python to obtain the target format data; the target format data is input into a new large language model, and the cross entropy loss function is used to calculate the loss of the new large language model under the target format data; the obtained loss is back-propagated in the new large language model and all activated parameters are updated. Here, the named entity recognition model is trained using the prepared data and model, and the data after data enhancement is processed into data in a training format using python, such as Figure 6 The format shown; send the data into the initialized LLaMA model that has been integrated into the lora structure, that is, send the data into the new large language model, and use the cross entropy loss function to calculate the loss lossl of the new large language model under the current data. The calculation method of the cross entropy loss function l is as follows:

[0061]

[0062] Among them, N represents the number of samples, i represents the i-th sample, and y ic represents the label (0 or 1) of the i-th sample c, p icis the probability of the i-th sample category c. The loss loss is back-propagated using gradient descent, and all activated parameters are updated so that the neural network can learn how to adjust the parameters to minimize the loss, thereby improving the generalization ability of new data and adapting to complex data distribution. Save the updated model. In the embodiment of the present application, a large language model is used for training, which has a stronger language understanding ability and can greatly improve the named entity recognition effect.

[0063] In addition, an embodiment of the present application also provides a computer-readable medium on which computer-readable instructions are stored. The computer-readable instructions can be executed by a processor to implement the aforementioned named entity recognition method based on a large language model.

[0064] In one embodiment of the present application, a named entity recognition device based on a large language model is also provided, the device comprising:

[0065] one or more processors; and

[0066] A memory storing computer readable instructions, which, when executed, cause the processor to perform the operations of the method as described above.

[0067] For example, the computer readable instructions, when executed, cause the one or more processors to:

[0068] Perform data cleaning on the training text, and perform data enhancement on the cleaned data to obtain the target data;

[0069] Integrate the lora structure into the large language model LLaMA model to obtain a new large language model;

[0070] Using the target data to train the new large language model to obtain a target model;

[0071] The target model is used to extract the target entity from the text.

[0072] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

[0073] It should be noted that the present application can be implemented in software and / or a combination of software and hardware, for example, can be implemented using an application specific integrated circuit (ASIC), a general purpose computer or any other similar hardware device. In one embodiment, the software program of the present application can be executed by a processor to implement the steps or functions described above. Similarly, the software program of the present application (including relevant data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and similar devices. In addition, some steps or functions of the present application can be implemented using hardware, for example, as a circuit that cooperates with a processor to perform each step or function.

[0074] In addition, a part of the present application may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present application through the operation of the computer. The program instruction for calling the method of the present application may be stored in a fixed or removable recording medium, and / or transmitted through a data stream in a broadcast or other signal-bearing medium, and / or stored in a working memory of a computer device that runs according to the program instruction. Here, according to an embodiment of the present application, a device is included, the device including a memory for storing computer program instructions and a processor for executing program instructions, wherein, when the computer program instruction is executed by the processor, the device is triggered to run the method and / or technical solution based on the aforementioned multiple embodiments according to the present application.

[0075] It is obvious to those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and that the present application can be implemented in other specific forms without departing from the spirit or basic features of the present application. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present application. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A method for named entity recognition based on a large language model, characterized in that: The method comprises: Perform data cleaning on the training text, and perform data enhancement on the cleaned data to obtain the target data; Integrate the lora structure into the large language model LLaMA model to obtain a new large language model; Using the target data to train the new large language model to obtain a target model; The target entity is extracted from the text using the target model.

2. The method according to claim 1, characterized in that Perform data cleaning on the training text, including: Based on Python, the noise data in the text to be trained is removed, and the text after noise removal is processed based on regular expressions and rules to obtain the cleaned data.

3. The method according to claim 1 or 2, characterized in that: Perform data enhancement on the cleaned data, including: Step a, performing synonym replacement on the cleaned data; Step b, converting the cleaned data into multiple languages; Step c, performing GPT4 conversion on the cleaned data; Repeat the above steps a to c multiple times to generate multiple different corpora and obtain the target data.

4. The method according to claim 1, characterized in that: Integrate the lora structure into the large language model LLaMA model, including: The values ​​of all parameters of LLaMA 7B were kept unchanged during the training process; Add two LoRa adapters to each of the original transformer blocks; Initialize the parameters in each lora adapter, activate all initialized parameters to accept gradient updates, and save the LLaMA model with the lora structure added.

5. The method according to claim 4, characterized in that Initialize the parameters in each lora adapter, including: The parameters in each lora adapter are initialized by random initialization, where the parameters in the adapter include a matrix of dimension h×r and a matrix of r×h, where h is the dimension of the input and r is a pre-configured hyperparameter.

6. The method according to claim 4, characterized in that Each original transformer block includes a self-attention layer and a feed-forward neural network layer. Two lora adapters are added to each original transformer block, including: One LoRa adapter is added after the self-attention layer and another LoRa adapter is added after the feed-forward neural network layer.

7. The method according to claim 1, characterized in that The new large language model is trained using the target data, including: Use Python to process the target data into data that conforms to the training format and obtain the target format data; Inputting the target format data into the new large language model, and calculating the loss of the new large language model under the target format data using a cross entropy loss function; The obtained loss is back-propagated through the new large language model and all activated parameters are updated.

8. A named entity recognition device based on a large language model, characterized in that: The device comprises: one or more processors; and A memory storing computer readable instructions which, when executed, cause the processor to perform the operations of the method as claimed in any one of claims 1 to 7.

9. A computer-readable medium having computer-readable instructions stored thereon, wherein the computer-readable instructions can be executed by a processor to implement the method according to any one of claims 1 to 7.

Images