Named entity recognition method, training method, device, equipment and medium

Abstract

The present application discloses a named entity recognition method, training method, apparatus, equipment and medium. It belongs to the field of artificial intelligence. The method comprises: obtaining input text, the input text comprising at least one character; performing character encoding on each character in the input text to obtain a first character encoding vector for each character; based on the first character encoding vector of each character, predicting a first probability that each character is the starting character of a candidate entity type, and a second probability that each character is the ending character of the candidate entity type; and identifying named entities belonging to the candidate entity type based on the first probability and the second probability. The present application can achieve higher recognition accuracy and recognition speed in named entity recognition scenarios involving multiple industries and multiple entities.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a named entity recognition method, training method, apparatus, equipment and medium. Background Art

[0002] Named Entity Recognition (NER) refers to the identification of entities with specific meanings in text, mainly including names of people, places, organizations, proper nouns and other characters.

[0003] In related technologies, named entity recognition is often solved using a Bi-directional Long Short-Term Memory (Bi-LSTM) + Conditional Random Field (CRF) model. Bi-LSTM + CRF has excellent context modeling capabilities.

[0004] The above solution requires modeling entity types and entity boundaries at the same time. In the named entity recognition scenario of multiple industries and multiple entities, the decoding speed and decoding accuracy are poor. Summary of the Invention

[0005] This application provides a named entity recognition method, training method, apparatus, device, and medium, providing a technical solution with fast decoding speed and good decoding accuracy in named entity recognition scenarios across multiple industries and entities. The technical solution is as follows:

[0006] According to one aspect of the present application, a method for named entity recognition is provided, the method comprising:

[0007] Obtaining input text, where the input text includes at least one character;

[0008] Performing character encoding on each character in the input text to obtain a first character encoding vector for each character;

[0009] Based on the first character encoding vector of each character, predict a first probability that each character is a start character of a candidate entity type, and a second probability that each character is an end character of the candidate entity type;

[0010] A named entity belonging to the candidate entity type is identified based on the first probability and the second probability.

[0011] According to another aspect of the present application, a method for training a named entity model is provided, wherein the named entity model includes a language model and a bigram model. The method includes:

[0012] Obtaining a training set, the training set including sample text and sample named entities in the sample text;

[0013] Calling the pre-trained language model to perform character encoding on each character in the sample text to obtain a first character encoding vector for each character;

[0014] Calling the bigram model to predict, based on the first character encoding vector of each character, a first probability that each character is a start character of a candidate entity type and a second probability that each character is an end character of the candidate entity type; identifying a predicted named entity belonging to the candidate entity type based on the first probability and the second probability;

[0015] Based on the error between the predicted named entity and the sample named entity, network parameters of the language model and the bigram model are trained.

[0016] According to another aspect of the present application, a named entity recognition device is provided, the device comprising:

[0017] An acquisition module, configured to acquire an input text, wherein the input text includes at least one character;

[0018] an encoding module, configured to perform character encoding on each character in the input text to obtain a first character encoding vector for each character;

[0019] a prediction module, configured to predict, based on the first character encoding vector of each character, a first probability that each character is a start character of a candidate entity type and a second probability that each character is an end character of the candidate entity type;

[0020] An identification module is configured to identify a named entity belonging to the candidate entity type based on the first probability and the second probability.

[0021] According to another aspect of the present application, a training device for a named entity model is provided, wherein the named entity model includes: a language model and a bigram model, and the device includes:

[0022] An acquisition module, configured to acquire a training set, wherein the training set includes sample texts and sample named entities in the sample texts;

[0023] An encoding module, configured to call the pre-trained language model to perform character encoding on each character in the sample text to obtain a first character encoding vector for each character;

[0024] a prediction module, configured to call the bigram model to predict, based on the first character encoding vector of each character, a first probability that each character is a start character of a candidate entity type and a second probability that each character is an end character of the candidate entity type; and identify a predicted named entity belonging to the candidate entity type based on the first probability and the second probability;

[0025] A training module is used to train network parameters of the language model and the bigram model based on the error between the predicted named entity and the sample named entity.

[0026] According to another aspect of the present application, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the named entity recognition method described above.

[0027] According to another aspect of the present application, a computer-readable storage medium is provided, in which at least one program code is stored. The program code is loaded and executed by a processor to implement the named entity recognition method described above.

[0028] According to another aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the named entity recognition method provided in various optional implementations of the above aspects.

[0029] The beneficial effects of the technical solution provided by this application include at least:

[0030] By encoding each character in the input text and predicting the first probability that each character is the starting character of the candidate entity type and the second probability that each character is the ending character of the candidate entity type based on the first character encoding vector of each character, the named entities belonging to the candidate entity type are identified. Since the start and end of the named entities are modeled hierarchically, the training process does not need to introduce entity boundary labels, thereby reducing the number of training labels in multi-industry and multi-entity naming recognition scenarios and improving the recognition speed and accuracy of named entity recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0032] Figure 1 is a schematic diagram of a computer system provided by an exemplary embodiment of the present application;

[0033] Figure 2 is a schematic diagram of a named entity recognition method provided by an exemplary embodiment of the present application;

[0034] Figure 3 This is a schematic diagram of another method for named entity recognition provided by an exemplary embodiment of the present application;

[0035] Figure 4 is a schematic diagram of a named entity recognition method provided by an exemplary embodiment of the present application;

[0036] Figure 5 This is a schematic diagram of the structure of the BERT model provided by an exemplary embodiment of the present application;

[0037] Figure 6 is a schematic diagram of the structure of a fully connected layer provided by an exemplary embodiment of the present application;

[0038] Figure 7 is a schematic structural diagram of a named entity recognition model provided by an exemplary embodiment of the present application;

[0039] Figure 8 is a schematic diagram of a named entity recognition scenario provided by an exemplary embodiment of the present application;

[0040] Figure 9 This is a schematic diagram of the structure of a user portrait tag provided by an exemplary embodiment of the present application;

[0041] Figure 10 This is a structural diagram of an advertisement push system provided by an exemplary embodiment of the present application;

[0042] Figure 11 is a schematic diagram of a training method for a named entity recognition model provided by an exemplary embodiment of the present application;

[0043] Figure 12 is a block diagram of a distributed system provided by an exemplary embodiment of the present application;

[0044] Figure 13is a schematic diagram of a block relationship between multiple blocks provided by an exemplary embodiment of the present application;

[0045] Figure 14 is a schematic diagram of a named entity recognition device provided by an exemplary embodiment of the present application;

[0046] Figure 15 is a schematic diagram of a named entity recognition device provided by an exemplary embodiment of the present application;

[0047] Figure 16 It is a structural diagram of a computer device provided by an exemplary embodiment of the present application.

[0048] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application. DETAILED DESCRIPTION

[0049] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0050] Before describing the various embodiments shown in this application, several concepts involved in this application are first introduced:

[0051] 1) Artificial Intelligence (AI)

[0052] Artificial intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0053] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0054] 2) Computer Vision (CV)

[0055] Computer vision is the science of making machines "see." Specifically, it refers to using cameras and computers to replace the human eye in identifying, tracking, and measuring objects. Further image processing is performed to transform the computer's image into an image more suitable for human observation or transmission to instrumentation. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies generally include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / action recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and other technologies. Common biometric recognition technologies include facial recognition and fingerprint recognition.

[0056] 3) Machine Learning (ML)

[0057] Machine learning is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0058] 4) Natural Language Processing (NLP) is a key area of research in computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. Natural Language Processing (NLP) integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language we use in everyday life—and is closely linked to the study of linguistics. Natural Language Processing technologies typically include text processing, semantic understanding, machine translation, robotic question answering, and knowledge graphs.

[0059] The solution of the embodiment of the present application includes a model training stage and a named entity recognition stage. Figure 1 FIG. 1 is a framework diagram of a computer system according to an exemplary embodiment. Figure 1As shown, in the model training phase, the model training device 110 trains a highly accurate named entity recognition model using a pre-set training sample set. In the named entity recognition phase, the named entity recognition device 120 identifies named entities of multiple industries and entities based on the trained named entity recognition model and the input target text.

[0060] Among them, the above-mentioned model training device 110 and named entity recognition device 120 can be computer devices with machine learning capabilities, for example, the computer device can be a terminal or a server.

[0061] Optionally, the above-mentioned model training device 110 and named entity recognition device 120 can be the same device, or the model training device 110 and named entity recognition device 120 can also be different devices. Moreover, when the model training device 110 and the named entity recognition device 120 are different devices, the model training device 110 and the named entity recognition device 120 can be devices of the same type, such as the model training device 110 and the named entity recognition device 120 can both be servers; or, the model training device 110 and the named entity recognition device 120 can also be devices of different types. The above-mentioned server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The above-mentioned terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server may be connected directly or indirectly via wired or wireless communication, which is not limited in this application.

[0062] Figure 2 A flowchart of a named entity recognition method provided by an exemplary embodiment of the present application is shown. Figure 1 The method is performed by the named entity recognition device 120 shown for illustration. The method includes:

[0063] Step 202: Obtain input text, where the input text includes at least one character;

[0064] The input text is the text for which named entity recognition is to be performed. The input text includes one or more characters. In one example, the input text is text in a user profile of a user account.

[0065] Step 204: performing character encoding on each character in the named entity recognition input text to obtain a first character encoding vector for each character;

[0066] Exemplary references Figure 3 , the named entity recognition model includes: a language model 122 and a binary model 124.

[0067] The standard definition of language model 122 is: for the language sequence W1, W2, ..., W n , a model for calculating the probability of the language sequence, W represents the character, and the subscript represents the order in which the character appears in the language sequence. From the perspective of machine learning, the language model is a model for modeling the probability distribution of each word in a sentence. A more popular understanding is that a language model is a model for judging whether a language sequence is a normal sentence, that is, judging whether a language sequence is a human language. Optionally, the language model is at least one of a Bidirectional Encoder Representations from Transformers (BERT) model, an Enhanced Representation from kNowledge IntEgration (ERNIE) model, an Autoregressive Language Model (XLNET), and a Span-BERT.

[0068] The bigram 124 is a classification model comprising two classification layers with an association between the two classification layers. In this embodiment, the bigram 124 is a model for classifying whether the current character is the start character of a candidate entity type and whether the current character is the end character of a candidate entity type.

[0069] The language model 122 is called to perform character encoding on each character in the named entity recognition input text to obtain a first character encoding vector for each character.

[0070] Step 206: Based on the first character encoding vector of each character, predict a first probability that each character is the start character of the candidate entity type, and a second probability that each character is the end character of the candidate entity type in named entity recognition;

[0071] The calling bigram 124 predicts a first probability that each character of the named entity recognition is a start character of a candidate entity type and a second probability that each character of the named entity recognition is an end character of the named entity recognition candidate entity type based on the first character encoding vector of each character of the named entity recognition.

[0072] Step 208: Identify named entities belonging to the candidate entity types of named entity recognition based on the first probability and the second probability.

[0073] For the candidate entity type X, identify the characters between the first character with the highest first probability and the second character with the highest second probability as a named entity Y. The entity type of this named entity Y is the candidate entity type X.

[0074] The candidate entity type X is one or more. Schematically, the candidate entity types include at least two entity types belonging to at least two industries. Also, it is called the named entity recognition scenario of multiple industries and multiple entities.

[0075] Exemplary reference Figure 3 , assuming the input text is "I like Cadillac cars", and the candidate entity type X includes car brands. When the character "Kai" has the highest first probability and the character "Ke" has the highest second probability, identify the character "Cadillac" as a named entity, and the entity type of the named entity "Cadillac" is car brand.

[0076] In summary, the method provided in this embodiment encodes each character in the input text, and based on the first character encoding vector of each character, predicts the first probability that each character is the start character of the candidate entity type and the second probability that each character is the end character of the candidate entity type, so as to identify the named entities belonging to the candidate entity types. Since the start and end of the named entities are modeled hierarchically, the training process does not need to introduce entity boundary labels, thereby reducing the number of training labels in the named entity recognition scenario of multiple industries and multiple entities, and improving the recognition speed and recognition accuracy of named entity recognition.

[0077] Figure 4 Shows a flowchart of a named entity recognition method provided by an exemplary embodiment of the present application. This embodiment is illustrated by taking the execution of this method by the named entity recognition device 120 shown in Figure 1 as an example. The method includes:

[0078] Step 401: Obtain an input text, where the input text includes at least one character;

[0079] The input text is the text for which named entity recognition is required. The input text includes one or more characters. In one example, the input text is the text in the user profile of a user account.

[0080] Step 402: Invoke a language model to perform character encoding on each character in the named entity recognition input text to obtain the first character encoding vector of each character;

[0081] In this embodiment, the language model is described using the encoder in the BERT model as an example. The BERT model is a pre-trained language model for natural language processing developed by Google that uses a bidirectional encoder representation. The BERT model is available on the arxiv website at https: / / arxiv.org / pdf / 1810.04805.pdf. The BERT model includes an encoder and a decoder. In this embodiment, only the encoder of the BERT model is required.

[0082] Illustratively, the BERT model is pre-trained using a general corpus. Alternatively, the BERT model is first trained using a general corpus and then retrained using domain-specific corpora from one or more scenarios related to named entity scenarios. This embodiment does not limit the specific training method of the BERT model.

[0083] Call the encoder in the BERT model to perform character encoding on each character in the named entity recognition input text to obtain the first character encoding vector of each character. Figure 5 Shows the structure of the BERT model.

[0084] exist Figure 5 In the text editor, the characters CLS indicate the beginning of the input text, and the characters SEP indicate the end of the input text. i represents the i-th character in the sentence, E i Represents the embedding vector of the i-th word in the sentence, T i Represents the encoding vector of the first character of the i-th word in the sentence. 0 represents the probability of 0, and B-PER represents the predicted probability. If the sentence contains N characters, then i is a positive integer not greater than N.

[0085] Step 403: calling the first classification layer in the bigram model to predict the first character encoding vector of each character, obtaining the second character encoding vector of each character and a first probability that the character is the starting character of the candidate entity type;

[0086] Exemplarily, the first classification layer is implemented by a first fully connected (FC) layer, and the second classification layer is implemented by a second fully connected layer. Figure 6 A typical implementation of a fully connected layer is shown.

[0087] Combined with reference Figure 7 , taking the input text "BMW releases new BMW 3 Series" as an example, the character CLS indicates the beginning of the input text, and the character SEP indicates the end of the input text. E represents the embedding vector, such as E 宝The embedding vector representing the character "Bao"; T represents the first character encoding vector, such as T 宝 represents the first character encoding vector of the character "Bao".

[0088] For example Figure 7 As shown, assume that the candidate entity type 51 is a car brand and the candidate entity type 52 is a vehicle model series. The start layer will predict that the first "Bao" is the start character of the candidate entity type 51, that is, the character with the highest first probability of the candidate entity type 51; the second "Bao" is the start character of the candidate entity type 52, that is, the character with the highest first probability of the candidate entity type 51.

[0089] Step 404: Concatenate the first character encoding vector and the second character encoding vector of each character to obtain the combined character encoding vector of the character;

[0090] For example, concatenate (concat) the first character encoding vector T 宝 of the character "Bao" and the second character encoding vector of the character "Bao" output by the start layer, and then obtain the combined character encoding vector of the character "Bao".

[0091] Step 405: Call the second classification layer of the binary model to predict the combined character encoding vector of each character, and obtain the second probability that each character is the end character of the candidate entity type;

[0092] For example Figure 7 As shown, the end layer will predict that the first "Ma" is the end character of the candidate entity type 51, that is, the character with the highest second probability of the candidate entity type 51; the character "Xi" is the end character of the candidate entity type 52, that is, the character with the highest second probability of the candidate entity type

[52] .[[]END]]

[0093] Step 406: Identify the named entity belonging to the candidate entity type of named entity recognition based on the first probability and the second probability.

[0094] For the candidate entity type X, identify the characters between the first character with the highest first probability and the second character with the highest second probability as a named entity Y. The entity type of the named entity Y is the candidate entity type X.

[0095] For example Figure 5 As shown, identify "BMW" as the first named entity with the entity type of car brand; identify "BMW 3 Series" as the second named entity with the entity type of vehicle model series.

[0096] To sum up, the method provided in this embodiment encodes each character in the input text and, based on the first character encoding vector of each character, predicts the first probability that each character is the starting character of the candidate entity type and the second probability that each character is the ending character of the candidate entity type, thereby identifying the named entities belonging to the candidate entity type. Since the start and end of the named entity are modeled hierarchically, the training process does not need to introduce entity boundary labels, thereby reducing the number of training labels in the naming recognition scenario of multiple industries and multiple entities, and improving the recognition speed and recognition accuracy of named entity recognition.

[0097] The above-mentioned named entity recognition method can be applied to the mining process of user tags. In an illustrative example, in the field of advertising push, it is necessary to mine the user's historical behavior data in order to label the user with a user profile. Figure 8 As shown:

[0098] In step 81, multi-scene text is obtained;

[0099] Obtain text from multiple scenarios, such as advertisements clicked by users, articles read by users, titles of products purchased by users, and descriptions of apps installed or downloaded by users, to obtain the multi-scenario text corresponding to the user account.

[0100] Multi-scenario text includes text generated by log data of a user account in at least two usage scenarios (or industries).

[0101] In step 82, multi-scene named entity recognition;

[0102] Perform named entity recognition on multi-scene texts of user accounts to obtain entity labels.

[0103] In step 83, user behavior is modeled;

[0104] Based on the entity label corresponding to each user account, the user's behavior pattern is modeled to obtain the user portrait label of each user account.

[0105] In step 84, a user entity word tag is created;

[0106] Determine each user's user preferences based on the user portrait tag and establish a user entity word tag for each user. In the process of constructing user features, user features include user attribute features, user preference features, and user statistical features. For easier use, feature data can be constructed in two formats (explicit features and implicit features):

[0107] Semantic features: also known as semantic tags, are display features (such as user attribute features such as education level and occupation, and user preference features such as technology geeks, fuel-efficient cars, and elderly phones).

[0108] Embedding features: These are implicit features, including user behavior embedding features (such as browsing Uniform Resource Locator (URL) vectors, click feed stream vectors, and other user statistical features); they also include semantic embedding features (such as user preference features such as semantic tag vectorization).

[0109] In step 85, user behavior interest orientation;

[0110] Determine user behavior interest orientation based on user entity word tags.

[0111] In step 86, user semantic features.

[0112] Determine user semantic features based on user entity word labels.

[0113] In this application's example business, we address named entity recognition across multiple scenarios and industries. Multi-scenario refers to processing text data from four scenarios: advertisements, app descriptions, e-commerce titles, and news articles. Multi-industry refers to focusing on fine-grained entities within each industry, encompassing 69 entity types across 13 industries, as shown in Table 1.

[0114] Table 1: Fine-grained named entity types

[0115]

[0116]

[0117]

[0118] Figure 8 The user portrait tag of a user account is schematically shown. Figure 8 Each named entity in is a user profile label of the user account. The named entity with a larger font size is more consistent with the user preference of the user account. Figure 8 Each named entity in is only for illustrative purposes, and this application does not limit the specific named entities. Figure 8 Each named entity in can be understood as a "user portrait label".

[0119] Figure 9 This diagram illustrates the use of user profile tags in an advertising push scenario, as provided by an exemplary embodiment of this application. Advertisers use the delivery endpoint to deliver ads to the database. During the delivery process, advertising parameters such as exposure / audience estimation, intelligent creative production, and targeted tag recommendations are configured.

[0120] Advertisements are stored in a database and indexed based on ad parameters.

[0121] When a user initiates an ad request on the traffic source used by a regular user, the ad playback engine performs a targeted search of the database based on the user ID, context, and user profile tags, thereby recalling candidate ads. For example, it can retrieve tens of thousands of candidate ads for the current user account.

[0122] During the rough sorting phase, the ad playback engine uses a rough sorting algorithm to sort candidate ads, which are measured in the tens of thousands, to produce a list of candidate ads, which are measured in the hundreds. This rough sorting algorithm primarily considers the click-through rate (CTR) and post-click conversion rate (CVR) metrics.

[0123] During the selection and ranking phase, the ad playback engine uses a selection and ranking algorithm to fine-tune candidate ads measured in the hundreds, resulting in candidates with a ranking below ten. The selection and ranking algorithm primarily considers CTR, CVR, and Negative Feedback Rate (NFR) metrics.

[0124] Table 2 shows the comparison of the effects of the "BERT+binary model" of this application and the "Bi-LSTM+CRF" in the related art:

[0125] Table 2: Comparison of the performance of BERT+ binary model and related technologies (advertising scenario)

[0126] Provide model F-number Accuracy Recall Bi-LSTM+CRF 0.8569 0.8246 0.89185 BERT+Bigram Model 0.9188 0.9178 0.9197

[0127] Taking advertising as an example, the "BERT+binary model" proposed in this application shows significant improvements in precision, recall, and F-value compared to existing solutions, with the F-value increasing by 6.19%. The F-value is the statistical value of the F test in statistics.

[0128] Figure 11 A flowchart of a method for training a named entity model provided by an exemplary embodiment of the present application is shown. The named entity model includes: a language model and a bigram model. The method includes:

[0129] Step 501: Obtain a training set, where the training set includes sample texts and sample named entities in the sample texts;

[0130] Step 502: Calling the language model to perform character encoding on each character in the sample text to obtain a first character encoding vector for each character;

[0131] Call the encoder in the BERT model to encode each character in the sample text and obtain the first character encoding vector of each character.

[0132] Step 503: Calling the bigram model to predict a first probability that each character is the start character of the candidate entity type and a second probability that each character is the end character of the candidate entity type based on the first character encoding vector of each character;

[0133] Illustratively, the first classification layer in the binary model is called to predict the first character encoding vector of the character, thereby obtaining the second character encoding vector of the character and a first probability that the character is the start character of the candidate entity type. The first character encoding vector and the second character encoding vector of the character are concatenated to obtain a combined character encoding vector of the character. The second classification layer in the binary model is called to predict the combined character encoding vector of the character, thereby obtaining a second probability that the character is the end character of the candidate entity type.

[0134] Illustratively, the first classification layer is a first fully connected layer, and the second classification layer is a second fully connected layer. The first fully connected layer is called to predict the first character encoding vector of a character, obtaining the second character encoding vector of the character and a first probability that the character is the starting character of a candidate entity type. The second fully connected layer is called to predict the combined character encoding vector of the character, obtaining a second probability that the character is the ending character of the candidate entity type.

[0135] Step 504: identifying a predicted named entity belonging to the candidate entity type based on the first probability and the second probability;

[0136] The prediction process from step 501 to step 504 can refer to the above Figure 2 or Figure 4 The method shown will not be described in detail in this embodiment.

[0137] Step 505: Based on the error between the predicted named entity and the sample named entity, the network parameters of the language model and the bigram model are trained.

[0138] In one example, the language model and the binary model have their own layer-by-layer learning rates. Setting learning rates in layers is useful when, for example, fine-tuning an existing model, you might want to keep some layers fixed, but sometimes you might want to update them at a lower learning rate than other layers. This is the case with layer-by-layer learning rate settings.

[0139] In one example, to reduce the negative impact of manually annotated labels on the model, this application considers label smoothing regularization in the loss function. The main idea of label smoothing is to assume that the sample is of another category with a probability of α, thereby reducing the label confidence. This prevents the model from "over-confiding" the labels of the training samples during training. Based on this, this application replaces the conventional cross-entropy loss function with a cross-entropy loss function with label smoothing.

[0140] The error between the predicted named entities and the sample named entities is calculated based on the cross entropy loss function with label smoothing, and the network parameters of the language model and the bigram model are trained.

[0141]

[0142] Among them, K is the number of categories of candidate entity types, α is the hyperparameter introduced by label smoothing, and y k When k is the correct category, it is 1, and otherwise it is 0, that is, yk∈{0,1}, where pt is the output probability corresponding to the correct category. k is the probability of belonging to candidate entity type k, p i is the probability of belonging to candidate entity type i.

[0143] The computer system involved in the embodiments of the present application can also be a distributed system formed by connecting a client and multiple nodes (any form of computing devices in the access network, such as servers and user terminals) through network communication.

[0144] Taking the distributed system as the blockchain system as an example, see Figure 12 , Figure 12 This is an optional structural diagram of the distributed system 100 provided in an embodiment of the present application applied to a blockchain system. The system is composed of multiple nodes (any form of computing device connected to the network, such as servers and user terminals) and clients. The nodes form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol that runs on top of the Transmission Control Protocol (TCP). In a distributed system, any machine, such as a server or terminal, can join and become a node. Nodes include a hardware layer, an intermediate layer, an operating system layer, and an application layer.

[0145] See also Figure 12 The functions of each node in the blockchain system shown include:

[0146] 1) Routing: A basic function of a node, used to support communication between nodes.

[0147] In addition to the routing function, nodes can also have the following functions:

[0148] 2) Applications, deployed in the blockchain, implement specific services based on actual business needs, record data related to the implementation of functions to form record data, carry digital signatures in the record data to indicate the source of the task data, and send the record data to other nodes in the blockchain system for other nodes to add the record data to a temporary block when they successfully verify the source and integrity of the record data.

[0149] For example, the services implemented by the application include:

[0150] 2.1) Wallet: This is used to provide the functionality for conducting electronic currency transactions, including initiating transactions (i.e., sending the transaction record of the current transaction to other nodes in the blockchain system. After successful verification by other nodes, as a response to acknowledge the transaction’s validity, the transaction record data is stored in a temporary block of the blockchain. Of course, the wallet also supports querying the remaining electronic currency in the electronic currency address.

[0151] 2.2) Shared ledgers are used to store, query, and modify account data. Records of operations on account data are sent to other nodes in the blockchain system. After verification, other nodes acknowledge the validity of the account data by storing the recorded data in a temporary block. They can also send a confirmation to the node that initiated the operation.

[0152] 2.3) Smart contracts are computerized protocols that can enforce the terms of a contract. They are implemented through code deployed on a shared ledger that is executed when certain conditions are met. Based on actual business needs, the code is used to complete automated transactions, such as querying the logistics status of a buyer's purchased goods and transferring the buyer's electronic currency to the merchant's address after the buyer signs for the goods. Of course, smart contracts are not limited to executing contracts for transactions, but can also execute contracts that process received information.

[0153] 3) Blockchain, including a series of blocks that are connected to each other in the order of their generation. Once a new block is added to the blockchain, it will not be removed. The block records the record data submitted by the nodes in the blockchain system.

[0154] See also Figure 13 , Figure 13This is an optional schematic diagram of the block structure provided by the embodiment of the present application. Each block includes the hash value of the transaction record stored in this block (the hash value of this block) and the hash value of the previous block. The blocks are connected by hash values to form a blockchain. In addition, the block can also include information such as the timestamp when the block was generated. Blockchain is essentially a decentralized database, a series of data blocks generated using cryptographic methods. Each data block contains relevant information used to verify the validity of its information (anti-counterfeiting) and generate the next block.

[0155] Figure 14 A block diagram of a named entity recognition device provided by an exemplary embodiment of the present application is shown. The device includes:

[0156] An acquisition module 1420 is configured to acquire an input text, where the input text includes at least one character.

[0157] An encoding module 1440 is configured to perform character encoding on each character in the input text to obtain a first character encoding vector for each character;

[0158] A prediction module 1460 is configured to predict, based on the first character encoding vector of each character, a first probability that each character is a start character of a candidate entity type and a second probability that each character is an end character of the candidate entity type;

[0159] The identification module 1480 is configured to identify a named entity belonging to the candidate entity type based on the first probability and the second probability.

[0160] In an example of the present application, the prediction module 1460 is used to call the first classification layer in the binary model to predict the first character encoding vector of the character to obtain the second character encoding vector of the character and the first probability that the character is the starting character of the candidate entity type; cascade the first character encoding vector and the second character encoding vector of the character to obtain the merged character encoding vector of the character; call the second classification layer of the binary model to predict the merged character encoding vector of the character to obtain the second probability that the character is the ending character of the candidate entity type.

[0161] In one example of the present application, the first classification layer is a first fully connected layer, and the second classification layer is a second fully connected layer;

[0162] The prediction module 1460 is configured to call the first fully connected layer to predict the first character encoding vector of the character to obtain a second character encoding vector of the character and a first probability that the character is the starting character of the candidate entity type;

[0163] The prediction module 1460 is used to call the second fully connected layer to predict the merged character encoding vector of the character to obtain a second probability that the character is the end character of the candidate entity type.

[0164] In an example of the present application, the encoding module 1440 is configured to call a language model to encode each character in the input text to obtain a character encoding vector for each character.

[0165] In an example of the present application, the encoding module 1440 is used to call the encoder in the BERT model to encode each character in the input text to obtain a character encoding vector for each character.

[0166] Figure 15 A block diagram of a training device for a named entity model provided by an exemplary embodiment of the present application is shown. The named entity model includes: a language model and a bigram model. The device includes:

[0167] An acquisition module 1420 is configured to acquire a training set, wherein the training set includes sample texts and sample named entities in the sample texts;

[0168] An encoding module 1440 is configured to call the pre-trained language model to perform character encoding on each character in the sample text to obtain a first character encoding vector for each character;

[0169] Prediction module 1460, configured to call the bigram model to predict, based on the first character encoding vector of each character, a first probability that each character is a start character of a candidate entity type and a second probability that each character is an end character of the candidate entity type;

[0170] an identification module 1480 for identifying a predicted named entity belonging to the candidate entity type based on the first probability and the second probability;

[0171] The training module 1490 is configured to train network parameters of the language model and the bigram model based on the error between the predicted named entity and the sample named entity.

[0172] In an example of the present application, the encoding module 1440 is used to call the encoder in the BERT model to perform character encoding on each character in the sample text to obtain a first character encoding vector for each character.

[0173] In an example of the present application, the prediction module 1460 is used to call the first classification layer in the binary model to predict the first character encoding vector of the character to obtain the second character encoding vector of the character and the first probability that the character is the starting character of the candidate entity type; cascade the first character encoding vector and the second character encoding vector of the character to obtain the merged character encoding vector of the character; call the second classification layer of the binary model to predict the merged character encoding vector of the character to obtain the second probability that the character is the ending character of the candidate entity type.

[0174] In one example of the present application, the first classification layer is a first fully connected layer, and the second classification layer is a second fully connected layer;

[0175] The prediction module 1460 is configured to call the first fully connected layer to predict the first character encoding vector of the character to obtain a second character encoding vector of the character and a first probability that the character is the starting character of the candidate entity type;

[0176] The prediction module 1460 is used to call the second fully connected layer to predict the merged character encoding vector of the character to obtain a second probability that the character is the end character of the candidate entity type.

[0177] In an example of the present application, the language model and the bigram model have respective layer-wise learning rates.

[0178] In an example of the present application, the training module 1490 is used to calculate the error between the predicted named entity and the sample named entity based on a cross-entropy loss function with label smoothing, and train the network parameters of the language model and the binary model.

[0179] It should be noted that the named entity recognition device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the named entity recognition device provided in the above embodiment and the named entity recognition method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0180] It should be noted that the training device provided in the above embodiment is merely exemplified by the division of the aforementioned functional modules. In actual applications, the aforementioned functions can be distributed among different functional modules as needed, i.e., the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the training device provided in the above embodiment and the training method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0181] An embodiment of the present application also provides a computer device, which includes: a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the named entity recognition method provided by the above-mentioned method embodiments.

[0182] Optionally, the computer device is a computer device. For example, Figure 16 It is a structural diagram of a computer device provided in an embodiment of the present application.

[0183] The computer device 1600 includes a central processing unit (CPU) 1601, a system memory 1604 including a random access memory (RAM) 1602 and a read-only memory (ROM) 1603, and a system bus 1605 connecting the system memory 1604 and the CPU 1601. The computer device 1600 also includes a basic input / output system (I / O system) 1606 for facilitating information transfer between various components within the computer device, and a mass storage device 1607 for storing an operating system 1613, application programs 1614, and other program modules 1615.

[0184] The basic input / output system 1606 includes a display 1608 for displaying information and an input device 1609 such as a mouse and keyboard for user input. The display 1608 and the input device 1609 are connected to the central processing unit 1601 via an input / output controller 1610 connected to the system bus 1605. The basic input / output system 1606 may also include an input / output controller 1610 for receiving and processing input from a variety of other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1610 also provides output to a display screen, printer, or other types of output devices.

[0185] The mass storage device 1607 is connected to the central processing unit 1601 via a mass storage controller (not shown) connected to the system bus 1605. The mass storage device 1607 and its associated computer-readable storage medium provide non-volatile storage for the computer device 1600. In other words, the mass storage device 1607 may include a computer-readable storage medium (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0186] Without loss of generality, the computer-readable storage medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable storage instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electronically Erasable Programmable Read-Only Memory (EEPROM), flash memory or other solid-state storage devices, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, tape cassettes, magnetic tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage media are not limited to the above-mentioned ones. The above-mentioned system memory 1604 and mass storage device 1607 can be collectively referred to as memory.

[0187] The memory stores one or more programs, and the one or more programs are configured to be executed by one or more central processing units 1601. The one or more programs include instructions for implementing the above-mentioned method embodiments. The central processing unit 1601 executes the one or more programs to implement the named entity recognition method or the named entity recognition model training method provided by the above-mentioned each method embodiment.

[0188] According to various embodiments of the present application, the computer device 1600 may also be connected to a remote computer device on a network such as the Internet for operation. That is, the computer device 1600 may be connected to a network 1612 via a network interface unit 1611 connected to the system bus 1605, or the network interface unit 1611 may be used to connect to other types of networks or remote computer device systems (not shown).

[0189] The memory also includes one or more programs, which are stored in the memory and include steps executed by a computer device in the method provided in the embodiment of the present application.

[0190] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0191] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores at least one program code. When the program code is loaded and executed by a processor of a computer device, the named entity recognition method or the named entity recognition model training method provided in the above-mentioned method embodiments is implemented.

[0192] The present application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the named entity recognition method or the named entity recognition model training method provided in each of the above method embodiments.

[0193] Those skilled in the art will understand that all or part of the steps of implementing the above embodiments may be accomplished by hardware, or may be accomplished by a program instructing the relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned readable storage medium may be a read-only memory, a disk or an optical disk, etc.

[0194] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent switches, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for named entity recognition, characterized in that: The method comprises: Acquire a multi-scenario text, the multi-scenario text including at least one character, the multi-scenario text including text generated by log data of a user account in at least two usage scenarios or industries; Calling the encoder in the bidirectional encoder representation BERT model from the transformer to perform character encoding on each character in the multi-scene text to obtain a first character encoding vector for each character; Calling the first classification layer in the binary model to predict the first character encoding vector of the character to obtain the second character encoding vector of the character and a first probability that the character is the starting character of the candidate entity type; cascading the first character encoding vector of the character and the second character encoding vector of the character to obtain a merged character encoding vector of the character; Calling the second classification layer of the binary model to predict the merged character encoding vector of the character to obtain a second probability that the character is the end character of the candidate entity type; Identify a named entity belonging to the candidate entity type based on the first probability and the second probability, and obtain an entity label corresponding to the user account; Based on the entity tag corresponding to the user account, the behavior pattern of the user account is modeled to obtain a user portrait tag of the user account. The user portrait tag of the user account is used in an advertisement push scenario.

2. The method according to claim 1, characterized in that The first classification layer is a first fully connected layer, and the second classification layer is a second fully connected layer; The calling of the first classification layer in the binary model to predict the first character encoding vector of the character to obtain the second character encoding vector of the character and a first probability that the character is the starting character of the candidate entity type includes: Calling the first fully connected layer to predict the first character encoding vector of the character to obtain a second character encoding vector of the character and a first probability that the character is the starting character of the candidate entity type; The calling the second classification layer of the binary model to predict the merged character encoding vector of the character to obtain a second probability that the character is the end character of the candidate entity type includes: The second fully connected layer is called to predict the merged character encoding vector of the character to obtain a second probability that the character is the end character of the candidate entity type.

3. A method for training a named entity model, characterized in that: The named entity model includes: a bidirectional encoder of a transformer represents an encoder and a bigram model in a BERT model, and the method includes: Obtaining a training set, the training set including sample text and sample named entities in the sample text; Calling the encoder in the BERT model to perform character encoding on each character in the sample text to obtain a first character encoding vector for each character; Calling the first classification layer in the binary model to predict the first character encoding vector of the character to obtain the second character encoding vector of the character and a first probability that the character is the starting character of the candidate entity type; cascading the first character encoding vector of the character and the second character encoding vector of the character to obtain a merged character encoding vector of the character; Calling the second classification layer of the binary model to predict the merged character encoding vector of the character to obtain a second probability that the character is the end character of the candidate entity type; identifying a predicted named entity belonging to the candidate entity type based on the first probability and the second probability; Based on the error between the predicted named entity and the sample named entity, the network parameters of the BERT model and the binary model are trained; the named entity model is used to obtain the entity label corresponding to the user account based on the identified named entity; based on the entity label corresponding to the user account, the behavior pattern of the user account is modeled to obtain the user portrait label of the user account, and the user portrait label of the user account is used in the advertising push scenario.

4. The method according to claim 3, characterized in that The first classification layer is a first fully connected layer, and the second classification layer is a second fully connected layer; The calling of the first classification layer in the binary model to predict the first character encoding vector of the character to obtain the second character encoding vector of the character and a first probability that the character is the starting character of the candidate entity type includes: Calling the first fully connected layer to predict the first character encoding vector of the character to obtain a second character encoding vector of the character and a first probability that the character is the starting character of the candidate entity type; The calling the second classification layer of the binary model to predict the merged character encoding vector of the character to obtain a second probability that the character is the end character of the candidate entity type includes: The second fully connected layer is called to predict the merged character encoding vector of the character to obtain a second probability that the character is the end character of the candidate entity type.

5. The method according to claim 3, characterized in that The BERT model and the bigram model have separate layer-wise learning rates.

6. The method according to any one of claims 3 to 5, characterized in that: The training of network parameters of the BERT model and the bigram model based on the error between the predicted named entity and the sample named entity includes: The error between the predicted named entity and the sample named entity is calculated based on a cross entropy loss function with label smoothing, and the network parameters of the BERT model and the binary model are trained.

7. A named entity recognition device, characterized in that: The device comprises: An acquisition module, configured to acquire multi-scenario text, the multi-scenario text including at least one character, the multi-scenario text including text generated by log data of a user account in at least two usage scenarios or industries; An encoding module, configured to call an encoder in a bidirectional encoder representation BERT model from a transformer to perform character encoding on each character in the multi-scene text to obtain a first character encoding vector for each character; A prediction module is configured to call the first classification layer in the binary model to predict the first character encoding vector of the character to obtain the second character encoding vector of the character and a first probability that the character is the start character of the candidate entity type; concatenate the first character encoding vector of the character and the second character encoding vector of the character to obtain a merged character encoding vector of the character; call the second classification layer of the binary model to predict the merged character encoding vector of the character to obtain a second probability that the character is the end character of the candidate entity type; An identification module is used to identify the named entity belonging to the candidate entity type based on the first probability and the second probability, and obtain the entity label corresponding to the user account; according to the entity label corresponding to the user account, the behavior pattern of the user account is modeled to obtain the user portrait label of the user account, and the entity user portrait label corresponding to the user account is used in the advertising push scenario.

8. A training device for a named entity model, characterized in that: The named entity model includes: a bidirectional encoder of a transformer representing an encoder and a bigram model in a BERT model, the apparatus comprising: An acquisition module, configured to acquire a training set, wherein the training set includes sample texts and sample named entities in the sample texts; An encoding module, configured to call an encoder in the BERT model to perform character encoding on each character in the sample text to obtain a first character encoding vector for each character; A prediction module is configured to call the first classification layer in the binary model to predict the first character encoding vector of the character to obtain the second character encoding vector of the character and a first probability that the character is the start character of the candidate entity type; concatenate the first character encoding vector of the character and the second character encoding vector of the character to obtain a merged character encoding vector of the character; call the second classification layer of the binary model to predict the merged character encoding vector of the character to obtain a second probability that the character is the end character of the candidate entity type; an identification module, configured to identify a predicted named entity belonging to the candidate entity type based on the first probability and the second probability; A training module is used to train the network parameters of the BERT model and the binary model based on the error between the predicted named entity and the sample named entity; the named entity model is used to obtain the entity label corresponding to the user account based on the identified named entity; based on the entity label corresponding to the user account, the behavior pattern of the user account is modeled to obtain the user portrait label of the user account, and the entity user portrait label corresponding to the user account is used in the advertising push scenario.

9. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the named entity recognition method as described in claim 1 or 2, or the training method of the named entity model as described in any one of claims 3 to 6.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one program code, which is loaded and executed by a processor to implement the named entity recognition method according to claim 1 or 2, or the named entity model training method according to any one of claims 3 to 6.

11. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. The processor of the computer device reads and executes the computer instructions from the computer-readable storage medium to implement the named entity recognition method as described in claim 1 or 2, or the training method of the named entity model as described in any one of claims 3 to 6.

Images