Named entity recognition method and its device, equipment, medium, and product
By coding and feature interaction of the word elements in the Bert model, the relative position feature information is generated, and the problem that the Bert model fails to perceive the word element position relationship in the name entity recognition is solved, and the accuracy of the name entity recognition is improved and Training efficiency, suitable for product information processing in the e-commerce field.
Patent Information
- Application Number
- CN202111667457.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-31
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-12-31
AI Technical Summary
The Bert model fails to effectively perceive the relative positional relationship between word elements in naming entity recognition, affecting its representation learning and recognition effect.
By coding multiple relative distance scales of word elements after identifying text word segmentation, relative position feature information is generated, and used as the correlation weight of the self-attention layer to generate text feature vectors, thereby enriching the context information of word element embedding .
It improves the accuracy and representation learning ability of named entity recognition, simplifies the coding network architecture, improves the training efficiency and recognition effect of neural network models, and is suitable for product information processing in the e-commerce field.
Smart Images

Figure CN114298049B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of natural language processing, and in particular to a method for named entity recognition and its corresponding apparatus, computer device, computer-readable storage medium, and computer program product. Background Art
[0002] Named entity recognition plays an important role in search, recommendation, and user portrait analysis in the e-commerce field. For example, when a user searches, the search box can guide the user to search for the product words they want to search for, thereby improving search efficiency. The associated words in the word association need to be based on the products in the product library, and the named entity recognition method can identify the product words. According to the user's search, the labels of the search keywords, such as product words, brands, etc., are identified, which can improve the effect of search optimization by adding user weights. At the same time, the user's search for products, click on products, add to cart, order products, etc., can use the named entity recognition method to identify the user's preferences for these behaviors, which can be used for user portrait analysis, and so on.
[0003] Using the Bert model for named entity recognition is a popular method in recent years. This method implements a two-stage training of the Bert model. After pre-training in the first stage, the pre-trained model is used as the base for fine-tuning training corresponding to the named entity recognition required for downstream tasks, which generally achieves corresponding results.
[0004] The only drawback is that the Bert model is different from the infrastructure it uses, namely the Transformer model. The Transformer model uses trigonometric position encoding to encode the relative position information of the text to be recognized. However, when the Bert model embeds words in the recognized text, although it gives token embedding, segment embedding and position embedding at the same time, the position embedding is only the absolute position information of each word unit. Therefore, in the representation learning process of the Bert model, the Bert model fails to acquire the ability to perceive the relative position relationship between word units, and the trigonometric function encoding method of the Transformer model is rigidly applied to the Bert model, which cannot play an effective role.
[0005] However, the relative position information in the text is very important. For example, in the English sentence "Wang founded Leadners IP Firm in 2019", in this sentence, the words after the preposition "in" are more likely to be locations and times than the words before it, and the words before "Firm" are more likely to be locations and times. It can be seen that the position where the token is located plays an important role in understanding the context. Similarly, in Chinese, for example, "Wang founded Lineng Firm in 2019". Similarly, generally, time or location follows immediately after "in". Thus, it can be seen that if the model can effectively perceive the distance between tokens in the text and obtain relative position information, it is beneficial for tasks such as named entity recognition.
[0006] In the e-commerce field in which the applicant is engaged, a large amount of text information needs to be processed for a long time and relies heavily on accurate named entity recognition results. Therefore, in the process of practice, a solution to improve the above deficiencies is explored, and this application is proposed. Summary of the Invention
[0007] The primary objective of this application is to provide a named entity recognition method, its corresponding device, computer device, computer-readable storage medium, and computer program product to solve at least one of the above problems.
[0008] To meet the various objectives of this application, the following technical solutions are adopted in this application:
[0009] Generate corresponding embedding vectors according to the encoded tokens obtained by tokenizing the text to be recognized.
[0010] Generate relative position feature information according to the encoded multiple relative distance scales between pairwise tokens among the multiple tokens.
[0011] Use the embedding vectors of each token as input, and perform feature interaction through at least one encoding network including a self-attention layer to generate a text feature vector, where the relative position feature information serves as the associated weight for generating the text feature vector.
[0012] Identify the named entities in the text to be recognized according to the text feature vector.
[0013] In a specific embodiment, generating corresponding embedding vectors according to the encoded tokens obtained by tokenizing the text to be recognized includes the following steps:
[0014] Obtain the text to be recognized, where the text to be recognized is a title text for describing product information.
[0015] Use a pre-trained tokenization model to tokenize the text to be recognized to obtain corresponding multiple tokens, where the tokens are words composed of English words or Chinese characters as single characters.
[0016] The embedding vectors of each word unit are encoded separately, and the embedding vector of each word unit includes the token embedding, position embedding and segment embedding of the word unit.
[0017] In a specific embodiment, the relative position feature information is generated according to a plurality of relative distance scales between two word-grams in the plurality of word-grams, including the following steps:
[0018] Encoding the relative distance information between each word unit in each word unit according to different relative distance scales to obtain a coding matrix corresponding to each relative distance scale;
[0019] The coding matrices corresponding to the relative distance scales are concatenated into a multi-channel coding matrix;
[0020] The multi-channel encoding matrix matching learnable weights are nonlinearly transformed to activate the output and obtain relative position feature information.
[0021] In a further embodiment, encoding the relative distance information between each word unit in each word unit according to different relative distance scales to obtain a coding matrix corresponding to each relative distance scale includes the following steps:
[0022] For each word unit, calculate the relative distance between the first character of the former and the first character of the latter between the word unit and any other word unit, obtain the first vector corresponding to each word unit, and concatenate the first vectors of all word units into a first encoding matrix;
[0023] For each word unit, the relative distance between the first character of the former and the last character of the latter is calculated with respect to any word unit, and the second vector corresponding to each word unit is obtained, and the second vectors of all word units are concatenated into a second encoding matrix;
[0024] For each word unit, calculate the relative distance between the last character of the former and the first character of the latter between the word unit and any other word unit, obtain the third vector corresponding to each word unit, and concatenate the third vectors of all word units into a third encoding matrix;
[0025] For each word unit, the relative distance between the last character of the former and the last character of the latter is calculated to obtain the fourth vector corresponding to each word unit, and the fourth vectors of all word units are concatenated into a fourth encoding matrix.
[0026] In a specific embodiment, the embedding vector of each word is used as input, and feature interaction is performed through at least one encoding network including a self-attention layer to generate a text feature vector, including the following steps performed by the first encoding network:
[0027] The self-attention layer in the current encoding network is used to construct the corresponding query vector, key vector and value vector for the embedding vector of each word unit;
[0028] The self-attention layer performs a weighted summation of the value vectors corresponding to all word units, where the weight of each value vector is the normalized score of the dot product of the query vector corresponding to its corresponding key vector and other embedding vectors;
[0029] The self-attention layer multiplies the weighted sum result by the sum of the conversion weight matrix and the relative position feature information to obtain an intermediate information matrix, so that the relative position feature information constitutes the associated weight of the intermediate information matrix;
[0030] The intermediate information matrix is subjected to feature extraction via a feedforward neural layer to obtain a text feature vector as the input required by the self-attention layer of the next encoding network.
[0031] In a specific embodiment, identifying the named entity in the text to be identified according to the text feature vector includes the following steps:
[0032] Using a conditional random field model to perform part-of-speech tagging on the named entities to determine the part-of-speech of each named entity;
[0033] According to the part of speech, each named entity is output to a storage location corresponding to the part of speech to form attribute data corresponding to the part of speech in the product information.
[0034] In a specific embodiment, identifying the named entity in the text to be identified according to the text feature vector includes the following steps:
[0035] Using a conditional random field model to perform part-of-speech tagging on the named entities to determine the part-of-speech of each named entity;
[0036] The named entities corresponding to the target part of speech are extracted according to the part of speech, the named entities are accurately matched with a preset entity dictionary, and the named entities that achieve accurate matching are output as keywords of the text to be recognized.
[0037] A named entity recognition device provided to meet one of the purposes of the present application includes: a word embedding module, a distance encoding module, a feature interaction module, and an annotation recognition module, wherein the word embedding module is used to generate corresponding embedding vectors based on multiple word-unit encodings obtained by word segmentation of a text to be recognized; the distance encoding module is used to generate relative position feature information based on multiple relative distance scale encodings between two word-units in the multiple word-units; the feature interaction module is used to take the embedding vector of each word-unit as input, perform feature interaction through at least one encoding network including a self-attention layer, and generate a text feature vector, wherein the relative position feature information is used as an associated weight for generating the text feature vector; the annotation recognition module is used to identify the named entities in the text to be recognized based on the text feature vector.
[0038] A computer device provided to meet one of the purposes of the present application includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the named entity recognition method described in the present application.
[0039] A computer-readable storage medium is provided to meet another purpose of the present application, which stores a computer program implemented according to the named entity recognition method in the form of computer-readable instructions. When the computer program is called and executed by a computer, the steps included in the method are executed.
[0040] A computer program product provided to meet another purpose of the present application includes a computer program / instruction, which, when executed by a processor, implements the steps of the method described in any embodiment of the present application.
[0041] Compared with the prior art, the advantages of this application are as follows:
[0042] First of all, for the purpose of named entity recognition, the present application, for the basic encoding network that uses the self-attention mechanism for context combing, on the basis of its original encoding method, encodes each of the words obtained from the word segmentation of the to-be-recognized text at multiple relative distance scales to obtain the relative position feature information between the words, and adds the relative position feature information as the association weight to the self-attention operation process to control the generation of the text feature vector output by the encoding network, so that the text feature vector output by the encoding network carries the distance information between the words, and enriches the context information of the original word embedding vector with the relative position relationship between the words, so that in the representation learning process, the deep semantic information of the words can be more effectively represented according to the relative position of the words, thereby achieving a more accurate named entity recognition effect.
[0043] Secondly, the present application does not need to change the architecture of the encoding network. It only uses the relative position feature information generated by the encoding as the associated weight for the self-attention layer in the encoding network to correct the generation weight of the text feature vector. It is simple to implement and efficient to process. The encoding network can obtain a more sensitive perception ability to the position information of the text, which can improve the representation learning ability of the neural network model where the encoding network is located, and help to improve the training efficiency of the neural network model, making it easier to be fine-tuned and trained to a convergence state using a small number of samples, thereby saving training costs.
[0044] In addition, the technical solution of the present application is more suitable for processing product information in the e-commerce field. In view of the semantic characteristics of product information, especially the product title text containing multiple parallel words, the named entities therein can be accurately identified through the present application and applied to other downstream tasks, which can improve the efficiency of e-commerce information processing and enable the e-commerce platform to reap economies of scale. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0046] Figure 1 A flowchart of a typical embodiment of the method for named entity recognition of the present application;
[0047] Figure 2 A schematic diagram of the network architecture of a neural network model for performing a named entity recognition method in this application;
[0048] Figure 3 A schematic diagram of the process of encoding the embedding vector of the text to be recognized in this application;
[0049] Figure 4 A schematic diagram of the process of encoding the relative position feature information of the text to be recognized in this application;
[0050] Figure 5 A schematic diagram of the process of encoding relative position feature information based on various relative distance scales in this application;
[0051] Figure 6 It is a coding matrix obtained by coding based on a relative distance scale in the example of this application;
[0052] Figure 7 A flowchart of the processing process of the encoding network of the improved neural network model of this application;
[0053] Figure 8 A flowchart of an application example of a named entity obtained in this application;
[0054] Fig. 9 A flowchart of another application example of the named entity obtained in this application;
[0055] Fig.10 This is a principle block diagram of the named entity recognition device of the present application;
[0056] Fig.11 A schematic diagram of the structure of a computer device used in this application. DETAILED DESCRIPTION
[0057] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.
[0058] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.
[0059] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.
[0060] It will be understood by those skilled in the art that the "client", "terminal" and "terminal device" used herein include both devices with wireless signal receivers, which are devices with only wireless signal receivers without transmission capabilities, and devices with receiving and transmitting hardware, which are devices with receiving and transmitting hardware capable of two-way communication on a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers, tablet computers, which have single-line displays or multi-line displays or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service, personal communication system), which can combine voice, data processing, fax and / or data communication capabilities; PDA (Personal Digital Assistant, personal digital assistant), which may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar and / or GPS (Global Positioning System, global positioning system) receiver; conventional laptop and / or palmtop computers or other devices, which have and / or include a conventional laptop and / or palmtop computer or other device with and / or including a radio frequency receiver. The "client", "terminal" and "terminal device" used herein may be portable, transportable, installed in a vehicle (air, sea and / or land), or suitable for and / or configured to run locally, and / or in a distributed form, at any other location on the earth and / or in space. The "client", "terminal" and "terminal device" used herein may also be a communication terminal, an Internet terminal, a music / video playing terminal, for example, a PDA, a MID (Mobile Internet Device) and / or a mobile phone with a music / video playing function, or a smart TV, a set-top box and other devices.
[0061] The hardware referred to by the names such as "server", "client", and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. The computer program is stored in its memory, and the central processing unit calls the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.
[0062] It should be pointed out that the concept of "server" referred to in this application can also be extended to the case of server clusters. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided. In physical space, these servers can be independent of each other but can be called through interfaces, or integrated into a physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility, and should not use it to restrict the implementation of the network deployment method of this application.
[0063] Unless expressly specified, one or more technical features of the present application can be deployed on a server for implementation and accessed by a client through a remote call to obtain an online service interface provided by the server, or can be directly deployed and run on a client for access.
[0064] The neural network models referenced or may be referenced in this application, unless expressly specified, can be deployed on a remote server and remotely called on the client, or can be deployed and directly called on a client with sufficient device capabilities. In some embodiments, when it runs on the client, its corresponding intelligence can be obtained through transfer learning to reduce the requirements for the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.
[0065] Unless explicitly specified, the various data involved in this application can be stored remotely on a server or on a local terminal device, as long as it is suitable for being called by the technical solution of this application.
[0066] Those skilled in the art should be aware that, although the various methods of the present application are described based on the same concept and thus present commonality to each other, unless otherwise specified, these methods can be independently executed. Similarly, for each embodiment disclosed in the present application, they are all proposed based on the same inventive concept, therefore, concepts with the same expression, and concepts that are appropriately changed for convenience despite different expressions, should be understood as equivalent.
[0067] Unless the mutually exclusive relationship between the embodiments to be disclosed in this application is explicitly stated, the relevant technical features involved in each embodiment can be cross-combined to flexibly construct a new embodiment, as long as such combination does not deviate from the creative spirit of this application and can meet the needs of the prior art or solve certain deficiencies in the prior art. Those skilled in the art should be aware of this flexibility.
[0068] A named entity recognition method of the present application can be programmed as a computer program product and deployed in a client or server for execution. The method can be executed by accessing an interface opened after the computer program product is running and performing human-computer interaction with the process of the computer program product through a graphical user interface.
[0069] See also Figure 1 In a typical embodiment, the named entity recognition method of the present application includes the following steps:
[0070] Step S1100: Generate corresponding embedding vectors according to multiple word element codes obtained by segmenting the text to be recognized:
[0071] First of all, it should be pointed out that since this application is mainly an improvement to adapt to the characteristics of the Bert model, and the Bert model is a basic framework, it adopts the encoding path of the Transformer model, and on the basis of the Bert model, Albert, Electra and other models with the same basic framework are further developed. Therefore, it is not difficult to understand that, Figure 2 As shown in the network architecture, all similar deep neural network models with the same basic architecture as the Bert model include multiple encoding networks, each of which includes a self-attention layer and a feedforward neural network layer, in which the first encoding network layer accepts the input of the embedding vector corresponding to the text to be recognized, and in each encoding network layer, after the self-attention layer performs feature interaction on the input received, the text feature vector is obtained through processing by its feedforward neural layer, and is output to the next encoding network as its input, and the final text feature vector after multi-layer feature interaction is output in the last encoding network. For such neural network models that are isomorphic to the Bert model, the technical solutions of the present application can be used for improvement.
[0072] The text to be recognized can be a text in various languages, such as Chinese and English. According to the basic principle of the neural network model applicable to this application, the text to be recognized will be segmented to obtain multiple word units, and the encoding layer of the model is responsible for implementing the encoding corresponding to the inherent principle of the model. Taking the Bert model as an example, each word unit in the text to be recognized will be encoded to obtain a corresponding embedding vector, which is composed of token embedding (Token Embedding), segment embedding (Segment Embedding) and position embedding (Position Embedding), in which token embedding represents the label of the word unit in the preset dictionary, segment embedding represents the chapter where the word unit is located, and position embedding represents the absolute position information of the word unit in the text to be recognized.
[0073] The word-gram, in this application, refers to a vocabulary that can represent an independent meaning unit determined according to the vocabulary list corresponding to the word segmentation. The vocabulary may include a single word or multiple words. For example, for Chinese text, the word-gram may include one or more Chinese characters; for English text, the word-gram may include one or more words.
[0074] After the encoding layer of the model encodes each word unit to generate the corresponding embedding vector of each word unit, these embedding vectors are input into the first encoding network of the model to start representation learning.
[0075] Step S1200: Generate relative position feature information according to a plurality of relative distance scales between two word-grams in the plurality of word-grams:
[0076] The inherent Bert model and its derivative models, the position embedding in the word unit's embedding vector only serves to mark the absolute position information of the word unit, and does not represent the relative position information between any word units. In order to meet the requirements of this application, the relative position information between word units is added to the named entity recognition process of the text to be recognized, so this step is extended to encode the relative position information between word units.
[0077] When encoding the relative position information of word units, it is mainly implemented based on the relative distance of a scale between any two word units in all word units of the text segmentation to be identified. By determining the relative distance between any two word units under the same measurement standard, the relative distance information between the two word units is vectorized. Therefore, for each word unit, the relative distance information between it and all arbitrary word units can be obtained, thereby forming a distance vector corresponding to the word unit, and the distance vectors of all word units are spliced to form a coding matrix, accordingly, each coding matrix corresponds to a relative distance scale. All coding matrices together constitute the relative position feature information.
[0078] In the present application, all word units can be encoded based on multiple relative distance scales to obtain the relative position feature information, and each relative distance scale is a benchmark for measuring the relative distance between word units. For example, the first relative distance scale can be defined as the distance in number of words between the first character of the first word unit and the first character of the second word unit in two word units; the second relative distance scale can be defined as the distance in number of words between the first character of the first word unit and the last character of the second word unit in two word units; the third relative distance scale can be defined as the distance in number of words between the last character of the first word unit and the first character of the second word unit in two word units; the fourth relative distance scale can be defined as the distance in number of words between the last character of the first word unit and the last character of the second word unit in two word units.
[0079] For example, for the text to be recognized "Off-road motorcycle new peak season special offer", assuming its word segmentation set is {off-road; motorcycle; new; peak season; special offer}, taking the first token as "off-road" and the second token as "motorcycle", then at the first relative distance scale, the corresponding character distance is the character distance between the position order of the character "越" and the position order of the character "摩", that is, 1 - 3 = -2. Similarly, at the second relative distance scale, the corresponding character distance of these two tokens is 1 - 5 = -4; at the third relative distance scale, the corresponding character distance of these two tokens is 2 - 3 = -1; at the fourth relative distance scale, the corresponding character distance of these two tokens is 2 - 5 = -3.
[0080] Thus, it is not difficult to understand that by encoding all the tokens obtained by word segmentation of the text to be recognized according to different relative distance scales, relative position feature information representing the relative position relationship between each token can be obtained. This relative position feature information represents the relative character distance between tokens at each relative distance scale, characterizes the distance information between tokens within the text to be recognized, and the obtained relative position feature information is suitable for participating in the self-attention layer of the encoding network of the neural network model described in this application, enriching the context reference information in the operation process of the self-attention layer, so that the model can perceive the relative distance between each token.
[0081] Step S1300: Using the embedding vectors of each token as input, perform feature interaction through at least one encoding network containing a self-attention layer to generate a text feature vector, where the relative position feature information serves as the associated weight for generating the text feature vector.
[0082] Furthermore, to adapt to the process of model training or inference, input the embedding vectors of the text to be recognized into the neural network model of this application. As Figure 2 shown, first enter the first encoding network among its multiple encoding networks. In this encoding network, the self-attention layer is responsible for performing feature interaction operations on the embedding vectors of each token, enabling the features of each token to fully interact and fuse, and finally uniformly mapping them to a normalized intermediate information matrix for output to the feed-forward neural layer of this encoding network for further feature extraction processing, and the feed-forward neural layer further extracts the text feature vector.
[0083] According to the inherent principle of the self-attention layer, in the process of normalizing to obtain the intermediate information matrix, the output of the fully interactive embedding vectors of each word unit will be multiplied by a conversion weight matrix. In the present application, the encoded relative position feature information is used to correct the conversion weight matrix, and the relative position feature information and the conversion weight matrix are accumulated to achieve the correction of the conversion weight matrix. As a result, the relative position feature information representing the relative distance information between word units is integrated into the conversion weight matrix, so that the intermediate information matrix can obtain the ability to perceive the relative distance between word units, and the relative position feature information naturally constitutes the associated weight in the process of generating the text feature vector.
[0084] The neural network model described in the present application includes a plurality of the coding networks described, and the text feature vector output by the previous coding network is used as the input of the next coding network. Therefore, starting from the second coding network, there is no need to introduce the relative position feature information separately, and it is only necessary to rely on the text feature vector output by the first coding network for processing. As an alternative embodiment, the feasibility of introducing the relative position feature information in the self-attention layer of the plurality of coding networks described is not excluded. Similarly, as an alternative embodiment, when introducing the relative position feature information, it can be introduced in the self-attention layer of any one of the plurality of coding networks of the neural network model, and those skilled in the art should understand this. After the last coding network completes the feature interaction and outputs the text feature vector, it can be used to perform specific downstream tasks, such as realizing the named entity recognition described in the present application.
[0085] In the process of encoding the relative position feature information, the relative position feature information is associated with a learnable weight. Therefore, in the process of training the neural network model of the present application, the relative position feature information continuously corrects its corresponding weight through model training. When the model is trained to a convergence state, the optimal parameter corresponding to the weight is obtained. Therefore, the learnable weight can be corrected during the training process. Therefore, it is easier to adjust the generation of the text feature vector output by the encoding network. The relative position feature information is a representation of the relative distance information between word units. Therefore, through training, the representation learning efficiency of the model can be improved through the relative distance information, so that the model can be trained to a convergence state more quickly.
[0086] After the neural network model of the present application introduces the relative position feature information to process the embedding vector of the word unit of the text to be recognized to obtain the text feature vector, the text feature vector finally obtained through layer-by-layer reasoning is the result of perceiving the relative distance information between the word units, and therefore has an advantage over the inherent encoding principle of the model.
[0087] Step S1400: Identify named entities in the text to be identified based on the text feature vector:
[0088] The text feature vector output by the last encoding network of the neural network model of the present application is a probability matrix, which can be multiplied by the state matrix provided by the conditional random field model to complete the part-of-speech tagging, thereby determining the named entities therein. Since the neural network model of the present application fully refers to the relative distance information between the word units of the text to be identified during the reasoning process, the identified named entities are more accurate.
[0089] The neural network model of the present application can be trained in two stages. After the first stage is pre-trained to a convergence state, it enters the second stage of fine-tuning training. During the fine-tuning training process, the technical solution of the present application is used for corresponding encoding. Under the iterative training of an appropriate amount of training samples consisting of the text to be recognized, the model is finally trained to a convergence state, so that it can be put into practical scenarios to perform named entity recognition for the text to be recognized. When the model is put into use, the encoding process of the present application is also implemented on the text to be recognized, so that the model can correctly and accurately recognize the named entities therein.
[0090] Since the neural network model of the present application is based on the Bert model and its derivative models, and the training scheme of such models for realizing named entity recognition is well known to those skilled in the art, the training process thereof will not be described in detail.
[0091] Through this typical embodiment and its alternative embodiments, it can be seen that the present application has many advantages, including but not limited to the following aspects:
[0092] First of all, for the purpose of named entity recognition, the present application, for the basic encoding network that uses the self-attention mechanism for context combing, on the basis of its original encoding method, encodes each of the words obtained from the word segmentation of the to-be-recognized text at multiple relative distance scales to obtain the relative position feature information between the words, and adds the relative position feature information as the association weight to the self-attention operation process to control the generation of the text feature vector output by the encoding network, so that the text feature vector output by the encoding network carries the distance information between the words, and enriches the context information of the original word embedding vector with the relative position relationship between the words, so that in the representation learning process, the deep semantic information of the words can be more effectively represented according to the relative position of the words, thereby achieving a more accurate named entity recognition effect.
[0093] Secondly, the present application does not need to change the architecture of the encoding network. It only uses the relative position feature information generated by the encoding as the associated weight for the self-attention layer in the encoding network to correct the generation weight of the text feature vector. It is simple to implement and efficient to process. The encoding network can obtain a more sensitive perception ability to the position information of the text, which can improve the representation learning ability of the neural network model where the encoding network is located, and help to improve the training efficiency of the neural network model, making it easier to be fine-tuned and trained to a convergence state using a small number of samples, thereby saving training costs.
[0094] In addition, the technical solution of the present application is more suitable for processing product information in the e-commerce field. In view of the semantic characteristics of product information, especially the product title text containing multiple parallel words, the named entities therein can be accurately identified through the present application and applied to other downstream tasks, which can improve the efficiency of e-commerce information processing and enable the e-commerce platform to reap economies of scale.
[0095] See also Figure 3 In a specific embodiment, the step S1100, generating corresponding embedding vectors according to multiple word element codes obtained by segmenting the text to be recognized, includes the following steps:
[0096] Step S1110: Obtain the text to be identified, where the text to be identified is a title text used to describe product information:
[0097] The text to be recognized can be obtained from a user request, or it can be called from a database, such as the product title of a product object in a product database, or from a local storage space, such as a system clipboard. Depending on the different downstream tasks applied in this application, the input text can be flexibly obtained as the text to be recognized, and is used to perform the named entity recognition task and extract one or more keywords corresponding to the named entities from the text to be recognized.
[0098] The downstream tasks described depend on the specific application scenarios of the technical solutions of this application, such as online search, search keyword association, data profiling, recommended search, etc.
[0099] In one example, in an online search scenario, after the user submits his original search expression on the client device, the original search expression is used as the text to be recognized for named entity recognition, and multiple keywords are extracted. The keywords are then logically combined to form an optimized search expression and then an online search is performed.
[0100] In the second example, in the search keyword association scenario, when the user enters the search keyword, the text he has entered is used as the text to be recognized for named entity recognition, and then the background performs word association based on the keywords obtained by the named entity recognition, and the related associated words are clearly displayed to the user for selection.
[0101] Example three, in the data portrait scenario, taking the e-commerce field as an example, the text information such as the product title or product details of the product object can be directly obtained from the client device or from the product database as the text to be identified, and named entity recognition is performed to obtain the corresponding keywords, and the corresponding product objects are labeled, and these keywords are used as data portrait labels for the product objects.
[0102] Example 4: In the recommendation search scenario, still taking the e-commerce field as an example, the input of the client device can be obtained as the text to be recognized, and named entity recognition can be performed on it. The corresponding product objects can be searched in the product database based on the keywords obtained by recognition, and then a recommended list of product objects is constructed and pushed to the user end.
[0103] This indicates that the corresponding input text information can be obtained as the text to be recognized according to different application scenarios, so as to start the naming recognition process of this application. In fact, named entity recognition is a basic function in natural language processing technology, which can be widely applied in various occasions. Those skilled in the art are aware of this. Therefore, the scope of protection that should be covered by the creative spirit of this application should not be limited to the application field.
[0104] Step S1120: Use a pre-trained word segmentation model to segment the text to be recognized, and obtain a corresponding plurality of word units, wherein the word units are words composed of English words or Chinese characters as single words:
[0105] The pre-trained word segmentation model in the prior art can be used to segment the text to be recognized, so as to obtain a word segmentation set corresponding to a word unit stored in a preset dictionary ( Figure 2 Examples are w1 to w9), and the word segmentation set contains multiple word units extracted from the text to be recognized. The word segmentation model is correspondingly provided with a dictionary, which contains various word units referring to named entities. In order to realize the word embedding of the text to be recognized, first, based on the dictionary, all possible word segments containing the word are matched from the dictionary based on each word of the text to be recognized, and a word segmentation set is constructed for each character, which contains all possible word segments where the character appears. Word2Vec is recommended for the word segmentation model.
[0106] For example, for the text to be recognized "Nanjing Yangtze River Bridge", after word segmentation, the possible word segmentation set obtained is {Nanjing; Nanjing City; City; Mayor; Yangtze River; Yangtze River Bridge; River; River Bridge; Bridge}.
[0107] Step S1130: Encode the embedding vectors of each word-unit respectively, where the embedding vector of each word-unit includes the token embedding, position embedding and segment embedding of the word-unit:
[0108] like Figure 2 As shown, the word segmentation set is input into the encoding layer of the neural network model for performing named entity recognition of the present application for encoding, and an embedding vector of each word unit in the word segmentation set is constructed accordingly. As mentioned above, each embedding vector is a synthesis result of the token embedding, position embedding and segment embedding of the word unit.
[0109] In this embodiment, the application scenarios in the e-commerce field are adapted, the title text of the product information is obtained as the text to be recognized required for named entity recognition, and the text is segmented and an embedding vector corresponding to each word is constructed to prepare for named entity recognition. Since the title text of the product information is often a list of multiple nouns and adjectives, the application of the technical solution of the present application in this specific scenario can reap the benefits of economies of scale. By improving the recognition accuracy of the named entities in the title text, the execution efficiency of the e-commerce platform in terms of keyword search, recommendation, matching, product data profiling, keyword association, etc. is improved, thereby improving the user experience.
[0110] See also Figure 4 In a specific embodiment, the step S1200, encoding and generating relative position feature information according to a plurality of relative distance scales between two word units in the plurality of word units, comprises the following steps:
[0111] Step S1210: Encode the relative distance information between each word unit in each word unit according to different relative distance scales to obtain a coding matrix corresponding to each relative distance scale:
[0112] Combined with the foregoing, in the process of encoding to obtain the relative position feature information, it is first necessary to encode the relative distance information between the two word units in the word set of the text to be recognized according to multiple different relative distance scales. As a variant embodiment, on the basis of the typical embodiment of the present application, a single relative distance scale, or two or three relative distance scales can be used for encoding, and finally a coding matrix corresponding to each relative distance scale is obtained. It is not difficult to understand that each coding matrix contains the word count distance from each word unit in the word set to all arbitrary word units under the same measurement basis, and is organized in order in units of word units, thereby forming a coding format that meets the specifications of this application.
[0113] Step S1220: splice the coding matrices corresponding to the relative distance scales into a multi-channel coding matrix:
[0114] In order to realize the integration of the encoding matrices corresponding to the plurality of relative distance scales, the encoding matrices may be simply spliced together to form a multi-channel encoding matrix.
[0115] Step S1230: performing nonlinear transformation on the multi-channel encoding matrix matching learnable weights to activate the output and obtain relative position feature information:
[0116] In order to allow the relative distance information of the word unit to be corrected by the model when the gradient is updated, a nonlinear activation layer is used to activate and output the multi-channel encoding matrix by matching the learnable weights, thereby obtaining a two-dimensional matrix to constitute the relative position feature information. Later, in the training process of the neural network model of the present application, when the gradient is updated, the learnable weight can be corrected accordingly until the model is trained to a convergence state, and the learnable weight is fixed to reach an optimal state, so that the generation of the relative position feature information can be associated and controlled, making it suitable for guiding the self-attention layer to control the output of the text feature vector.
[0117] The example formula of the whole process of this embodiment is as follows:
[0118]
[0119] Among them, h represents the first word in the word unit, t represents the last word in the word unit, P represents the encoding matrix, d represents a relative distance scale, R i,j refers to the position encoding of the i-th word and the j-th word, W r is a learnable weight and ReLU is the activation function used by the nonlinear layer.
[0120] In this embodiment, the relative position feature information is obtained by simply splicing the encoding matrices corresponding to the respective relative distance scales obtained by encoding and matching the learnable weight activation outputs, so that the relative position feature information can participate in the model training process for parameter correction, thereby making the model easier to train to a convergence state and improving the training efficiency of the model.
[0121] See also Figure 5 In a further embodiment, the step S1210, encoding the relative distance information between each word unit in each word unit according to different relative distance scales to obtain a coding matrix corresponding to each relative distance scale, includes the following steps:
[0122] Step S1211: for each word unit, calculate the relative distance between the first character of the former and the first character of the latter between the word unit and any other word unit, obtain the first vector corresponding to each word unit, and concatenate the first vectors of all word units into a first encoding matrix:
[0123] Adapting to the first relative distance scale, according to the distance information between any two words determined by the position of the word in the text to be recognized, the relative distance between each word and each word in the word segmentation set is calculated. The specific algorithm is to subtract the position of the first word of the current word from the position of the first word of another word to obtain the relative distance between the two. The formula example is as follows:
[0124]
[0125] Among them, head represents the starting position of the word, that is, the position sequence number of its first character in the text to be recognized, and i and j represent two words respectively, the former is the current word, and the latter is another word relative to it.
[0126] After calculation, each current word can be compared with each word in the word set, and the corresponding distance value is calculated based on the relative distance between each other's first letters. All distance values are organized in order to form the first vector corresponding to the current word, and the first vectors of all word units are spliced together to form a two-dimensional matrix to obtain the corresponding first encoding matrix.
[0127] Step S1212: For each word unit, calculate the relative distance between the first character of the former and the last character of the latter with respect to any word unit, obtain the second vector corresponding to each word unit, and concatenate the second vectors of all word units into a second encoding matrix:
[0128] Adapting to the second relative distance scale, according to the distance information between any two words determined by the position of the word in the text to be recognized, the relative distance between each word and each word in the word segmentation set is calculated. The specific algorithm is to subtract the position of the first word of the current word from the position of the last word of another word to obtain the relative distance between the two. The formula example is as follows:
[0129]
[0130] Among them, head represents the starting position of the word, that is, the position sequence number of its first character in the text to be recognized, tail represents the ending position of the word, that is, the position sequence number of its last character in the text to be recognized, i, j represent two words respectively, the former is the current word, and the latter is the other word relative to it.
[0131] After calculation, each current word-unit can be compared with each word-unit in the word segmentation set, and the corresponding distance value is calculated based on the relative distance between the first character of the former and the last character of the latter. All distance values are organized in order to form the second vector corresponding to the current word-unit, and the second vectors of all word-units are spliced together to form a two-dimensional matrix to obtain the corresponding second encoding matrix.
[0132] Step S1213: for each word unit, calculate the relative distance between the last character of the former and the first character of the latter between the word unit and any other word unit, obtain the third vector corresponding to each word unit, and concatenate the third vectors of all word units into a third encoding matrix;
[0133] Adapting to the third relative distance scale, according to the distance information between any two words determined by the position of the word in the text to be recognized, the relative distance between each word and each word in the word segmentation set is calculated. The specific algorithm is to subtract the position of the first word of another word from the position of the last word of the current word to obtain the relative distance between the two. The formula example is as follows:
[0134]
[0135] Among them, head represents the starting position of the word, that is, the position sequence number of its first character in the text to be recognized, tail represents the ending position of the word, that is, the position sequence number of its last character in the text to be recognized, i, j represent two words respectively, the former is the current word, and the latter is the other word relative to it.
[0136] After calculation, each current word-unit can be compared with each word-unit in the word segmentation set, and the corresponding distance value can be calculated based on the relative distance between the last character of the former and the first character of the latter. All distance values are organized in order to form the third vector corresponding to the current word-unit, and the third vectors of all word-units are spliced together to form a two-dimensional matrix to obtain the corresponding third encoding matrix.
[0137] Step S1214: for each word unit, calculate the relative distance between the last character of the former and the last character of the latter word unit, obtain the fourth vector corresponding to each word unit, and splice the fourth vectors of all word units into a fourth encoding matrix:
[0138] Adapting to the fourth relative distance scale, according to the distance information between any two word units determined by the position of the word unit in the text to be recognized, the relative distance between each word unit and each word unit in the word segmentation set is calculated. The specific algorithm is to subtract the position sequence of the last word of the current word unit from the position sequence of the last word of another word unit to obtain the relative distance between the two. The formula example is as follows:
[0139]
[0140] Among them, tail represents the end position of the word unit, that is, the position sequence number of its last character in the text to be recognized, and i, j represent two words respectively, the former is the current word unit, and the latter is another word unit opposite to it.
[0141] After calculation, each current word-unit can be compared with each word-unit in the word segmentation set, and the corresponding distance value can be calculated based on the relative distance between each other's last characters. All distance values are organized in order to form the fourth vector corresponding to the current word-unit, and the fourth vectors of all word-units are spliced together to form a two-dimensional matrix to obtain the corresponding fourth encoding matrix.
[0142] According to this embodiment, the word segmentation set of the above example sentence "Nanjing Yangtze River Bridge" is: {Nanjing; Nanjing City; City; Mayor; Yangtze River; Yangtze River Bridge; River; River Bridge; Bridge}, and the first encoding matrix obtained corresponding to the first relative distance scale is as follows Figure 6 As shown in FIG. 1 , it is not difficult to see from the figure that the encoding matrix formed by encoding is a square matrix. Similarly, the encoding matrices corresponding to other relative distance scales can be calculated by those skilled in the art, so please forgive me for omitting them.
[0143] According to this embodiment, it can be understood that by vectorizing the relative distance information between word units in the word set from different relative distance scales, the encoding matrices corresponding to each relative distance scale are obtained, and each encoding matrix represents the relative distance from each word unit to any word unit in an orderly manner based on its corresponding relative distance scale, thereby achieving an effective representation of the relative position relationship between word units, which can be introduced into the neural network model of the present application to provide the self-attention layer with relative position information between word units, thereby enhancing the representation learning ability of the model and enabling the model to more accurately identify named entities in the text to be identified.
[0144] See also Figure 7 In a specific embodiment, the step S1300, taking the embedding vector of each word as input, performing feature interaction through at least one encoding network including a self-attention layer to generate a text feature vector, includes the following steps performed by the first encoding network:
[0145] Step S1310: Use the self-attention layer in the current encoding network to construct the corresponding query vector, key vector and value vector for the embedding vector of each word unit:
[0146] Please continue to combine Figure 2 , the current encoding network, in this embodiment, is the first encoding network at the input front end of the neural network model adopted by the present application.
[0147] Still taking the inherent principle of the encoding network of the Bert model as an example, the self-attention layer in the encoding network, when it receives the input of the word-embedded vector, can construct the corresponding query vector (Q), key vector (K), and value vector (V) for it. Constructing Q, K, and V vectors is the basic principle of the multi-head attention mechanism and the basic means to achieve deep interaction at the feature level.
[0148] Step S1320: The self-attention layer performs a weighted summation of the value vectors corresponding to all word units, where the weight corresponding to each value vector is the normalized score of the dot product of the corresponding key vector and the query vector corresponding to the other embedding vectors:
[0149] According to the inherent algorithm of the self-attention mechanism, for each embedding vector, the dot product of its key vector and the query vector corresponding to other embedding vectors is first calculated, and then the dot product result is normalized using the Softmax function so that the sum of multiple dot products is 1, and the normalized value is used as the weight corresponding to each embedding vector.
[0150] On this basis, the value vectors corresponding to all embedded vectors are multiplied by their corresponding weights and then added to achieve weighted summation, and a weighted summation matrix is obtained.
[0151] Step S1330: The self-attention layer multiplies the weighted summation result by the sum of the conversion weight matrix and the relative position feature information to obtain an intermediate information matrix, so that the relative position feature information constitutes the associated weight of the intermediate information matrix:
[0152] In the inherent algorithm of the self-attention mechanism, in order to standardize the output scale, the weighted summation result, that is, the summation matrix, is multiplied by a conversion weight matrix to obtain an intermediate information matrix. In this application, in order to allow the self-attention mechanism to introduce the relative position information of the word unit to enrich the context, in this step, the relative position feature information obtained by the encoding of this application is first matrix-added with the conversion weight matrix to obtain the summation result, and then the summation result is multiplied with the weighted summation result of the previous step, and then the intermediate information matrix is obtained. It can be understood that after the operation between Q, K, and V and the superposition of the relative position feature information obtained by the encoding of this application, the intermediate information matrix has realized the full interaction of the deep semantic features between each embedded vector, and the relative position feature information is used as the association weight, and the relative position information between the word units is introduced in the process of full feature interaction. Compared with the algorithm inherent to the model, the relative distance information of the word unit is added on the basis of the absolute position information inherently adopted by the model.
[0153] Step S1340: extract features from the intermediate information matrix through the feedforward neural layer to obtain a text feature vector as the input required by the self-attention layer of the next encoding network:
[0154] The intermediate information matrix is output to the multilayer perceptron of the feedforward neural layer of the current coding network for high-level semantic feature extraction to obtain a text feature vector, which can then be output to the next coding network. For the next coding network, it has the same structure as the first coding network, so there is no need to introduce the relative position feature information. Therefore, the text feature vector is subjected to the QKV operation of the inherent algorithm according to the process of the first coding network. If the coding network that outputs the text feature vector is the last coding network, the text feature vector is directly output as the result and connected to the conditional random field for part-of-speech tagging.
[0155] In the subsequent encoding network, although there is no need to introduce the relative position feature information separately, since the relative position feature information has been used as the associated weight for generating the text feature vector in the first encoding network, deep feature interaction is still performed on the basis of the text feature vector carrying the relative position feature information. Therefore, introducing the relative position feature information in the self-attention layer of the first encoding network can ensure that all feature extraction processes of the neural network model of the present application refer to the pre-encoded relative position feature information, thereby ensuring that the relative position feature information provides an important reference for the named entity recognition process.
[0156] In this embodiment, the organic combination of the relative position feature information obtained by encoding each word in the text to be recognized and the inherent algorithm of the self-attention layer is disclosed in detail. The relative position feature information is used as the associated weight that affects the generation of the text feature vector, which can comprehensively and collectively have a comprehensive impact and control on the calculation results of all embedded vectors of the self-attention layer, ensuring the effective generation of the text feature vector. The relative position feature information carries a learnable weight, so it is convenient to correct the model during training, thereby making the model easier to train, ensuring that under the action of the relative position feature information, the model is quickly trained to a convergence state and enables the model to learn the ability to accurately recognize named entities.
[0157] See also Figure 8 In a specific embodiment, the step S1400, identifying the named entity in the text to be identified according to the text feature vector, comprises the following steps:
[0158] Step S1411: Use a conditional random field model to perform part-of-speech tagging on the named entities to determine the part-of-speech of each named entity:
[0159] Please continue to combine Figure 2The conditional random field model (CRF) shown in the figure is used to implement part-of-speech tagging of the text feature vector obtained by the last encoding network of the Bert model. Specifically, the Bert model outputs a probability matrix, and the conditional random field model outputs a state matrix. After multiplying and converting the two, part-of-speech tagging can be completed, and each word unit marked with part of speech in the text to be recognized can be obtained as the recognized named entity.
[0160] Step S1412: Output each named entity to a storage location corresponding to the part of speech according to the part of speech, so as to form attribute data corresponding to the part of speech in the product information:
[0161] Since the named entities in the text to be identified output by the conditional random field model contain part-of-speech annotations, these parts of speech can be used as attribute data of product information and stored in a database corresponding to the parts of speech to achieve labeling of product objects. For example, a user enters a summary text for placing an order in the communication interface of the intelligent customer service system of an e-commerce platform as the text to be identified, such as "XX mobile phone, somewhere in Guangzhou, 135XXXXXXXX". After being identified by the neural network model of this application, "XX mobile phone" is the attribute data corresponding to the target product field in the order of product information, "somewhere in Guangzhou" is the attribute data corresponding to the delivery address field in the order information, and "135XXXXXXXX" is the attribute data corresponding to the contact number field of the order. These parts of speech can be stored one by one.
[0162] This embodiment illustrates an example of applying the neural network model for performing named entity tasks of the present application to obtain attribute data for product information. From this example, it can be seen that after accurately identifying the named entities in the text to be identified based on the technical solution of the present application, it can be automatically matched to the corresponding storage location, thereby improving the efficiency and accuracy of human-computer interaction. For large-scale network platforms such as e-commerce, it can greatly improve service efficiency and improve user experience.
[0163] See also Fig. 9 In a specific embodiment, the step S1400, identifying the named entity in the text to be identified according to the text feature vector, comprises the following steps:
[0164] Step S1421: Use a conditional random field model to perform part-of-speech tagging on the named entities to determine the part-of-speech of each named entity:
[0165] Similarly, please continue to refer to Figure 2The conditional random field model (CRF) is used to implement part-of-speech tagging of the text feature vector obtained by the last encoding network of the Bert model. Specifically, the Bert model outputs a probability matrix, and the conditional random field model outputs a state matrix. After multiplying and converting the two, part-of-speech tagging can be completed, and each word unit marked with part of speech in the text to be recognized can be obtained as the recognized named entity.
[0166] Step S1422: extract the named entity corresponding to the target part of speech according to the part of speech, accurately match the named entity with the preset entity dictionary, and output the accurately matched named entity as the keyword of the text to be recognized:
[0167] The named entities can be matched by pre-constructing multiple entity dictionaries belonging to different categories, so as to classify and identify the various keywords with different functions in the named entities. For example, multiple entity dictionaries are preset, corresponding to geographical nouns, commodity nouns, and commodity feature nouns. When the conditional random field identifies a named entity whose part of speech is a geographical noun, the named entity corresponding to the geographical noun can be matched with the entity dictionary that includes the geographical noun, generally for an exact match, wherein the named entity that achieves an exact match can be confirmed as a keyword contained in the text to be identified, and the matching of entity nouns of other parts of speech is the same.
[0168] This embodiment illustrates an example of applying the neural network model for performing named entity tasks of the present application to keyword recognition. It can be seen from this example that after accurately identifying the named entities in the text to be recognized based on the technical solution of the present application, it can be further used for keyword matching and recognition. For specific application scenarios such as search engines and product recommendations, it can play an auxiliary effect of accurate matching. For large-scale network platforms such as e-commerce, it can greatly improve service efficiency and improve user experience.
[0169] The coding idea of the present application is a very basic method, which is suitable for all models based on the Bert model and its derivatives. The improvement of such models will have a far-reaching impact in the field of natural language processing. Therefore, in addition to the various specific applications exemplified in the present application, those skilled in the art can of course also apply it to solve other specific technical problems. In this regard, it should be understood that it still does not deviate from the scope of protection that should be covered by the creative spirit of the present application.
[0170] See also Fig.10A named entity recognition device adapted to one of the purposes of the present application comprises: a word embedding module 1100, a distance encoding module 1200, a feature interaction module 1300, and an annotation recognition module 1400, wherein the word embedding module 1100 is used to generate corresponding embedding vectors according to multiple word-unit encodings obtained from word segmentation of a to-be-recognized text; the distance encoding module 1200 is used to generate relative position feature information according to multiple relative distance scale encodings between two word-units in the multiple word-units; the feature interaction module 1300 is used to take the embedding vectors of each word-unit as input, perform feature interaction through at least one encoding network including a self-attention layer, and generate a text feature vector, wherein the relative position feature information is used as an associated weight for generating the text feature vector; the annotation recognition module 1400 is used to identify the named entities in the to-be-recognized text according to the text feature vector.
[0171] In a specific embodiment, the word embedding module 1100 includes: a title acquisition submodule, used to acquire the text to be recognized, wherein the text to be recognized is a title text used to describe product information; a word segmentation processing submodule, used to use a pre-trained word segmentation model to segment the text to be recognized, and obtain a corresponding plurality of word units, wherein the word units are words composed of English words or Chinese characters; a vector encoding submodule, used to respectively encode the embedding vectors of each word unit, wherein the embedding vector of each word unit includes the token embedding, position embedding and segment embedding of the word unit.
[0172] In a specific embodiment, the distance coding module 1200 includes: a scale coding submodule, which is used to encode the relative distance information between each word element in each word element according to different relative distance scales to obtain a coding matrix corresponding to each relative distance scale; a coding splicing submodule, which is used to splice the coding matrices corresponding to each relative distance scale into a multi-channel coding matrix; and a coding synthesis submodule, which is used to perform nonlinear transformation on the multi-channel coding matrix matching learnable weights to activate the output and obtain relative position feature information.
[0173] In a further embodiment, the scale coding submodule includes: a first coding unit, which is used to calculate the relative distance between the first character of each word and the first character of any word, obtain the first vector corresponding to each word, and splice the first vectors of all word elements into a first coding matrix; a second coding unit, which is used to calculate the relative distance between the first character of each word and the last character of each word, obtain the second vector corresponding to each word, and splice the second vectors of all word elements into a second coding matrix; a third coding unit, which is used to calculate the relative distance between the last character of each word and the first character of the latter word, obtain the third vector corresponding to each word, and splice the third vectors of all word elements into a third coding matrix; a fourth coding unit, which is used to calculate the relative distance between the last character of each word and the last character of any word, obtain the fourth vector corresponding to each word, and splice the fourth vectors of all word elements into a fourth coding matrix.
[0174] In a specific embodiment, the first encoding network in the feature interaction module 1300 includes: a vector construction unit, which is used to use the self-attention layer in the current encoding network to construct the corresponding query vector, key vector and value vector for the embedding vector of each word unit; a weighted summation unit, which is used to perform weighted summation of the value vectors corresponding to all word units by the self-attention layer, and the corresponding weight of each value vector is the normalized score of the dot product of the corresponding key vector and the query vector corresponding to the other embedding vectors; a distance association unit, which is used to multiply the weighted summation result by the self-attention layer by the sum of the conversion weight matrix and the relative position feature information to obtain an intermediate information matrix, so that the relative position feature information constitutes the association weight of the intermediate information matrix; a feedforward processing unit, which is used to perform feature extraction on the intermediate information matrix through a feedforward neural layer to obtain a text feature vector as the input required by the self-attention layer of the next encoding network.
[0175] In a specific embodiment, the tagging identification module 1400 includes: a part-of-speech tagging submodule, which is used to use a conditional random field model to perform part-of-speech tagging on the named entities to determine the part of speech of each named entity; an information entry submodule, which is used to output each named entity to a storage location corresponding to the part of speech according to the part of speech, so as to constitute attribute data corresponding to the part of speech in the product information.
[0176] In a specific embodiment, the annotation recognition module 1400 includes: a part-of-speech tagging submodule, which is used to use a conditional random field model to perform part-of-speech tagging on the named entities to determine the part-of-speech of each named entity; a keyword extraction submodule, which is used to extract the named entities corresponding to the target part-of-speech according to the part-of-speech, accurately match the named entities with a preset entity dictionary, and output the accurately matched named entities as keywords of the text to be identified.
[0177] A word embedding module 1100, a distance encoding module 1200, a feature interaction module 1300, and an annotation recognition module 1400, wherein the word embedding module 1100 is used to generate corresponding embedding vectors according to multiple word-unit encodings obtained from word segmentation of a text to be recognized; the distance encoding module 1200 is used to generate relative position feature information according to multiple relative distance scale encodings between two word-units in the multiple word-units; the feature interaction module 1300 is used to take the embedding vector of each word-unit as input, perform feature interaction through at least one encoding network including a self-attention layer, and generate a text feature vector, wherein the relative position feature information is used as an associated weight for generating the text feature vector; the annotation recognition module 1400 is used to identify named entities in the text to be recognized according to the text feature vector.
[0178] In a specific embodiment, the word embedding module 1100 includes: a title acquisition submodule, used to acquire the text to be recognized, wherein the text to be recognized is a title text used to describe product information; a word segmentation processing submodule, used to use a pre-trained word segmentation model to segment the text to be recognized, and obtain a corresponding plurality of word units, wherein the word units are words composed of English words or Chinese characters; a vector encoding submodule, used to respectively encode the embedding vectors of each word unit, wherein the embedding vector of each word unit includes the token embedding, position embedding and segment embedding of the word unit.
[0179] In a specific embodiment, the distance coding module 1200 includes: a scale coding submodule, which is used to encode the relative distance information between each word element in each word element according to different relative distance scales to obtain a coding matrix corresponding to each relative distance scale; a coding splicing submodule, which is used to splice the coding matrices corresponding to each relative distance scale into a multi-channel coding matrix; and a coding synthesis submodule, which is used to perform nonlinear transformation on the multi-channel coding matrix matching learnable weights to activate the output and obtain relative position feature information.
[0180] In a further embodiment, the scale coding submodule includes: a first coding unit, which is used to calculate the relative distance between the first character of each word and the first character of any word, obtain the first vector corresponding to each word, and splice the first vectors of all word elements into a first coding matrix; a second coding unit, which is used to calculate the relative distance between the first character of each word and the last character of each word, obtain the second vector corresponding to each word, and splice the second vectors of all word elements into a second coding matrix; a third coding unit, which is used to calculate the relative distance between the last character of each word and the first character of the latter word, obtain the third vector corresponding to each word, and splice the third vectors of all word elements into a third coding matrix; a fourth coding unit, which is used to calculate the relative distance between the last character of each word and the last character of any word, obtain the fourth vector corresponding to each word, and splice the fourth vectors of all word elements into a fourth coding matrix.
[0181] In a specific embodiment, the first encoding network in the feature interaction module 1300 includes: a vector construction unit, which is used to use the self-attention layer in the current encoding network to construct the corresponding query vector, key vector and value vector for the embedding vector of each word unit; a weighted summation unit, which is used to perform weighted summation of the value vectors corresponding to all word units by the self-attention layer, and the corresponding weight of each value vector is the normalized score of the dot product of the corresponding key vector and the query vector corresponding to the other embedding vectors; a distance association unit, which is used to multiply the weighted summation result by the self-attention layer by the sum of the conversion weight matrix and the relative position feature information to obtain an intermediate information matrix, so that the relative position feature information constitutes the association weight of the intermediate information matrix; a feedforward processing unit, which is used to perform feature extraction on the intermediate information matrix through a feedforward neural layer to obtain a text feature vector as the input required by the self-attention layer of the next encoding network.
[0182] In a specific embodiment, the tagging identification module 1400 includes: a part-of-speech tagging submodule, which is used to use a conditional random field model to perform part-of-speech tagging on the named entities to determine the part of speech of each named entity; an information entry submodule, which is used to output each named entity to a storage location corresponding to the part of speech according to the part of speech, so as to constitute attribute data corresponding to the part of speech in the product information.
[0183] In a specific embodiment, the annotation recognition module 1400 includes: a part-of-speech tagging submodule, which is used to use a conditional random field model to perform part-of-speech tagging on the named entities to determine the part-of-speech of each named entity; a keyword extraction submodule, which is used to extract the named entities corresponding to the target part-of-speech according to the part-of-speech, accurately match the named entities with a preset entity dictionary, and output the accurately matched named entities as keywords of the text to be identified.
[0184] In order to solve the above technical problems, the present application also provides a computer device. Fig.11 As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a named entity recognition method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute the named entity recognition method of the present application. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0185] In this embodiment, the processor is used to execute Fig.10 The memory stores the program code and various data required to execute the above modules or submodules. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules / submodules in the named entity recognition device of this application, and the server can call the program code and data of the server to execute the functions of all submodules.
[0186] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the named entity recognition method of any embodiment of the present application.
[0187] The present application also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any embodiment of the present application when executed by one or more processors.
[0188] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0189] In summary, the present application improves the encoding method, encodes the word units in the text to be recognized at multiple distance scales to obtain the corresponding relative position feature information, so that the self-attention layer in the encoding network used for named entity recognition can use the relative position feature information to control the representation learning process and obtain the corresponding text feature vector, so that the text feature vector refers to the relative position information of the word unit, thereby improving the accuracy of named entity recognition. It is suitable for processing downstream tasks such as keyword extraction for product text information in the e-commerce field, and has a basic and broad application prospect.
[0190] Those skilled in the art will appreciate that the various operations, methods, steps, measures, and schemes in the processes discussed in this application may be alternated, altered, combined, or deleted. Further, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted. Further, the steps, measures, and schemes in the prior art that are similar to those disclosed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted.
[0191] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A method for named entity recognition, characterized in that: The steps include: Generate corresponding embedding vectors based on multiple word-element encodings obtained from the word segmentation of the text to be recognized; Generate relative position feature information according to a plurality of relative distance scales between two word-grams in the plurality of word-grams; Taking the embedding vector of each word as input, at least one encoding network including a self-attention layer performs feature interaction to generate a text feature vector, wherein the relative position feature information is used as the associated weight for generating the text feature vector, including the following steps performed by the first encoding network: using the self-attention layer in the current encoding network to construct the corresponding query vector, key vector and value vector for the embedding vector of each word; the self-attention layer performs weighted summation on the value vectors corresponding to all word units, and the corresponding weight of each value vector is the normalized score of the dot product of the corresponding key vector and the query vector corresponding to the other embedding vectors; the self-attention layer multiplies the weighted summation result by the sum of the conversion weight matrix and the relative position feature information to obtain an intermediate information matrix, so that the relative position feature information constitutes the associated weight of the intermediate information matrix; the feedforward neural layer performs feature extraction on the intermediate information matrix to obtain a text feature vector as the input required by the self-attention layer of the next encoding network; The named entities in the text to be recognized are identified according to the text feature vector.
2. The method for named entity recognition according to claim 1, characterized in that: Generate corresponding embedding vectors based on multiple word element encodings obtained from the word segmentation of the text to be recognized, including the following steps: Acquire the text to be recognized, where the text to be recognized is a title text used to describe product information; Using a pre-trained word segmentation model to segment the text to be recognized, and obtaining a corresponding plurality of word units, wherein the word units are words composed of English words or Chinese characters as single words; The embedding vectors of each word unit are encoded separately, and the embedding vector of each word unit includes the token embedding, position embedding and segment embedding of the word unit.
3. The method for named entity recognition according to claim 1, characterized in that: The relative position feature information is generated according to a plurality of relative distance scales between two word units in the plurality of word units, comprising the following steps: Encoding the relative distance information between each word unit in each word unit according to different relative distance scales to obtain a coding matrix corresponding to each relative distance scale; The coding matrices corresponding to the relative distance scales are concatenated into a multi-channel coding matrix; The multi-channel encoding matrix matching learnable weights are nonlinearly transformed to activate the output and obtain relative position feature information.
4. The method for named entity recognition according to claim 3, characterized in that: Encoding the relative distance information between each word unit in each word unit according to different relative distance scales to obtain a coding matrix corresponding to each relative distance scale includes the following steps: For each word unit, calculate the relative distance between the first character of the former and the first character of the latter between the word unit and any other word unit, obtain the first vector corresponding to each word unit, and concatenate the first vectors of all word units into a first encoding matrix; For each word unit, the relative distance between the first character of the former and the last character of the latter is calculated with respect to any word unit, and the second vector corresponding to each word unit is obtained, and the second vectors of all word units are concatenated into a second encoding matrix; For each word unit, calculate the relative distance between the last character of the former and the first character of the latter between the word unit and any other word unit, obtain the third vector corresponding to each word unit, and concatenate the third vectors of all word units into a third encoding matrix; For each word unit, the relative distance between the last character of the former and the last character of the latter is calculated to obtain the fourth vector corresponding to each word unit, and the fourth vectors of all word units are concatenated into a fourth encoding matrix.
5. The method for named entity recognition according to any one of claims 1 to 4, characterized in that: Identifying the named entities in the text to be identified according to the text feature vector includes the following steps: Using a conditional random field model to perform part-of-speech tagging on the named entities to determine the part-of-speech of each named entity; According to the part of speech, each named entity is output to a storage location corresponding to the part of speech to form attribute data corresponding to the part of speech in the product information.
6. The method for named entity recognition according to any one of claims 1 to 4, characterized in that: Identifying the named entities in the text to be identified according to the text feature vector includes the following steps: Using a conditional random field model to perform part-of-speech tagging on the named entities to determine the part-of-speech of each named entity; The named entities corresponding to the target part of speech are extracted according to the part of speech, the named entities are accurately matched with a preset entity dictionary, and the named entities that achieve accurate matching are output as keywords of the text to be recognized.
7. A computer device comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 6.
8. A computer-readable storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 6 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.
9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.
