Named entity identification method and device, computer equipment and storage medium

Abstract

The invention relates to the field of natural language processing, and provides a named entity recognition method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining a to-be-recognized target text and a target image corresponding to the target text; encoding the target text through a text encoding network to obtain target text features, and performing gridding processing on the target text features through a multi-head attention network to obtain target class image features; performing feature extraction on the target image through a feature extraction network to obtain a target image feature, the target image feature and the target image feature having the same representation form; and fusing the target image features and the target image features through a feature fusion layer, and performing entity recognition on the fused features through an entity recognition layer to obtain an entity recognition result. According to the method, the accuracy of multi-modal named entity recognition is improved.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a method, apparatus, computer device and storage medium for named entity recognition. Background Art

[0002] Named entity recognition is a basic problem in natural language processing, which aims to discover named entities in text and classify them into predefined categories, such as names of people, places or organizations. With the continuous increase in data volume and the diversification of data forms, single text data processing can no longer meet the needs of complex scenarios. Therefore, multimodal named entity recognition technology has emerged.

[0003] At present, the mainstream research trend of traditional multimodal named entity recognition is how to effectively integrate information from different modalities. Currently, a two-stream structure based on the attention mechanism is mostly used, one stream processes images and the other stream processes texts, and then features are aligned and fused through the attention layer, and finally the sequence labeling method is used to identify entities. However, these methods have the problem of poor alignment between image features and text features, and cannot guarantee the accuracy of named entity recognition in complex scenes.

[0004] For example, in the insurance claim scenario, traditional multimodal named entity recognition cannot accurately identify the entities contained in medical texts such as medical records and examination sheets uploaded by patients, as well as the corresponding medical images. For another example, in the field of medical health, traditional multimodal named entity recognition cannot accurately identify the entities contained in medical record text data and the corresponding medical imaging data. Therefore, how to improve the accuracy of multimodal named entity recognition is an urgent problem to be solved. Summary of the invention

[0005] The embodiments of the present application provide a method, apparatus, computer device and storage medium for named entity recognition, aiming to improve the accuracy of multimodal named entity recognition.

[0006] In a first aspect, an embodiment of the present application provides a method for named entity recognition, which is applied to a computer device, wherein the computer device is deployed with a target entity recognition model, wherein the target entity recognition model includes a text encoding network, a multi-head attention network, a feature extraction network, a feature fusion layer, and an entity recognition layer, and the method includes:

[0007] Acquire a target text to be recognized and a target image corresponding to the target text;

[0008] The target text is encoded by the text encoding network to obtain text features, and the text features are gridded by the multi-head attention network to obtain target class image features;

[0009] Extracting features of the target image through the feature extraction network to obtain target image features, wherein the target class image features have the same expression form as the target image features;

[0010] The target class image features and the target image features are fused through the feature fusion layer, and entity recognition is performed on the fused features through the entity recognition layer to obtain an entity recognition result.

[0011] In a second aspect, an embodiment of the present application further provides a named entity recognition device, which is applied to a computer device, wherein the computer device is deployed with a target entity recognition model, wherein the target entity recognition model includes a text encoding network, a multi-head attention network, a feature extraction network, a feature fusion layer, and an entity recognition layer, and the named entity recognition device includes:

[0012] An acquisition module, used to acquire a target text to be recognized and a target image corresponding to the target text;

[0013] A text encoding module, used for encoding the target text through the text encoding network to obtain target text features;

[0014] A grid processing module, used for performing grid processing on the target text features through the multi-head attention network to obtain target class image features;

[0015] An extraction module, used for performing feature extraction on the target image through the feature extraction network to obtain target image features, wherein the target class image features have the same expression form as the target image features;

[0016] A feature fusion module, used for fusing the target class image features and the target image features through the feature fusion layer;

[0017] The entity recognition module is used to perform entity recognition on the fused features through the entity recognition layer to obtain an entity recognition result.

[0018] In a third aspect, an embodiment of the present application further provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the named entity recognition method described in the first aspect are implemented.

[0019] In a fourth aspect, an embodiment of the present application further provides a storage medium for computer-readable storage, wherein a computer program is stored on the storage medium, wherein when the computer program is executed by a processor, the steps of the named entity recognition method as described in the first aspect are implemented.

[0020] The embodiments of the present application provide a method, apparatus, computer device and storage medium for named entity recognition. The embodiments of the present application convert text features obtained by encoding a target text into target-class image features, and extract features of a target image through a feature extraction network to obtain target image features. Meanwhile, the target-class image features and the target image features have the same expression form, so that after the target-class image features and the target image features are fused, the text features and the image features can be well aligned, and multimodal information is effectively aggregated, so that the entity recognition layer can accurately perform entity recognition on the fused features, effectively improving the accuracy of multimodal named entity recognition in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 This is a schematic diagram of a scenario of a method for named entity recognition provided by an embodiment of the present application;

[0023] Figure 2 is a schematic diagram of a network structure of a target entity recognition model in an embodiment of the present application;

[0024] Figure 3 It is a schematic diagram of the steps of a named entity recognition method provided by an embodiment of the present application;

[0025] Figure 4 is another schematic diagram of the network structure of the target entity recognition model in the embodiment of the present application;

[0026] Figure 5 is a schematic block diagram of a named entity recognition device provided in an embodiment of the present application;

[0027] Figure 6 It is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0028] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0031] At present, the mainstream research trend of traditional multimodal named entity recognition is how to effectively integrate information from different modalities. At present, a two-stream structure based on the attention mechanism is mostly used, one stream processes images and the other stream processes texts, and then the features are aligned and fused through the attention layer, and finally the sequence labeling method is used to identify entities. However, these methods have the problem of poor alignment between image features and text features, and cannot guarantee the accuracy of named entity recognition in complex scenarios. For example, in the insurance claims scenario, traditional multimodal named entity recognition cannot accurately identify the medical texts such as medical records and examination sheets uploaded by patients, as well as the entities contained in the corresponding medical images. For another example, in the field of medical health, traditional multimodal named entity recognition cannot accurately identify the entities contained in medical record text data and the corresponding medical imaging data.

[0032] To solve the above problems, the embodiments of the present application provide a method, apparatus, computer device and storage medium for named entity recognition. The embodiments of the present application convert the text features obtained by encoding the target text into target class image features, and extract the features of the target image through a feature extraction network to obtain target image features. At the same time, the target class image features and the target image features have the same expression form, so that after the target class image features and the target image features are fused, the text features and the image features can be well aligned, and multimodal information is effectively aggregated, so that the entity recognition layer can accurately perform entity recognition on the fused features, effectively improving the accuracy of multimodal named entity recognition in complex scenarios.

[0033] Among them, the named entity recognition method can be applied to a computer device, which can be a terminal device or a server. The terminal device can include a mobile phone, a tablet computer, a laptop computer, a desktop computer, a personal digital assistant or a wearable device, etc., but is not limited to this. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms, but is not limited to this.

[0034] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0035] See also Figure 1 , Figure 1 This is a scenario diagram of a named entity recognition method provided in an embodiment of the present application.

[0036] like Figure 1 As shown, the scenario includes a server 11 and a terminal device 12, and the server 11 is connected to the terminal device 12 in communication. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this. Among them, the target entity recognition model can be deployed on the server 11 or on the terminal device 12. Figure 2 As shown, the target entity recognition model deployed on the server 11 or the terminal device 12 includes a text encoding network 110, a multi-head attention network 120, a feature extraction network 130, a feature fusion layer 140 and an entity recognition layer 150. Among them, the text encoding network 110 is connected to the multi-head attention network 120, the multi-head attention network 120 and the feature extraction network 130 are respectively connected to the feature fusion layer 140, and the feature fusion layer 140 is connected to the entity recognition layer 150.

[0037] In some embodiments, the target entity recognition model is deployed on the server 11. The server 11 obtains the target text to be recognized and the target image corresponding to the target text sent by the terminal device; encodes the target text through the text encoding network to obtain the target text features, and grids the text features through the multi-head attention network to obtain the target class image features; extracts the features of the target image through the feature extraction network to obtain the target image features, and the target class image features have the same expression form as the target image features; fuses the target class image features and the target image features through the feature fusion layer, and performs entity recognition on the fused features through the entity recognition layer to obtain the entity recognition result, and sends the entity recognition result to the terminal device 12 so that the terminal device 12 outputs the entity recognition result.

[0038] For example, in the insurance claim process, the user uploads the patient's medical records, examination sheets and other medical texts and corresponding medical images to the server 12 through the terminal device 12. The server 12 uses the target entity recognition model to perform entity recognition on the medical texts and the corresponding medical images to obtain the entities required for insurance claims, such as the patient's name, hospital, department, time of visit, diagnosis content, etc. For another example, in the medical auxiliary diagnosis scenario, the doctor can upload the medical record text data and the corresponding medical image data to the server 11 through the terminal device 12. The server 11 uses the target entity recognition model to perform entity recognition on the medical record text data and the corresponding medical image data to obtain the entities required for medical auxiliary diagnosis.

[0039] In some embodiments, multiple terminal devices 12 are deployed with target entity recognition models. The terminal device 12 obtains the target text to be recognized and the target image corresponding to the target text; encodes the target text through a text encoding network to obtain target text features, and grids the text features through a multi-head attention network to obtain target class image features; extracts features of the target image through a feature extraction network to obtain target image features, and the target class image features have the same expression form as the target image features; fuses the target class image features and the target image features through a feature fusion layer, and performs entity recognition on the fused features through an entity recognition layer to obtain entity recognition results, and sends the entity recognition results to a server 11, which stores the entity recognition results sent by multiple terminal devices 12, so as to subsequently construct a knowledge graph based on the stored entity data. For example, in investment research, the named entity recognition method provided in the embodiment of the present application can be used to extract investment-related entities to construct a knowledge graph, so as to find out valuable information such as leading companies in the industry, potential partners, and development trends of the industry, etc.

[0040] See also Figure 3 , Figure 3It is a flowchart of a named entity recognition method provided in an embodiment of the present application.

[0041] like Figure 3 As shown, the named entity recognition method includes steps S101 to S104.

[0042] Step S101: Obtain a target text to be recognized and a target image corresponding to the target text.

[0043] In this embodiment, the target text and the target image can be any matching text-image pair, and this embodiment does not specifically limit this. For example, the target text is a medical text such as a patient's medical record or examination form, and the target image is a medical image corresponding to the medical text, such as a patient's imaging examination image or pathological image. For another example, the target text is an advertisement text posted on social media, and the target image is an advertisement image corresponding to the advertisement text.

[0044] Step S102: encode the target text through a text encoding network to obtain target text features, and grid the target text features through a multi-head attention network to obtain target class image features.

[0045] In this embodiment, the target text feature is a sequence of multiple one-dimensional text vectors. The target image feature is a sequence of multiple feature maps.

[0046] In some embodiments, Figure 4 As shown, the text encoding network includes a BERT (Bidirectional Encoder Representation from Transformers) model, a BiLSTM (Bi-directional Long Short-Term Memory) model and a linear layer. Encoding the target text through the text encoding network to obtain the target text feature may include: encoding the target text through the BERT model to obtain a first text feature; context encoding the first text feature through the BiLSTM model to obtain a second text feature; processing the second text feature through the linear layer to obtain the target text feature. For example, the input target text is X = {x1, x2, ..., x n}, the target text feature obtained by processing the target text with the BERT model, BiLSTM model and linear layer is H x , and H x =BiLSTM(BERT(X)). This embodiment performs dual encoding on the target text through the BERT model and the BiLSTM model, so as to more deeply extract and understand various features and dependencies in the text, obtain rich text features, and improve the accuracy of named entity recognition.

[0047] In some embodiments, the multi-head attention network only includes a multi-head attention layer, the weight parameters of the multi-head attention layer include a Query weight matrix and a Key weight matrix, the bias parameters of the multi-head attention layer include a Query bias vector and a Key bias vector, and gridding the target text features through the multi-head attention network to obtain target class image features may include: determining the Query vector matrix based on the Query weight matrix, the Query bias vector and the target text features; determining the Key vector matrix based on the Key weight matrix, the Key bias vector and the target text features; determining the transposed matrix of the Query vector matrix, and multiplying the transposed matrix of the Query vector matrix with the Key vector matrix to obtain the target class image features.

[0048] In some embodiments, Figure 4 As shown, the multi-head attention network includes a multi-head attention layer and a first multi-layer perceptron (MLP) layer. Gridding the target text features through the multi-head attention network to obtain the target class image features may include: gridding the target text features through the multi-head attention layer to obtain the initial class image features; processing the initial class image features through the first multi-layer perceptron layer to obtain the target class image features. In this embodiment, after gridding the target text features through the multi-head attention layer, the gridded target text features are processed by the first multi-layer perceptron layer, so that more discriminative and representative high-level target class image features can be obtained, thereby improving the accuracy of named entity recognition.

[0049] In some embodiments, the weight parameters of the multi-head attention layer include a Query weight matrix and a Key weight matrix, the bias parameters of the multi-head attention layer include a Query bias vector and a Key bias vector, and gridding the target text features through the multi-head attention layer to obtain initial image-like features may include: determining the Query vector matrix based on the Query weight matrix, the Query bias vector and the target text features; determining the Key vector matrix based on the Key weight matrix, the Key bias vector and the target text features; determining the transposed matrix of the Query vector matrix, and multiplying the transposed matrix of the Query vector matrix with the Key vector matrix to obtain the initial image-like features.

[0050] For example, Figure 4 As shown, the target text feature is H x , the query vector matrix is ​​Q, and Q = W q H x +b q , the Key vector matrix is ​​K, and K = W k Hx +b k Among them, W q is the Query weight matrix, b q is the Query bias vector, W k is the Key weight matrix, b k is the Key bias vector, so the initial class image feature can be expressed as The target image feature can be expressed as

[0051] Step S103: extract features of the target image through a feature extraction network to obtain target image features. The target class image features have the same expression form as the target image features.

[0052] In this embodiment, the target class image feature and the target image feature have the same expression form, that is, both the target class image feature and the target image feature are sequences containing multiple feature graphs. Figure 4 As shown, the target image features output by the first multi-layer perceptron layer are a feature map sequence, and the target image features output by the second multi-layer perceptron layer are another feature map sequence.

[0053] In some embodiments, Figure 4 As shown, the feature extraction network includes a convolution layer and a second multi-layer perceptron layer. Extracting features of the target image through the feature extraction network to obtain the target image features may include: performing convolution processing on the target image through the convolution layer to obtain initial image features; and processing the initial image features through the second multi-layer perceptron layer to obtain target image features. For example, the input target image is M, and the target image feature obtained after the target image M is processed by the convolution layer and the second multi-layer perceptron layer is H. m =MLP(Convs(M)). This embodiment processes the target image through the convolution layer and the multi-layer perceptron layer, and can obtain more discriminative and representative high-level target image features, thereby improving the accuracy of named entity recognition.

[0054] Step S104: The target class image features and the target image features are fused through the feature fusion layer, and entity recognition is performed on the fused features through the entity recognition layer to obtain an entity recognition result.

[0055] In this embodiment, fusing the target class image features and the target image features may include concatenating the target class image features and the target image features, and the entity recognition layer may be a multi-layer perceptron layer. Figure 4 As shown, the third multi-layer perceptron layer concatenates the concatenated features concat(S,H m) to perform entity recognition, and the entity recognition result can be expressed as: P = MLP (concat (S, H m ), S is the target image feature, H m is the target image feature, P is t l×l matrices, t is the total number of entity categories, l is the length of the word, if you need to judge (x i ,x j ) is an entity, then take the entity category with the largest probability value among the t entity categories at position (i, j). If the largest probability value is greater than zero, then we can determine whether (x i ,x j ) is an entity, and the maximum probability value is less than zero, then (x i ,x j ) is not an entity. Figure 4 The network structures of the first multi-layer perceptron layer, the second multi-layer perceptron layer and the third multi-layer perceptron layer are different, and the model parameters are different.

[0056] In some embodiments, before step S101, the method further includes: obtaining a training sample set, the training sample set includes multiple training samples, the training samples include training text, training images matching the training text, and annotated entity sequences; according to the training sample set, training the initial entity recognition model until convergence to obtain a target entity recognition model, the network structure of the initial entity recognition model and the target entity recognition model are the same, and the model parameters of the initial entity recognition model and the target entity recognition model are different. For example, the network structures of the initial entity recognition model and the target entity recognition model are both as follows: Figure 2 or Figure 4 The network structure shown in this embodiment can train a target entity recognition model suitable for complex scenarios, thereby improving the accuracy of multimodal named entity recognition.

[0057] In some embodiments, according to the training sample set, training the initial entity recognition model to convergence to obtain the target entity recognition model may include: selecting a training sample from the training sample set as the current training sample; encoding the training text in the current training sample through the text encoding network in the initial entity recognition model to obtain training text features; gridding the training text features through the multi-head attention network in the initial entity recognition model to obtain training class image features; extracting features of the training image in the current training sample through the feature extraction network in the initial entity recognition model to obtain training image features; fusing the training class image features and the training image features through the feature fusion layer in the initial entity recognition model to obtain fused features; performing entity recognition on the fused features through the entity recognition layer in the initial entity recognition model to obtain a predicted entity sequence; calculating a cross entropy loss value based on the predicted entity sequence and the labeled entity sequence in the current training sample; when the cross entropy loss value is greater than a preset loss value, updating the model parameters of the initial entity recognition model, and returning to execute the step of selecting a training sample from the training sample set as the current training sample; when the cross entropy loss value is less than or equal to the preset loss value, determining that the initial entity recognition model converges to obtain the target entity recognition model.

[0058] In some embodiments, the text encoding network includes a BERT model, a BiLSTM model and a linear layer, and the training text is encoded by the text encoding network in the initial entity recognition model to obtain training text features, including: encoding the training text by the BERT model to obtain a first initial text feature; context encoding the first initial text feature by the BiLSTM model to obtain a second initial text feature; processing the second initial text feature by a linear layer to obtain a training text feature.

[0059] In some embodiments, the multi-head attention network includes a multi-head attention layer and a first multi-layer perceptron layer. Gridding the training text features through the multi-head attention network in the initial entity recognition model to obtain training image features may include: gridding the training text features through the multi-head attention layer to obtain candidate image features; processing the candidate image features through the first multi-layer perceptron layer to obtain training image features.

[0060] Furthermore, the weight parameters of the multi-head attention layer include a Query weight matrix and a Key weight matrix, the bias parameters of the multi-head attention layer include a Query bias vector and a Key bias vector, and gridding the training text features through the multi-head attention layer to obtain candidate image features may include: determining a Query vector matrix based on the Query weight matrix, the Query bias vector and the training text features; determining a Key vector matrix based on the Key weight matrix, the Key bias vector and the training text features; determining a transposed matrix of the Query vector matrix, and multiplying the transposed matrix of the Query vector matrix with the Key vector matrix to obtain candidate image features.

[0061] In some embodiments, the feature extraction network includes a convolutional layer and a second multi-layer perceptron layer. The feature extraction network in the initial entity recognition model performs feature extraction on the training image to obtain training image features, including: performing convolution processing on the training image through the convolutional layer to obtain candidate image features; and processing the candidate image features through the second multi-layer perceptron layer to obtain training image features.

[0062] See also Figure 5 , Figure 5 It is a schematic block diagram of the structure of a named entity recognition device provided in an embodiment of the present application.

[0063] like Figure 5 As shown, the named entity recognition device 100 includes:

[0064] An acquisition module 110 is used to acquire a target text to be recognized and a target image corresponding to the target text;

[0065] A text encoding module 120, configured to encode the target text through the text encoding network to obtain a target vector;

[0066] A gridding processing module 130 is used to perform gridding processing on the target vector through the multi-head attention network to obtain target class image features;

[0067] An extraction module 140 is used to extract features of the target image through the feature extraction network to obtain target image features, wherein the target class image features have the same expression form as the target image features;

[0068] A feature fusion module 150, configured to fuse the target class image feature and the target image feature through the feature fusion layer;

[0069] The entity recognition module 160 is used to perform entity recognition on the fused features through the entity recognition layer to obtain an entity recognition result.

[0070] In some embodiments, the multi-head attention network includes a multi-head attention layer and a first multi-layer perceptron layer, and the grid processing module 130 is further used to:

[0071] Gridding the target text features through the multi-head attention layer to obtain initial image-like features;

[0072] The initial class image features are processed by the first multi-layer perceptron layer to obtain target class image features.

[0073] In some embodiments, the weight parameters of the multi-head attention layer include a Query weight matrix and a Key weight matrix, the bias parameters of the multi-head attention layer include a Query bias vector and a Key bias vector, and the grid processing module 130 is further used to:

[0074] Determine a Query vector matrix according to the Query weight matrix, the Query bias vector and the target text feature;

[0075] Determine a Key vector matrix according to the Key weight matrix, the Key bias vector and the target text feature;

[0076] Determine the transposed matrix of the Query vector matrix, and perform multiplication operation on the transposed matrix of the Query vector matrix and the Key vector matrix to obtain initial class image features.

[0077] In some embodiments, the feature extraction network includes a convolutional layer and a second multi-layer perceptron layer, and the extraction module 140 is further used to:

[0078] Performing convolution processing on the target image through the convolution layer to obtain initial image features;

[0079] The initial image features are processed by the second multi-layer perceptron layer to obtain target image features.

[0080] In some embodiments, the text encoding network includes a BERT model, a BiLSTM model, and a linear layer, and the text encoding module 120 is further used to:

[0081] Encoding the target text through the BERT model to obtain a first text feature;

[0082] Performing context encoding on the first text feature through the BiLSTM model to obtain a second text feature;

[0083] The second text feature is processed by the linear layer to obtain a target text feature.

[0084] In some embodiments, the named entity recognition apparatus 100 further includes a model training module, wherein the model training module is used to:

[0085] Acquire a training sample set, wherein the training sample set includes a plurality of training samples, and the training samples include training text, training images matching the training text, and annotated entity sequences;

[0086] According to the training sample set, the initial entity recognition model is trained until convergence to obtain the target entity recognition model, the network structure of the initial entity recognition model and the target entity recognition model are the same, and the model parameters of the initial entity recognition model and the target entity recognition model are different.

[0087] In some embodiments, the model training module is further used to:

[0088] Selecting a training sample from the training sample set as a current training sample;

[0089] Encoding the training text in the current training sample through the text encoding network in the initial entity recognition model to obtain training text features;

[0090] Gridding the training text features through the multi-head attention network in the initial entity recognition model to obtain training class image features;

[0091] Extracting features of the training image in the current training sample through the feature extraction network in the initial entity recognition model to obtain training image features;

[0092] The training class image features and the training image features are fused through a feature fusion layer in the initial entity recognition model to obtain fused features;

[0093] Performing entity recognition on the fused features through the entity recognition layer in the initial entity recognition model to obtain a predicted entity sequence;

[0094] Calculating a cross entropy loss value according to the predicted entity sequence and the labeled entity sequence in the current training sample;

[0095] When the cross entropy loss value is greater than a preset loss value, the model parameters of the initial entity recognition model are updated, and the step of selecting a training sample from the training sample set as the current training sample is returned to be executed;

[0096] When the cross entropy loss value is less than or equal to a preset loss value, it is determined that the initial entity recognition model converges to obtain the target entity recognition model.

[0097] It should be noted that technical personnel in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working process of the above-described device and each module and unit can refer to the corresponding process in the aforementioned named entity recognition method embodiment, and will not be repeated here.

[0098] The apparatus provided in the above embodiment may be implemented in the form of a computer program. The computer program may be Figure 6 Runs on the computer device shown.

[0099] See also Figure 6 , Figure 6 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server or a terminal device.

[0100] like Figure 6 As shown, the computer device includes a processor, a memory, and a network interface connected via a system bus, wherein the memory may include a storage medium and an internal memory.

[0101] The storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any one of the named entity recognition methods.

[0102] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0103] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0104] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0105] In some embodiments, the computer device is deployed with a target entity recognition model, the target entity recognition model includes a text encoding network, a multi-head attention network, a feature extraction network, a feature fusion layer and an entity recognition layer, and the processor is used to run a computer program stored in the memory to implement the following steps:

[0106] Acquire a target text to be recognized and a target image corresponding to the target text;

[0107] The target text is encoded by the text encoding network to obtain target text features, and the target text features are gridded by the multi-head attention network to obtain target class image features;

[0108] Extracting features of the target image through the feature extraction network to obtain target image features, wherein the target class image features have the same expression form as the target image features;

[0109] The target class image features and the target image features are fused through the feature fusion layer, and entity recognition is performed on the fused features through the entity recognition layer to obtain an entity recognition result.

[0110] In some embodiments, the multi-head attention network includes a multi-head attention layer and a first multi-layer perceptron layer, and the processor, when implementing gridding processing of the target text features through the multi-head attention network to obtain target class image features, is used to implement:

[0111] Gridding the target text features through the multi-head attention layer to obtain initial image-like features;

[0112] The initial class image features are processed by the first multi-layer perceptron layer to obtain target class image features.

[0113] In some embodiments, the weight parameters of the multi-head attention layer include a Query weight matrix and a Key weight matrix, the bias parameters of the multi-head attention layer include a Query bias vector and a Key bias vector, and the processor, when implementing gridding processing of the target text features through the multi-head attention layer to obtain the initial image-like features, is used to implement:

[0114] Determine a Query vector matrix according to the Query weight matrix, the Query bias vector and the target text feature;

[0115] Determine a Key vector matrix according to the Key weight matrix, the Key bias vector and the target text feature;

[0116] Determine the transposed matrix of the Query vector matrix, and perform multiplication operation on the transposed matrix of the Query vector matrix and the Key vector matrix to obtain initial class image features.

[0117] In some embodiments, the feature extraction network includes a convolutional layer and a second multi-layer perceptron layer, and when the processor extracts features of the target image through the feature extraction network to obtain the target image features, it is used to implement:

[0118] Performing convolution processing on the target image through the convolution layer to obtain initial image features;

[0119] The initial image features are processed by the second multi-layer perceptron layer to obtain target image features.

[0120] In some embodiments, the text encoding network includes a BERT model, a BiLSTM model, and a linear layer. When the processor encodes the target text through the text encoding network to obtain the target text features, it is used to implement:

[0121] Encoding the target text through the BERT model to obtain a first text feature;

[0122] Performing context encoding on the first text feature through the BiLSTM model to obtain a second text feature;

[0123] The second text feature is processed by the linear layer to obtain a target text feature.

[0124] In some embodiments, the processor is further configured to implement:

[0125] Acquire a training sample set, wherein the training sample set includes a plurality of training samples, and the training samples include training text, training images matching the training text, and annotated entity sequences;

[0126] According to the training sample set, the initial entity recognition model is trained until convergence to obtain the target entity recognition model, the network structure of the initial entity recognition model and the target entity recognition model are the same, and the model parameters of the initial entity recognition model and the target entity recognition model are different.

[0127] In some embodiments, when the processor implements the step of training the initial entity recognition model to convergence according to the training sample set to obtain the target entity recognition model, the processor is used to implement:

[0128] Selecting a training sample from the training sample set as a current training sample;

[0129] Encoding the training text in the current training sample through the text encoding network in the initial entity recognition model to obtain training text features;

[0130] Gridding the training text features through the multi-head attention network in the initial entity recognition model to obtain training class image features;

[0131] Extracting features of the training image in the current training sample through the feature extraction network in the initial entity recognition model to obtain training image features;

[0132] The training class image features and the training image features are fused through a feature fusion layer in the initial entity recognition model to obtain fused features;

[0133] Performing entity recognition on the fused features through the entity recognition layer in the initial entity recognition model to obtain a predicted entity sequence;

[0134] Calculating a cross entropy loss value according to the predicted entity sequence and the labeled entity sequence in the current training sample;

[0135] When the cross entropy loss value is greater than a preset loss value, the model parameters of the initial entity recognition model are updated, and the step of selecting a training sample from the training sample set as the current training sample is returned to be executed;

[0136] When the cross entropy loss value is less than or equal to a preset loss value, it is determined that the initial entity recognition model converges to obtain the target entity recognition model.

[0137] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the computer device described above can refer to the corresponding process in the aforementioned named entity recognition method embodiment, and will not be repeated here.

[0138] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0139] An embodiment of the present application further provides a storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the various embodiments of the named entity recognition method of the present application.

[0140] The storage medium may be volatile or non-volatile. The storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0141] Furthermore, the storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0142] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0143] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0144] It should also be understood that the term "and / or" used in the specification of this application and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0145] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for named entity recognition, characterized in that: Applied to a computer device, the computer device is deployed with a target entity recognition model, the target entity recognition model includes a text encoding network, a multi-head attention network, a feature extraction network, a feature fusion layer and an entity recognition layer, the method includes: Acquire a target text to be recognized and a target image corresponding to the target text; The target text is encoded by the text encoding network to obtain target text features, and the target text features are gridded by the multi-head attention network to obtain target class image features; Extracting features of the target image through the feature extraction network to obtain target image features, wherein the target class image features have the same expression form as the target image features; The target class image features and the target image features are fused through the feature fusion layer, and entity recognition is performed on the fused features through the entity recognition layer to obtain an entity recognition result.

2. The method for named entity recognition according to claim 1, characterized in that: The multi-head attention network includes a multi-head attention layer and a first multi-layer perceptron layer, and the target text features are gridded by the multi-head attention network to obtain target class image features, including: Gridding the target text features through the multi-head attention layer to obtain initial image-like features; The initial class image features are processed by the first multi-layer perceptron layer to obtain target class image features.

3. The method for named entity recognition according to claim 2, characterized in that: The weight parameters of the multi-head attention layer include a Query weight matrix and a Key weight matrix, the bias parameters of the multi-head attention layer include a Query bias vector and a Key bias vector, and the gridding process of the target text features through the multi-head attention layer to obtain the initial image-like features includes: Determine a Query vector matrix according to the Query weight matrix, the Query bias vector and the target text feature; Determine a Key vector matrix according to the Key weight matrix, the Key bias vector and the target text feature; A transposed matrix of the Query vector matrix is ​​determined, and a multiplication operation is performed on the transposed matrix of the Query vector matrix and the Key vector matrix to obtain an initial class image feature.

4. The method for named entity recognition according to claim 1, characterized in that: The feature extraction network includes a convolutional layer and a second multi-layer perceptron layer. The feature extraction of the target image through the feature extraction network to obtain the target image features includes: Performing convolution processing on the target image through the convolution layer to obtain initial image features; The initial image features are processed by the second multi-layer perceptron layer to obtain target image features.

5. The method for named entity recognition according to claim 1, characterized in that: The text encoding network includes a BERT model, a BiLSTM model and a linear layer. The target text is encoded by the text encoding network to obtain target text features, including: Encoding the target text through the BERT model to obtain a first text feature; Performing context encoding on the first text feature through the BiLSTM model to obtain a second text feature; The second text feature is processed by the linear layer to obtain a target text feature.

6. The method for named entity recognition according to any one of claims 1 to 5, characterized in that: The method further comprises: Acquire a training sample set, wherein the training sample set includes a plurality of training samples, and the training samples include training text, training images matching the training text, and annotated entity sequences; According to the training sample set, the initial entity recognition model is trained until convergence to obtain the target entity recognition model, the network structure of the initial entity recognition model and the target entity recognition model are the same, and the model parameters of the initial entity recognition model and the target entity recognition model are different.

7. The method for named entity recognition according to claim 6, characterized in that: The step of training the initial entity recognition model to convergence according to the training sample set to obtain the target entity recognition model includes: Selecting a training sample from the training sample set as a current training sample; Encoding the training text in the current training sample through the text encoding network in the initial entity recognition model to obtain training text features; Gridding the training text features through the multi-head attention network in the initial entity recognition model to obtain training class image features; Extracting features of the training image in the current training sample through the feature extraction network in the initial entity recognition model to obtain training image features; The training class image features and the training image features are fused through a feature fusion layer in the initial entity recognition model to obtain fused features; Performing entity recognition on the fused features through the entity recognition layer in the initial entity recognition model to obtain a predicted entity sequence; Calculating a cross entropy loss value according to the predicted entity sequence and the labeled entity sequence in the current training sample; When the cross entropy loss value is greater than a preset loss value, the model parameters of the initial entity recognition model are updated, and the step of selecting a training sample from the training sample set as the current training sample is returned to be executed; When the cross entropy loss value is less than or equal to a preset loss value, it is determined that the initial entity recognition model converges to obtain the target entity recognition model.

8. A named entity recognition device, characterized in that: Applied to a computer device, the computer device is deployed with a target entity recognition model, the target entity recognition model includes a text encoding network, a multi-head attention network, a feature extraction network, a feature fusion layer and an entity recognition layer, and the named entity recognition device includes: An acquisition module, used to acquire a target text to be recognized and a target image corresponding to the target text; A text encoding module, used for encoding the target text through the text encoding network to obtain a target vector; A grid processing module, used for performing grid processing on the target vector through the multi-head attention network to obtain target class image features; An extraction module, used for performing feature extraction on the target image through the feature extraction network to obtain target image features, wherein the target class image features have the same expression form as the target image features; A feature fusion module, used for fusing the target class image features and the target image features through the feature fusion layer; The entity recognition module is used to perform entity recognition on the fused features through the entity recognition layer to obtain an entity recognition result.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the named entity recognition method according to any one of claims 1 to 7 are implemented.

10. A storage medium for computer-readable storage, characterized in that: The storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the named entity recognition method according to any one of claims 1 to 7 are implemented.

Images