Entity recognition method, device, equipment and storage medium

Abstract

The present disclosure relates to an entity recognition method, apparatus, device and storage medium. The present disclosure determines the association relationship between the multiple first units, the association relationship between the multiple second units, and the association relationship between the multiple first units and the multiple second units through feature information corresponding to the multiple first units in the first modality and feature information corresponding to the multiple second units in the second modality related to the first modality. Further, according to the importance levels corresponding to the multiple first units and the importance levels corresponding to the multiple second units, the association relationship between the multiple first units and the multiple second units is calibrated, so that the association relationship between the multiple first units and the multiple second units, that is, the association relationship between different modalities, is more accurate. Thereby, the entities in the text information can be accurately identified, that is, the accuracy of identifying named entities through multimodality is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of information technology, and in particular to an entity recognition method, device, equipment and storage medium. Background Art

[0002] Entities are an important part of natural language texts, such as named entities, which are very important for downstream tasks such as information extraction, entity linking, and question-answering systems. Therefore, they have attracted the attention of academia and industry.

[0003] The prior art proposes a multimodal named entity recognition technology, that is, identifying named entities in natural language texts given multimodal information.

[0004] However, the inventors of the present application have discovered that the multimodal named entity recognition technology in the prior art cannot accurately recognize the named entities. Summary of the invention

[0005] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides an entity recognition method, apparatus, device and storage medium to improve the accuracy of multimodal recognition of named entities.

[0006] In a first aspect, an embodiment of the present disclosure provides an entity recognition method, including:

[0007] Acquire a first modality and a second modality, wherein the first modality is related to the second modality, the first modality includes a plurality of first units, and the second modality includes a plurality of second units;

[0008] Determine, according to the feature information respectively corresponding to the plurality of first units and the feature information respectively corresponding to the plurality of second units, the association relationship between the plurality of first units, and the association relationship between the plurality of first units and the plurality of second units;

[0009] Calibrate the association relationship between the plurality of first units and the plurality of second units according to the importance levels respectively corresponding to the plurality of first units and the importance levels respectively corresponding to the plurality of second units;

[0010] According to the association relationship between the multiple first units, the association relationship between the multiple second units, and the calibrated association relationship between the multiple first units and the multiple second units, entities in text information are identified, and the text information is the first modality or the second modality.

[0011] In a second aspect, an embodiment of the present disclosure provides an entity recognition device, including:

[0012] An acquisition module, configured to acquire a first modality and a second modality, wherein the first modality is related to the second modality, the first modality includes a plurality of first units, and the second modality includes a plurality of second units;

[0013] a determination module, configured to determine, according to the feature information respectively corresponding to the plurality of first units and the feature information respectively corresponding to the plurality of second units, an association relationship between the plurality of second units, and an association relationship between the plurality of first units and the plurality of second units;

[0014] A calibration module, configured to calibrate the association relationship between the plurality of first units and the plurality of second units according to the importance levels respectively corresponding to the plurality of first units and the importance levels respectively corresponding to the plurality of second units;

[0015] An identification module is used to identify entities in text information based on the association relationship between the multiple first units, the association relationship between the multiple second units, and the calibrated association relationship between the multiple first units and the multiple second units, where the text information is the first modality or the second modality.

[0016] In a third aspect, an embodiment of the present disclosure provides an electronic device, including:

[0017] Memory;

[0018] Processor; and

[0019] Computer programs;

[0020] The computer program is stored in the memory and is configured to be executed by the processor to implement the method as described in the first aspect.

[0021] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the method described in the first aspect.

[0022] The entity recognition method, device, equipment and storage medium provided by the embodiments of the present disclosure determine the association relationship between the multiple first units, the association relationship between the multiple second units, and the association relationship between the multiple first units and the multiple second units through the feature information corresponding to the multiple first units in the first modality and the feature information corresponding to the multiple second units in the second modality related to the first modality. Further, according to the importance degree corresponding to the multiple first units and the importance degree corresponding to the multiple second units, the association relationship between the multiple first units and the multiple second units is calibrated, so that the association relationship between the multiple first units and the multiple second units, that is, the association relationship between different modalities, is more accurate. Therefore, according to the association relationship between the multiple first units, the association relationship between the multiple second units, and the calibrated association relationship between the multiple first units and the multiple second units, the entities in the text information can be accurately identified, that is, the accuracy of identifying named entities through multimodality is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0025] Figure 1 A flow chart of an entity recognition method provided by an embodiment of the present disclosure;

[0026] Figure 2 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0027] Figure 3 A schematic diagram of the structure of an entity recognition model provided in an embodiment of the present disclosure;

[0028] Figure 4 A flow chart of an entity recognition method provided by another embodiment of the present disclosure;

[0029] Figure 5 A flowchart of a multi-head calibration attention mechanism provided for another embodiment of the present disclosure;

[0030] Figure 6 A schematic diagram of calibrating T2I provided by another embodiment of the present disclosure;

[0031] Figure 7A schematic diagram of calibrating I2T provided by another embodiment of the present disclosure;

[0032] Figure 8 A schematic diagram of the structure of an entity identification device provided in an embodiment of the present disclosure;

[0033] Fig. 9 A schematic diagram of the structure of an electronic device embodiment provided by the present disclosure. DETAILED DESCRIPTION

[0034] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0035] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0036] As network information continues to become richer, the analysis and identification of network information has become increasingly important. For example, users or businesses may publish different types of information on the Internet, such as text information, image information, video information, voice information, etc. These different types of information are generally referred to as multimodality. Broadly speaking, each source or form of information is a modality. In the embodiments of the present disclosure, two modalities are involved: text information and image information.

[0037] In some cases, the terminal or server can process the information published by the user or the merchant to identify the core content of the information. For example, the user or the merchant may publish the relevant information of a certain product on the network, and the terminal or the server can identify the core attributes of the product (goods), the core needs of the user (people), the supply and demand situation of the market (field), etc. from the relevant information. Among them, the core content usually includes named entities (Named Entity). Named entities are specific things referred to by a name, such as the name of a person, the name of a place, the name of an organization, etc. Therefore, identifying named entities from text information is a relatively important task. In this context, multimodal named entity recognition technology came into being. Among them, named entity recognition (Named Entity Recognition) refers to a given text, further, identifying the named entities therein, and predicting the category of each named entity, which often comes from a predefined category set, for example, the category set contains categories such as people, organizations, and places. Multimodal Named Entity Recognition refers to the process of identifying named entities in a given text and pictures related to the text, and predicting the category of each named entity. In addition, multimodal named entity recognition is an important foundation for many fields, such as reading comprehension, question-answering systems, and text comprehension tasks, as well as knowledge graph construction.

[0038] Although the current multimodal named entity recognition technology can identify named entities in text information, for example, it can extract core types (for example, product attributes, characters, places, selling points, etc.) from complex and diverse user-generated content (UGC). However, the current multimodal named entity recognition technology is not accurate enough in the recognition process of named entities. To address this problem, the embodiments of the present disclosure provide an entity recognition method, which is introduced below in conjunction with specific embodiments.

[0039] Figure 1 The flowchart of the entity recognition method provided by the embodiment of the present disclosure. The method can be executed by an entity recognition device, which can be implemented in software and / or hardware, and can be configured in an electronic device, such as a server or a terminal, wherein the terminal specifically includes a mobile phone, a computer or a tablet computer, etc. The method can be applied to Figure 2The application scenario shown includes a terminal 21 and a server 22. The terminal 21 can send text information and image information related to the text information to the server 22, so that the server 22 can identify the named entities in the text information according to the two modalities of text information and image information. Alternatively, the terminal 21 can identify the named entities in the text information according to the two modalities of text information and image information. In other words, the entity recognition method described in the embodiment of the present disclosure can be executed by the terminal or the server, and the server is used as an example for schematic description below.

[0040] like Figure 1 As shown, the specific steps of this method are as follows:

[0041] S101. Acquire a first modality and a second modality, where the first modality is related to the second modality, the first modality includes a plurality of first units, and the second modality includes a plurality of second units.

[0042] like Figure 2 As shown, the server 22 may obtain a first modality and a second modality, wherein the first modality and the second modality are related, and the first modality includes a plurality of first units, and the second modality includes a plurality of second units.

[0043] In the embodiments of the present disclosure, the first modality and the second modality may be two different modalities, and one of the first modality and the second modality may be text information. For example, when the first modality is text information, the second modality may be a modality other than text information, such as a voice modality, a video modality, etc. When the second modality is text information, the first modality may be a modality other than text information, such as a voice modality, a video modality, etc.

[0044] In the following, it is taken as an example that the first modality is text information and the second modality is image information. Accordingly, the first unit is a text unit and the second unit is an image area.

[0045] like Figure 2As shown, the terminal 21 can send text information and image information related to the text information to the server 22, so that the server 22 can obtain the text information and image information. It is understandable that the way in which the server 22 obtains the text information and image information is not limited to this method. For example, the server 22 can also obtain the text information and image information from other network devices, or the server 22 can obtain the text information and image information from a local or corresponding database of the server 22. It is understandable that the text information can be a piece of text, and the text includes multiple text units. The text unit is not specifically limited in the embodiment of the present disclosure. For example, a text unit can be a phrase (for example, a word, a phrase, etc.), or a text unit can be a character (for example, a Chinese character, a letter, etc.). This embodiment is schematically explained by taking a text unit as an example of a phrase. For example, the server 22 can identify multiple text units in the text information. In addition, the server 22 can also divide the image information corresponding to the text information into multiple image areas, each image area is a block. The specific number of image regions formed after segmentation is not limited here. For example, in this embodiment, the image information can be segmented into three image regions.

[0046] S102. Determine association relationships between the multiple first units, association relationships between the multiple second units, and association relationships between the multiple first units and the multiple second units based on feature information corresponding to the multiple first units and feature information corresponding to the multiple second units.

[0047] For example, after identifying multiple text units in the text information and dividing the image information into multiple image regions, the server 22 can map each of the multiple text units into a representation vector (embedding), and the representation vector can be used as feature information, that is, one text unit can correspond to a representation vector, i.e., feature information. The process of mapping each text unit in the text information into a representation vector can be a process of feature extraction of the text information, and after feature extraction, the representation vectors corresponding to the multiple text units are obtained.

[0048] In addition, the server 22 can also perform feature extraction on the image information, so as to obtain representation vectors corresponding to the multiple image areas in the image information, that is, one image area can correspond to one representation vector, i.e., feature information. Further, the server 22 can determine the association relationship between the multiple text units, the association relationship between the multiple image areas, and the association relationship between the multiple text units and the multiple image areas based on the feature information corresponding to the multiple text units and the feature information corresponding to the multiple image areas. For example, the server 22 can determine these association relationships through an attention mechanism. Specifically, the feature information corresponding to the multiple text units and the feature information corresponding to the multiple image areas are used as inputs to the attention mechanism, and the attention mechanism can output these association relationships. In this embodiment, the association relationship can also be recorded as correlation.

[0049] S103: Calibrate the association relationship between the plurality of first units and the plurality of second units according to the importance levels respectively corresponding to the plurality of first units and the importance levels respectively corresponding to the plurality of second units.

[0050] For example, the server 22 may expand the entity label words (Entity Label Words) in advance through a knowledge spectrum or other means to obtain expanded entity label words (Expanded Entity Label Words). Further, the importance of the multiple text units and the importance of the multiple image areas can be determined based on the expanded entity label words. Among them, the importance of the multiple text units can be the importance of each of the multiple text units for the entity recognition task. The importance of the multiple image areas can be the importance of each of the multiple image areas for the entity recognition task. It can be understood that the method for determining the importance of the multiple text units and the importance of the multiple image areas is not limited to the expanded entity label words described in this embodiment. In other embodiments, the importance of the multiple text units and the importance of the multiple image areas can be determined by other methods. Further, the server 22 can calibrate the association between the multiple text units and the multiple image areas according to the importance of the multiple text units and the importance of the multiple image areas.

[0051] S104. Identify entities in text information based on the association relationship between the multiple first units, the association relationship between the multiple second units, and the calibrated association relationship between the multiple first units and the multiple second units, where the text information is the first modality or the second modality.

[0052] For example, after the server 22 calibrates the association relationship between the multiple text units and the multiple image areas, it can identify entities in the text information, for example, identify named entities in the text information, based on the association relationship between the multiple text units, the association relationship between the multiple image areas, and the calibrated association relationship between the multiple text units and the multiple image areas.

[0053] The disclosed embodiment determines the association relationship between the multiple first units, the association relationship between the multiple second units, and the association relationship between the multiple first units and the multiple second units through the feature information corresponding to the multiple first units in the first modality and the feature information corresponding to the multiple second units in the second modality related to the first modality. Further, according to the importance degree corresponding to the multiple first units and the importance degree corresponding to the multiple second units, the association relationship between the multiple first units and the multiple second units is calibrated, so that the association relationship between the multiple first units and the multiple second units, that is, the association relationship between different modalities, is more accurate. Therefore, according to the association relationship between the multiple first units, the association relationship between the multiple second units, and the calibrated association relationship between the multiple first units and the multiple second units, the entities in the text information can be accurately identified, that is, the accuracy of identifying named entities through multimodality is improved.

[0054] Based on the above embodiment, an entity recognition model can be deployed in the server 22, and the entity recognition model can execute the entity recognition method described above. It can be understood that if the terminal executes the entity recognition method, the entity recognition model can also be deployed on the terminal. Figure 3 The figure shows a schematic diagram of the structure of the entity recognition model. The entity recognition method described above is introduced in detail below in conjunction with the structure of the entity recognition model.

[0055] like Figure 3As shown, the entity recognition model includes a first converter, a second converter and an entity prediction module (Span-based Prediction), wherein the first converter can be specifically a C-Transformer, and the second converter can be an R-Transformer. The function of C-Transformer is to process external knowledge, and the external knowledge can be the expanded entity category words as described above. R-Transformer is used to process feature information corresponding to multiple text units and feature information corresponding to multiple image regions. The entity prediction module is used to predict entities in text information, such as named entities. Among them, C-Transformer and R-Transformer respectively include N parts with the same structure, and the N parts are connected in sequence, and each part includes the following Figure 4 The 4 layers shown. For example, each part of C-Transformer includes a multi-head feed forward layer (Multi-Head Feed Forward), Add&Norm, a feed forward layer (Feed Forward), and Add&Norm. Among them, Add&Norm is a layer that first does residual (residua) and then does layer normalization (Layer Norm). Each part of R-Transformer includes a multi-head calibrated attention mechanism (Multi-Head Refined Attention), Add&Norm, a feed forward layer (Feed Forward), and Add&Norm. In addition, the layers of the same level and the same name in R-Transformer and C-Transformer can share parameters (Share Weights). For example, the Add&Norm of the same level in R-Transformer and C-Transformer can share parameters, and the feed forward layers of the same level in R-Transformer and C-Transformer can share parameters.

[0056] like Figure 3 As shown in the figure, the two parts with the same level in R-Transformer and C-Transformer are Figure 3 The bold and black line shown is connected. Similarly, other two parts of the same level in R-Transformer and C-Transformer that are not shown can also be connected in a similar way.

[0057] like Figure 3 As shown, after acquiring the text information and the image information related to the text information, the server 22 may further perform feature extraction on the text information and the image information.

[0058] Specifically, when the server 22 performs feature extraction on the text information, all text units in the text information can be identified. For example, the text information includes 5 text units. The server 22 can use the embedding layer of the pre-trained language model to map the 5 text units in the text information into a series of continuous representation vectors. For example, one text unit can correspond to one representation vector. Figure 3 The 33 shown indicates the representation vectors corresponding to the five text units.

[0059] When the server 22 extracts features from the image information, the image information may be divided into multiple image regions, for example, into three image regions. Further, the server 22 may use an image transformer model to obtain representation vectors corresponding to the three image regions in the image information, for example, one image region may correspond to one representation vector. Figure 3 The 32 shown indicates the representation vectors corresponding to the three image regions respectively.

[0060] Figure 4 This is a flow chart of an entity recognition method provided by another embodiment of the present disclosure. Figure 4 As shown, the method includes the following steps:

[0061] S401: Acquire a first modality and a second modality, where the first modality is related to the second modality, the first modality includes a plurality of first units, and the second modality includes a plurality of second units.

[0062] Specifically, the implementation methods and specific principles of S401 and S101 are consistent, and are not repeated here. For example, the first modality is text information, and the second modality is image information. The first unit is a text unit, and the second unit is an image area.

[0063] S402. Determine, based on the feature information respectively corresponding to the multiple first units and the feature information respectively corresponding to the multiple second units, the association relationship between the multiple second units, and the association relationship between the multiple first units and the multiple second units.

[0064] like Figure 3 As shown, 30 represents the expanded entity classifier, and there may be multiple expanded entity classifiers. Further, the representation vector of each entity classifier in the expanded entity classifier can be obtained, that is, one entity classifier can correspond to one representation vector. Taking four entity classifiers as an example, Figure 3 The 31 shown represents the representation vectors corresponding to the four entity category words respectively.

[0065] In addition, if Figure 3The input of the multi-head calibrated attention mechanism shown includes the representation vectors corresponding to the four entity category words, the representation vectors corresponding to the three image regions, and the representation vectors corresponding to the five text units. Furthermore, the multi-head calibrated attention mechanism can calculate the association between the five text units, the association between the three image regions, and the association between the five text units and the three image regions based on the representation vectors corresponding to the three image regions and the representation vectors corresponding to the five text units. Figure 5 Shown is the calculation process of the multi-head calibration attention mechanism.

[0066] like Figure 5 As shown, the three image regions correspond to the representation vectors (e.g. Figure 5 32) and the five text units correspond to the representation vectors (e.g. Figure 5 33) shown in FIG. 33 can be used as the input of the attention mechanism, which can output the association relationship between the 5 text units, the association relationship between the 3 image regions, and the association relationship between the 5 text units and the 3 image regions. Among them, the association relationship between the 5 text units can be the association relationship between any two text units among the 5 text units, and the any two text units can be the same or different. Specifically, the association relationship between the any two text units is calculated based on the representation vectors corresponding to the any two text units. Therefore, the association relationship between the 5 text units can be a 5*5 matrix, for example Figure 5 As shown in 51, in addition, the association relationship between the five text units can be recorded as T2T, wherein T represents text information (Text).

[0067] Similarly, the association relationship between the three image regions can be a 3*3 matrix, for example Figure 5 53 shown. In addition, the association relationship between the three image regions can be expressed as I2I, where I represents image information (Image).

[0068] The association relationship between the five text units and the three image regions includes an association relationship T2I between each of the five text units and each of the three image regions, and an association relationship I2T between each of the three image regions and each of the five text units. T2I can be as follows: Figure 5 The 3*5 matrix 52 shown. I2T can be as follows Figure 5 A 5*3 matrix 54 is shown.

[0069] It can be understood that T2T and I2I are the associations within the same modality, while T2I and I2T are the associations between different modalities. There may be gaps (GAP) between different modalities. Therefore, in order to eliminate the gaps, in this embodiment, T2I and I2T can be calibrated according to the representation vector of each entity classifier in the expanded entity classifier. Figure 5 The 55 shown here represents the calibrated T2I. Figure 5 56 shown in FIG. 56 represents the calibrated I2T. The process of calibrating T2I is as follows: Figure 6 As shown in Figure 2, the process of calibrating I2T is as follows: Figure 7 shown.

[0070] S403: Determine an association relationship between the one or more entity classifiers and the multiple first units according to feature information respectively corresponding to the preset one or more entity classifiers and feature information respectively corresponding to the multiple first units.

[0071] For example, the server 22 may expand the entity label words (Entity Label Words) in advance through a knowledge spectrum or other means, and the expanded entity label words (Expanded Entity Label Words) may be used as one or more preset entity label words described in this embodiment. Alternatively, the server 22 may obtain the preset one or more entity label words from other network devices or terminals, wherein each entity label word may correspond to a representation vector, which may be used as feature information, that is, an entity label word may correspond to a representation vector, i.e., feature information. Taking four entity label words as an example, Figure 6 31 shows the representation vectors corresponding to the four entity category words, and 33 shows the representation vectors corresponding to the five text units. Figure 6 As shown, the representation vectors corresponding to the four entity category words and the representation vectors corresponding to the five text units can be used as inputs of the attention mechanism, and the attention mechanism can output the association relationship between the four entity category words and the five text units. The association relationship can be a 4*5 matrix, such as Figure 6 61 shown. In some other embodiments, the representation vectors of multiple text units can also be recorded as multiple text features (Textual Features), the representation vectors of multiple entity category words can also be recorded as multiple category features (Label Features), and the association between the four entity category words and the five text units can also be recorded as the association between multiple entity category words and multiple text features (Correlation of Label Words and Textual Features).

[0072] S404: Determine importance levels corresponding to the multiple first units respectively according to the association relationship between the one or more entity classifiers and the multiple first units.

[0073] like Figure 6 As shown in FIG. 1 , after calculating the association between the four entity classifiers and the five text units, the importance of the five text units can be determined based on the association. The importance of the five text units can be the importance of each of the five text units to the entity recognition task (specifically, the named entity recognition task). The importance can be understood as the contribution rate score of the text information (Saliency Score of T). Specifically, the importance of the five text units can be recorded as s T .

[0074] Optionally, according to the association relationship between the one or more entity category words and the multiple first units, the importance levels respectively corresponding to the multiple first units are determined, including: performing a pooling operation on the association relationship between the one or more entity category words and the multiple first units to obtain the importance levels respectively corresponding to the multiple first units.

[0075] like Figure 6 As shown, the association relationship between the four entity category words and the five text units can be input into a pooling layer (Avg-Pool), and the pooling layer can perform a pooling operation on the association relationship between the four entity category words and the five text units, so as to obtain the importance levels corresponding to the five text units respectively. The importance levels corresponding to the five text units respectively can be a 1*5 matrix, for example Figure 6 62 shown.

[0076] S405: Determine the association relationship between the one or more entity classifiers and the multiple second units according to the feature information respectively corresponding to the preset one or more entity classifiers and the feature information respectively corresponding to the multiple second units.

[0077] Figure 7 31 shows the representation vectors corresponding to the four entity category words, and 32 shows the representation vectors corresponding to the three image regions. Figure 6 As shown, the representation vectors corresponding to the four entity category words and the representation vectors corresponding to the three image regions can be used as inputs of the attention mechanism, and the attention mechanism can output the association relationship between the four entity category words and the three image regions. The association relationship can be a 4*3 matrix, such as Figure 771. In some other embodiments, the representation vectors of the multiple image regions can also be recorded as visual features (Visual Features), and the association between the four entity category words and the three image regions can be recorded as the association between multiple entity category words and multiple visual features (Correlation of Label Words and Visual Features).

[0078] S406: Determine importance levels corresponding to the multiple second units respectively according to the association relationship between the one or more entity classifiers and the multiple second units.

[0079] like Figure 7 As shown in FIG. 1 , after calculating the association relationship between the four entity category words and the three image regions, the importance of the three image regions can be determined according to the association relationship. The importance of the three image regions can be the importance of each of the three image regions for the entity recognition task (specifically, the named entity recognition task). The importance can be understood as the contribution rate score of the image information (Saliency Score of I). Specifically, the importance of the three image regions can be recorded as s I .

[0080] Optionally, according to the association relationship between the one or more entity category words and the multiple second units, the importance levels respectively corresponding to the multiple second units are determined, including: performing a pooling operation on the association relationship between the one or more entity category words and the multiple second units to obtain the importance levels respectively corresponding to the multiple second units.

[0081] like Figure 7 As shown, the association relationship between the four entity category words and the three image regions can be input into a pooling layer (Avg-Pool), and the pooling layer can perform a pooling operation on the association relationship between the four entity category words and the three image regions, so as to obtain the importance levels corresponding to the three image regions respectively. The importance levels corresponding to the three image regions respectively can be a 1*3 matrix, for example Figure 7 72 shown.

[0082] S407: Calibrate the association relationship between the plurality of first units and the plurality of second units according to the importance levels respectively corresponding to the plurality of first units and the importance levels respectively corresponding to the plurality of second units.

[0083] like Figure 6 and Figure 7As shown, the association relationship between the five text units and the three image regions can be calibrated according to the importance levels corresponding to the five text units and the importance levels corresponding to the three image regions.

[0084] Optionally, the association relationship between the multiple first units and the multiple second units is calibrated according to the importance levels respectively corresponding to the multiple first units and the importance levels respectively corresponding to the multiple second units, including: calibrating the association relationship between each first unit in the multiple first units and each second unit in the multiple second units according to the importance levels respectively corresponding to the multiple first units; calibrating the association relationship between each second unit in the multiple second units and each first unit in the multiple first units according to the importance levels respectively corresponding to the multiple second units.

[0085] like Figure 6 As shown, T2I can be calibrated according to the importance levels corresponding to the five text units. In this embodiment, calibration can also be called correction. Figure 6 The reference numeral 52 indicates T2I, and the reference numeral 55 indicates T2I after calibration. The meaning of T2I here is the same as that of T2I in the above embodiment.

[0086] like Figure 7 As shown, I2T can be calibrated according to the importance of the three image regions, as shown in Figure 7 The reference numeral 54 indicates I2T, and the reference numeral 56 indicates the calibrated I2T. The meaning of I2T here is the same as that of I2T in the above embodiment.

[0087] Optionally, according to the respective importance levels corresponding to the multiple first units, the association relationship between each first unit in the multiple first units and each second unit in the multiple second units is calibrated, including: determining a first weight coefficient according to the feature information respectively corresponding to the multiple first units; according to the first weight coefficient and the respective importance levels corresponding to the multiple first units, calibrating the association relationship between each first unit in the multiple first units and each second unit in the multiple second units, wherein the first weight coefficient is used to control the calibration level.

[0088] like Figure 6 As shown in FIG. 1 , in the process of calibrating T2I according to the importance levels corresponding to the five text units, the representation vectors corresponding to the five text units can be input into the Feed Forward Network (FFN), and then processed by the hyperbolic tangent (Tanh) to obtain the first weight coefficient, which is recorded as gT Further, according to the first weight coefficient g T The importance levels corresponding to the five text units are used to calibrate T2I. During the calibration process, the importance levels corresponding to the five text units can be applied to T2I through a gate mechanism (Gate), and the degree of application is determined by the first weight coefficient g T Therefore, the first weight coefficient g T Used to control the degree of calibration. Figure 6 The 3*5 matrix shown in 52 may also be referred to as the original T2I attention mechanism map (Original T2I Attention Map) in some embodiments. The original T2I attention mechanism map may be denoted as A T2I .like Figure 6 The 3*5 matrix shown in 55 may also be referred to as a calibrated T2I attention mechanism map (Refined T2I Attention Map) in some embodiments.

[0089] Optionally, according to the respective importance levels corresponding to the multiple second units, the association relationship between each of the multiple second units and each of the multiple first units is calibrated, including: determining a second weight coefficient according to the feature information respectively corresponding to the multiple second units; according to the second weight coefficient and the respective importance levels corresponding to the multiple second units, calibrating the association relationship between each of the multiple second units and each of the multiple first units according to the second weight coefficient, and the second weight coefficient is used to control the calibration level.

[0090] like Figure 7 As shown in FIG. 1 , in the process of calibrating I2T according to the importance of the three image regions, the representation vectors corresponding to the three image regions can be input into the Feed Forward Network (FFN), and then processed by the hyperbolic tangent (Tanh) to obtain the second weight coefficient, which can be recorded as g I Further, according to the second weight coefficient g I The importance levels corresponding to the three image regions are used to calibrate I2T. During the calibration process, the importance levels corresponding to the three image regions can be applied to I2T through a gate mechanism (Gate), and the degree of application is determined by the second weight coefficient g. I Therefore, the second weight coefficient g I Used to control the degree of calibration. Figure 7The 5*3 matrix shown in 54 may also be referred to as the original I2T attention mechanism map (Original I2T Attention Map) in some embodiments. The original I2T attention mechanism map may be denoted as A I2T .like Figure 7 The 5*3 matrix shown in 56 may also be referred to as a calibrated I2T attention mechanism map (Refined I2T Attention Map) in some embodiments.

[0091] S408. Identify entities in text information based on the association relationship between the multiple first units, the association relationship between the multiple second units, and the calibrated association relationship between the multiple first units and the multiple second units, where the text information is the first modality or the second modality.

[0092] like Figure 5 The two calibration processes shown are respectively Figure 6 and Figure 7 Introduced so that Figure 5 The T2I (e.g., the matrix shown in 52) can be calibrated to the matrix shown in 55, and the I2T (e.g., the matrix shown in 54) can be calibrated to the matrix shown in 56. Further, the T2T (e.g., the matrix shown in 51), the calibrated T2I (e.g., the matrix shown in 55), the I2I (e.g., the matrix shown in 53), and the calibrated I2T (e.g., the matrix shown in 56) can be integrated. The integrated result can be obtained by Figure 5 The output layer (Output Layer) in the multi-head calibration attention mechanism shown in FIG. is transmitted to the previous layer of the multi-head calibration attention mechanism, for example, the previous layer is Add&Norm. After processing layer by layer through the layers above Add&Norm, the processing results can be transmitted to Figure 3 The entity prediction module shown can identify entities in text information based on the processing results.

[0093] Optionally, identifying entities in text information includes: identifying one or more character strings in the text information; predicting whether each character string is an entity; and if the character string is an entity, identifying the type of the entity.

[0094] For example, the entity prediction module can predict the starting position and the ending position of each entity included in the text information, and predict the type of the entity. For example, the entity prediction module can enumerate all possible character strings in the text information, and then extract the characters at the starting position, the characters between the starting position and the ending position, and the characters at the ending position from the text information according to the starting position and the ending position of each character string, and splice the characters at the starting position, the characters between the starting position and the ending position, and the characters at the ending position to obtain the character string. Further, a linear classifier is used to predict whether the character string is an entity. If the character string is an entity, the type of the entity is further predicted. The type of the entity can be selected from several preset types, and the preset types can include human entities, organizational entities, institutional entities, other entities, etc. It can be understood that in some other embodiments, whether the character string is an entity can also be used as a type of the character string. In this case, the character string can be input into the linear classifier, and the linear classifier can predict the type of the character string, for example, whether it is an entity, a human entity, an organizational entity, an institutional entity, other entities, etc.

[0095] This embodiment calibrates the associations between different modalities through external knowledge, making the associations between different modalities more accurate. Since cross-modal associations are complex, implicit, and not easy to find, if the associations between different modalities are not calibrated, some important but implicit associations between different modalities may be ignored or lost. Therefore, this embodiment introduces external knowledge to evaluate the importance of each modality in the multi-modality for the entity recognition task, thereby increasing the proportion of more important modalities in the attention mechanism, correcting the associations between different modalities, and avoiding important but implicit associations between different modalities from being ignored or lost. Ultimately, the accuracy of entity recognition is improved.

[0096] In addition, another application scenario of the entity recognition method described in this embodiment may be that when a user or a merchant publishes relevant information of a product on the Internet, entities are identified from the relevant information through the entity recognition method described in this embodiment. These entities can constitute the core attributes of the product, thereby identifying the core attributes of the product in relatively lengthy user-generated content and associating the core attributes with the published product.

[0097] Figure 8 The schematic diagram of the structure of the entity recognition device provided in the embodiment of the present disclosure. The entity recognition device provided in the embodiment of the present disclosure can execute the processing flow provided in the embodiment of the entity recognition method, such as Figure 8 As shown, the entity identification device 80 includes:

[0098] An acquisition module 81, configured to acquire a first modality and a second modality, wherein the first modality is related to the second modality, the first modality includes a plurality of first units, and the second modality includes a plurality of second units;

[0099] A first determining module 82, configured to determine, according to the feature information respectively corresponding to the plurality of first units and the feature information respectively corresponding to the plurality of second units, the association relationship between the plurality of second units, and the association relationship between the plurality of first units and the plurality of second units;

[0100] A calibration module 83, configured to calibrate the association relationship between the plurality of first units and the plurality of second units according to the importance levels respectively corresponding to the plurality of first units and the importance levels respectively corresponding to the plurality of second units;

[0101] The identification module 84 is used to identify entities in the text information based on the association relationship between the multiple first units, the association relationship between the multiple second units, and the calibrated association relationship between the multiple first units and the multiple second units, where the text information is the first modality or the second modality.

[0102] Optionally, the entity identification device 80 also includes a second determination module 85, which is used to determine the association relationship between the one or more entity classifiers and the multiple first units according to the feature information corresponding to the preset one or more entity classifiers and the feature information corresponding to the multiple first units before the calibration module 83 calibrates the association relationship between the multiple first units and the multiple second units according to the importance degrees corresponding to the multiple first units and the importance degrees corresponding to the multiple second units; determine the importance degrees corresponding to the multiple first units according to the association relationship between the one or more entity classifiers and the multiple first units; determine the association relationship between the one or more entity classifiers and the multiple second units according to the feature information corresponding to the preset one or more entity classifiers and the feature information corresponding to the multiple second units; and determine the importance degrees corresponding to the multiple second units according to the association relationship between the one or more entity classifiers and the multiple second units.

[0103] Optionally, when the calibration module 83 calibrates the association relationship between the plurality of first units and the plurality of second units according to the importance degrees respectively corresponding to the plurality of first units and the importance degrees respectively corresponding to the plurality of second units, it is specifically used to:

[0104] Calibrate the association relationship between each first unit in the plurality of first units and each second unit in the plurality of second units according to the importance levels respectively corresponding to the plurality of first units;

[0105] According to the importance levels respectively corresponding to the plurality of second units, the association relationship between each second unit in the plurality of second units and each first unit in the plurality of first units is calibrated.

[0106] Optionally, when the calibration module 83 calibrates the association relationship between each first unit in the plurality of first units and each second unit in the plurality of second units according to the importance levels respectively corresponding to the plurality of first units, it is specifically used to:

[0107] Determining a first weight coefficient according to the feature information respectively corresponding to the plurality of first units;

[0108] According to the first weight coefficient and the respective corresponding importance levels of the multiple first units, the association relationship between each first unit in the multiple first units and each second unit in the multiple second units is calibrated, and the first weight coefficient is used to control the calibration degree.

[0109] Optionally, when the calibration module 83 calibrates the association relationship between each second unit in the plurality of second units and each first unit in the plurality of first units according to the importance levels respectively corresponding to the plurality of second units, it is specifically used to:

[0110] Determining a second weight coefficient according to the feature information respectively corresponding to the plurality of second units;

[0111] According to the second weight coefficient and the respective corresponding importance levels of the multiple second units, the association relationship between each second unit in the multiple second units and each first unit in the multiple first units is calibrated, and the second weight coefficient is used to control the calibration degree.

[0112] Optionally, when the second determination module 85 determines the importance levels corresponding to the multiple first units respectively according to the association relationship between the one or more entity classifiers and the multiple first units, it is specifically used to: perform a pooling operation on the association relationship between the one or more entity classifiers and the multiple first units to obtain the importance levels corresponding to the multiple first units respectively; when the second determination module 85 determines the importance levels corresponding to the multiple second units respectively according to the association relationship between the one or more entity classifiers and the multiple second units, it is specifically used to: perform a pooling operation on the association relationship between the one or more entity classifiers and the multiple second units to obtain the importance levels corresponding to the multiple second units respectively.

[0113] Optionally, when the recognition module 84 recognizes an entity in the text information, it is specifically used to:

[0114] Identifying one or more character strings in the text information;

[0115] Predict whether each string is an entity;

[0116] If the string is an entity, the type of the entity is identified.

[0117] Optionally, the first modality is text information, and the second modality is image information; the first unit is a text unit, and the second unit is an image area.

[0118] Figure 8 The entity recognition device of the illustrated embodiment can be used to execute the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0119] The internal functions and structure of the entity identification device are described above. The device can be implemented as an electronic device. Fig. 9 This is a schematic diagram of the structure of an electronic device embodiment provided by the present disclosure. Fig. 9 As shown, the electronic device includes a memory 91 and a processor 92 .

[0120] The memory 91 is used to store programs. In addition to the above-mentioned programs, the memory 91 can also be configured to store various other data to support operations on the electronic device. Examples of these data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0121] The memory 91 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0122] The processor 92 is coupled to the memory 91 and executes the program stored in the memory 91 to:

[0123] Acquire a first modality and a second modality, wherein the first modality is related to the second modality, the first modality includes a plurality of first units, and the second modality includes a plurality of second units;

[0124] Determine, according to the feature information respectively corresponding to the plurality of first units and the feature information respectively corresponding to the plurality of second units, the association relationship between the plurality of first units, and the association relationship between the plurality of first units and the plurality of second units;

[0125] Calibrate the association relationship between the plurality of first units and the plurality of second units according to the importance levels respectively corresponding to the plurality of first units and the importance levels respectively corresponding to the plurality of second units;

[0126] According to the association relationship between the multiple first units, the association relationship between the multiple second units, and the calibrated association relationship between the multiple first units and the multiple second units, entities in text information are identified, and the text information is the first modality or the second modality.

[0127] Further, if Fig. 9 As shown, the electronic device may also include: a communication component 93, a power component 94, an audio component 95, a display 96 and other components. Fig. 9 Only some components are shown schematically, which does not mean that the electronic device only includes Fig. 9 Components shown.

[0128] The communication component 93 is configured to facilitate wired or wireless communication between the electronic device and other devices. The electronic device can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 93 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 93 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0129] The power supply component 94 provides power to various components of the electronic device. The power supply component 94 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.

[0130] The audio component 95 is configured to output and / or input audio signals. For example, the audio component 95 includes a microphone (MIC), and when the electronic device is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 91 or sent via the communication component 93. In some embodiments, the audio component 95 also includes a speaker for outputting audio signals.

[0131] The display 96 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0132] In addition, an embodiment of the present disclosure further provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the entity recognition method described in the above embodiment.

[0133] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0134] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for entity recognition, wherein: The method comprises: Acquire a first modality and a second modality, wherein the first modality is related to the second modality, the first modality includes a plurality of first units, and the second modality includes a plurality of second units; Determine, according to the feature information respectively corresponding to the plurality of first units and the feature information respectively corresponding to the plurality of second units, the association relationship between the plurality of first units, and the association relationship between the plurality of first units and the plurality of second units; Calibrate the association relationship between the plurality of first units and the plurality of second units according to the importance levels respectively corresponding to the plurality of first units and the importance levels respectively corresponding to the plurality of second units; According to the association relationship between the multiple first units, the association relationship between the multiple second units, and the calibrated association relationship between the multiple first units and the multiple second units, entities in text information are identified, and the text information is the first modality or the second modality.

2. The method according to claim 1, wherein: Before calibrating the association relationship between the plurality of first units and the plurality of second units according to the importance levels respectively corresponding to the plurality of first units and the importance levels respectively corresponding to the plurality of second units, the method further includes: Determine, according to the feature information respectively corresponding to the one or more preset entity classifiers and the feature information respectively corresponding to the multiple first units, the association relationship between the one or more entity classifiers and the multiple first units; Determining the importance levels respectively corresponding to the multiple first units according to the association relationship between the one or more entity classifiers and the multiple first units; Determine the association relationship between the one or more entity classifiers and the multiple second units according to the feature information respectively corresponding to the preset one or more entity classifiers and the feature information respectively corresponding to the multiple second units; According to the association relationship between the one or more entity classifiers and the plurality of second units, the importance levels respectively corresponding to the plurality of second units are determined.

3. The method according to claim 1, wherein: Calibrate the association relationship between the plurality of first units and the plurality of second units according to the importance levels respectively corresponding to the plurality of first units and the importance levels respectively corresponding to the plurality of second units, including: Calibrate the association relationship between each first unit in the plurality of first units and each second unit in the plurality of second units according to the importance levels respectively corresponding to the plurality of first units; According to the importance levels respectively corresponding to the plurality of second units, the association relationship between each second unit in the plurality of second units and each first unit in the plurality of first units is calibrated.

4. The method according to claim 3, wherein: According to the importance levels respectively corresponding to the plurality of first units, calibrating the association relationship between each first unit in the plurality of first units and each second unit in the plurality of second units respectively includes: Determining a first weight coefficient according to the feature information respectively corresponding to the plurality of first units; According to the first weight coefficient and the respective corresponding importance levels of the multiple first units, the association relationship between each first unit in the multiple first units and each second unit in the multiple second units is calibrated, and the first weight coefficient is used to control the calibration degree.

5. The method according to claim 3, wherein: According to the importance levels respectively corresponding to the plurality of second units, calibrating the association relationship between each second unit in the plurality of second units and each first unit in the plurality of first units respectively includes: Determining a second weight coefficient according to the feature information respectively corresponding to the plurality of second units; According to the second weight coefficient and the respective corresponding importance levels of the multiple second units, the association relationship between each second unit in the multiple second units and each first unit in the multiple first units is calibrated, and the second weight coefficient is used to control the calibration degree.

6. The method according to claim 2, wherein: Determining the importance levels respectively corresponding to the plurality of first units according to the association relationship between the one or more entity classifiers and the plurality of first units includes: Performing a pooling operation on the association relationship between the one or more entity classifiers and the multiple first units to obtain importance levels corresponding to the multiple first units respectively; Determining the importance levels respectively corresponding to the plurality of second units according to the association relationship between the one or more entity classifiers and the plurality of second units includes: A pooling operation is performed on the association relationship between the one or more entity classifiers and the plurality of second units to obtain importance degrees corresponding to the plurality of second units respectively.

7. The method according to claim 1, wherein: Identify entities in text messages, including: Identifying one or more character strings in the text information; Predict whether each string is an entity; If the string is an entity, the type of the entity is identified.

8. The method according to claim 1, wherein: The first modality is text information, and the second modality is image information; The first unit is a text unit, and the second unit is an image area.

9. An entity recognition device, wherein: include: An acquisition module, configured to acquire a first modality and a second modality, wherein the first modality is related to the second modality, the first modality includes a plurality of first units, and the second modality includes a plurality of second units; a determination module, configured to determine, according to the feature information respectively corresponding to the plurality of first units and the feature information respectively corresponding to the plurality of second units, an association relationship between the plurality of second units, and an association relationship between the plurality of first units and the plurality of second units; A calibration module, configured to calibrate the association relationship between the plurality of first units and the plurality of second units according to the importance levels respectively corresponding to the plurality of first units and the importance levels respectively corresponding to the plurality of second units; An identification module is used to identify entities in text information based on the association relationship between the multiple first units, the association relationship between the multiple second units, and the calibrated association relationship between the multiple first units and the multiple second units, where the text information is the first modality or the second modality.

10. An electronic device, wherein: include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images