Entity image recognition method and device based on knowledge reasoning, storage medium and electronic device

Abstract

The application provides an entity image identification method and device based on knowledge reasoning, a storage medium and an electronic device, relates to the technical field of entity image identification, and comprises the following steps: inputting a target text into a language model to obtain a plurality of entity mentions corresponding to an entity and vector representations of a text block to which the entity belongs; processing the plurality of entity mentions corresponding to the entity based on a pooling layer and a knowledge graph to obtain a first optimization result; processing entity representations of the entity and the vector representations of the text block to which the entity belongs based on an attention module and a sentiment prediction module to obtain a second optimization result; and performing fusion calculation on the optimization results to obtain optimized entity representations. The entity image identification method and device based on knowledge reasoning, the storage medium and the electronic device provided by the application realize comprehensive and accurate identification of complex entity images by constructing an entity image knowledge graph and combining deep learning and knowledge graph technology for comprehensive knowledge reasoning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of entity image recognition, and in particular to an entity image recognition method and device based on knowledge reasoning, a storage medium and an electronic device. BACKGROUND

[0002] Entity image recognition aims to identify the attitude and evaluation of a specific entity from public discourse. These entities can cover a wide range of categories, including individuals, organizations, and enterprises. In public perception, the image of these entities presents multi-dimensional and multi-level characteristics. Entity image reflects the overall impression and evaluation of the public on a specific entity, not only having a profound impact on the social status and development prospects of the entity, but also playing a key role in its strategic planning and decision-making process.

[0003] In related technologies, the recognition of entity image mainly relies on manual analysis, such as content analysis and case analysis. These analysis methods analyze news reports, social media content, and other content manually to identify different aspects of entity image.

[0004] However, these traditional methods are difficult to meet the real-time analysis needs of massive data and require a large amount of manpower and time. SUMMARY

[0005] The purpose of the present application is to provide an entity image recognition method and device based on knowledge reasoning, a storage medium and an electronic device, which realizes comprehensive and accurate modeling and recognition of complex entity image by constructing an entity image knowledge graph and combining deep learning and knowledge graph technology for comprehensive knowledge reasoning.

[0006] The present application provides an entity image recognition method based on knowledge reasoning, comprising:

[0007] inputting a target text and a named entity recognition result of the target text into a language model to obtain a plurality of entity mentions corresponding to each entity and a vector representation of a text block to which each entity belongs; the named entity recognition result includes a plurality of entities extracted based on the target text; processing a plurality of entity mentions corresponding to each entity based on a pooling layer and a knowledge graph to obtain a first optimization result of an entity representation of each entity, and processing the entity representation of each entity and the vector representation of the text block to which each entity belongs based on an attention module and a sentiment prediction module to obtain a second optimization result of the entity representation of each entity; and performing fusion calculation on the first optimization result and the second optimization result to obtain a second entity representation of each entity after optimization.

[0008] Optionally, the processing of multiple entity mentions corresponding to each entity based on the pooling layer and the knowledge graph to obtain a first optimization result of the entity representation of each entity includes: inputting the multiple entity mentions corresponding to each entity into the pooling layer corresponding to the entity to perform a pooling operation to obtain the first entity representation of each entity; one entity corresponds to one pooling layer; determining at least one associated entity with which each entity has an associated relationship based on the triples of the knowledge graph, and optimizing the first entity representation of each entity based on the relationship vector between each entity and the at least one associated entity to obtain the first optimization result.

[0009] Optionally, the relationship vector contained in the triple in the knowledge graph is optimized by the following steps: obtaining any target triple from the multiple triples contained in the knowledge graph, and constructing a fictitious triple corresponding to the target triple based on the two entities contained in the target triple; the fictitious triple is a negative sample of the target triple; using a distance function to calculate the first distance of the target triple, and using a distance function to calculate the second distance of the fictitious triple, and calculating a loss function value based on the first distance and the second distance; adjusting the relationship vector in the target triple based on the loss function value; wherein the relationship vector is used to represent the association relationship between the two entities contained in the target triple.

[0010] Optionally, the attention module and the sentiment prediction module are based on processing the entity representation of each entity and the vector representation of the text block to which it belongs to obtain a second optimization result of the entity representation of each entity, including: inputting the vector representation of the text block to which each entity belongs and the first entity representation of each entity into the attention module corresponding to the text block to which the entity belongs to obtain the text representation of each text block; the target text is divided into multiple text blocks, each text block contains multiple entities, and each text block corresponds to an attention module; the text representation of each text block is input into the sentiment prediction module corresponding to the text block for sentiment prediction to obtain the second optimization result.

[0011] Optionally, the vector representation of the text block to which each entity belongs and the first entity representation of each entity are input into the attention module corresponding to the text block to which the entity belongs to obtain the text representation of each text block, including: performing a dot product calculation on the entity identifier corresponding to any target entity and the latent vector representation of each word in the vector representation of the text block to which the target entity belongs to obtain the attention score corresponding to each word; normalizing the attention score corresponding to each word to obtain the attention weight of each word, and performing weighted averaging on the latent vector representation of each word based on the attention weight of each word to obtain the text representation of the text block to which the target entity belongs.

[0012] Optionally, the emotion prediction module is trained by the following steps: using a normalized exponential function to calculate the text representation of the text block input into the emotion prediction module to obtain a first image probability distribution corresponding to the text block; calculating a loss function value based on the first image probability distribution and a second image probability distribution corresponding to the entity identified from the text block, and optimizing the emotion prediction module by minimizing the loss function value.

[0013] Optionally, the first optimization result and the second optimization result are fused and calculated to obtain the optimized second entity representation of each entity, including: performing a fusion calculation based on the first optimization result corresponding to any target entity among the multiple entities and the second optimization result corresponding to the text block to which the target entity belongs to obtain the optimized entity representation of the target entity.

[0014] The present application also provides an entity image recognition device based on knowledge reasoning, comprising:

[0015] A data processing module is used to input the target text and the named entity recognition results of the target text into the language model to obtain multiple entity mentions corresponding to each entity and a vector representation of the text block to which each entity belongs; the named entity recognition results include: multiple entities extracted based on the target text; the data processing module is also used to process the multiple entity mentions corresponding to each entity based on the pooling layer and the knowledge graph to obtain a first optimization result of the entity representation of each entity, and to process the entity representation of each entity and the vector representation of the text block to which it belongs based on the attention module and the sentiment prediction module to obtain a second optimization result of the entity representation of each entity; an optimization module is used to perform a fusion calculation on the first optimization result and the second optimization result to obtain an optimized second entity representation for each entity.

[0016] Optionally, the data processing module is specifically used to input multiple entity mentions corresponding to each entity into the pooling layer corresponding to the entity to perform a pooling operation to obtain a first entity representation of each entity; one entity corresponds to one pooling layer; the data processing module is also specifically used to determine at least one associated entity with which each entity has an associated relationship based on the triples of the knowledge graph, and optimize the first entity representation of each entity based on the relationship vector between each entity and at least one associated entity to obtain the first optimization result.

[0017] Optionally, the apparatus further comprises an obtaining module and a training module; the obtaining module is configured to obtain any target triple in a plurality of triples contained in the knowledge graph, and construct a fictitious triple corresponding to the target triple based on two entities contained in the target triple; the fictitious triple is a negative sample of the target triple; the training module is configured to calculate a first distance of the target triple using a distance function, calculate a second distance of the fictitious triple using a distance function, and calculate a loss function value based on the first distance and the second distance; the training module is further configured to adjust a relation vector in the target triple based on the loss function value; wherein the relation vector is used to represent an association between the two entities contained in the target triple.

[0018] Optionally, the data processing module is specifically configured to input the vector representation of the text block to which each entity belongs and the first entity representation of each entity into an attention module corresponding to the text block to which the entity belongs, to obtain a text representation of each text block; the target text is segmented into a plurality of text blocks, each text block containing a plurality of entities, and each text block corresponding to an attention module; the data processing module is specifically further configured to input the text representation of each text block into a sentiment prediction module corresponding to the text block to perform sentiment prediction, to obtain the second optimization result.

[0019] Optionally, the data processing module is specifically configured to perform dot product calculation on the entity identifier corresponding to any target entity and the hidden vector representation of each word in the vector representation of the text block to which the target entity belongs, to obtain an attention score corresponding to each word; the data processing module is specifically further configured to normalize the attention score corresponding to each word to obtain an attention weight of each word, and perform weighted average on the hidden vector representation of each word based on the attention weight of each word, to obtain the text representation of the text block to which the target entity belongs.

[0020] Optionally, the training module is further configured to calculate the text representation of the text block input into the sentiment prediction module using a normalized exponential function to obtain a first image probability distribution corresponding to the text block; the training module is further configured to calculate a loss function value based on the first image probability distribution and a second image probability distribution corresponding to the entity identified from the text block, and optimize the sentiment prediction module by minimizing the loss function value.

[0021] Optionally, the optimization module is specifically configured to perform fusion calculation based on the first optimization result corresponding to any target entity in the plurality of entities and the second optimization result corresponding to the text block to which the target entity belongs, to obtain an optimized entity representation of the target entity.

[0022] The present application also provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of any of the above-mentioned entity image recognition methods based on knowledge reasoning.

[0023] The present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of any of the above-described entity image recognition methods based on knowledge reasoning are implemented.

[0024] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of any of the above-mentioned entity image recognition methods based on knowledge reasoning are implemented.

[0025] The entity image recognition method, device, storage medium and electronic device based on knowledge reasoning provided by the present application first input the target text and the named entity recognition result of the target text into the language model to obtain multiple entity mentions corresponding to each entity; the named entity recognition result includes: multiple entities extracted based on the target text; then, the multiple entity mentions corresponding to each entity are processed based on the pooling layer and the knowledge graph to obtain a first optimization result of the entity representation of each entity, and the entity representation of each entity and the vector representation of the text block to which it belongs are processed based on the attention module and the sentiment prediction module to obtain a second optimization result of the entity representation of each entity; finally, the first optimization result and the second optimization result are fused and calculated to obtain the optimized second entity representation of each entity. In this way, by constructing an entity image knowledge graph and combining deep learning and knowledge graph technology for comprehensive knowledge reasoning, comprehensive and accurate modeling and recognition of complex entity images are achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0027] Figure 1 This is the framework intention of the entity image recognition method based on knowledge reasoning provided by this application;

[0028] Figure 2 This is a flow chart of the entity image recognition method based on knowledge reasoning provided by this application;

[0029] Figure 3This is a schematic diagram of the structure of the entity image recognition device based on knowledge reasoning provided by this application;

[0030] Figure 4 It is a structural diagram of the electronic device provided in this application. DETAILED DESCRIPTION

[0031] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0032] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0033] In response to the above-mentioned technical problems existing in the related technologies, the embodiments of the present application provide an entity image recognition method based on knowledge reasoning. This method comprehensively considers the complex influences between different entities, constructs an entity image knowledge graph, and combines deep learning and knowledge graph technology for comprehensive knowledge reasoning, thereby achieving comprehensive and accurate modeling and recognition of complex entity images. This method can not only effectively process massive amounts of heterogeneous data, but also capture the complex relationships and dynamic changes between entities. This provides more comprehensive and reliable decision-making support for image shaping, communication strategy formulation, and crisis management of various complex entities.

[0034] like Figure 1 The figure shows a schematic diagram of the framework of the entity image recognition method based on knowledge reasoning provided by an embodiment of the present application, including a large language model for encoding text and extracting text latent vector sequences, as well as a pooling layer that outputs entity representations and an attention module that generates text representations.

[0035] The entity image recognition method based on knowledge reasoning provided by the embodiment of the present application is described in detail below with reference to specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0036] like Figure 2As shown, an embodiment of the present application provides a method for entity image recognition based on knowledge reasoning, which may include the following steps 201 to 203:

[0037] Step 201: Input a target text and a named entity recognition result of the target text into a language model to obtain multiple entity mentions corresponding to each entity.

[0038] The named entity recognition result includes: multiple entities extracted based on the target text.

[0039] For example, the embodiments of this application utilize entity extraction technology to accurately identify various entities from input text (i.e., the target text). Given the relative maturity of named entity recognition technology, the embodiments of this application utilize widely recognized open-source tools, such as the Stanford Named Entity Recognizer or the Language Technology Platform (LTP). These tools can efficiently identify various named entities, including names of people, places, and organizations, from text, resulting in the aforementioned named entity recognition results.

[0040] For example, based on entity extraction, the embodiment of the present application further applies entity linking technology to accurately match the entities identified in the text with the corresponding entities in the pre-built knowledge graph. Specifically, for each entity in the text , uniquely associate it with the corresponding knowledge base entity The ID is linked and this link relationship is expressed as .

[0041] For example, after obtaining the named entity recognition results of the target text, the input text can be encoded using a large language model pre-trained on a large-scale text dataset. This step ensures that the generated word representation can fully capture and encode rich linguistic features. Specifically, the embodiment of the present application uses the GPT model to encode the input text and obtain a latent vector sequence of the text, that is, multiple entity mentions corresponding to each entity.

[0042] For example, for each entity in the knowledge graph , which has k mentions , where the latent vector of each entity mention is represented as .

[0043] Step 202: Based on the pooling layer and the knowledge graph, multiple entity mentions corresponding to each entity are processed to obtain a first optimization result of the entity representation of each entity, and based on the attention module and the sentiment prediction module, the entity representation of each entity and the vector representation of the corresponding text block are processed to obtain a second optimization result of the entity representation of each entity.

[0044] For example, after obtaining multiple entity mentions of each entity, the entity representation of each entity and the text representation of each text block can be obtained through pooling operations and attention mechanisms.

[0045] Specifically, the step of processing the multiple entity mentions corresponding to each entity based on the pooling layer and the knowledge graph in step 202 to obtain the first optimization result of the entity representation of each entity may further include the following steps 202a1 and 202a2:

[0046] Step 202a1: Input multiple entity mentions corresponding to each entity into the pooling layer corresponding to the entity to perform a pooling operation to obtain the first entity representation of each entity.

[0047] Among them, one entity corresponds to one pooling layer.

[0048] For example, in the embodiment of the present application, a pooling operation is used to generate an entity representation from the representations of multiple entity mentions, which can be specifically expressed by the following formula 1:

[0049] (Formula 1)

[0050] in, For Entity The entity representation of the entity image provided in the embodiment of the present application is mainly optimized for the entity representation.

[0051] For example, Figure 1 As shown in the figure, after performing named entity recognition on the target text, multiple entities are identified (including entities A, B, C, D, etc.). Then, a large language model is used to encode the text to obtain entity mentions corresponding to each entity (each entity corresponds to k entity mentions). The multiple entity mentions of each entity are output to the pooling layer corresponding to the entity and pooled to obtain the first entity representation of each entity.

[0052] Step 202a2: Determine at least one associated entity with which each entity has an associated relationship based on the triples of the knowledge graph, and optimize the first entity representation of each entity based on the relationship vector between each entity and the at least one associated entity to obtain the first optimization result.

[0053] Exemplarily, after obtaining the first entity representation of each entity, the relationship between entities in the knowledge graph is used to perform knowledge reasoning and optimize the first entity representation.

[0054] For example, Figure 2 As shown in the figure, after obtaining the entity representation of entity D through the pooling layer corresponding to entity D, it can be inferred through the knowledge graph that entity D has an association relationship with entity E and entity F. The relationship vector between entity D and entity E is obtained through To express it, the relationship vector between entity D and entity F is expressed by Through the above relationship vector and relationship vector , the first entity representation of the above entity D can be optimized.

[0055] Specifically, the relationship vectors contained in the triples in the knowledge graph are optimized through the following steps 301 to 303:

[0056] Step 301: obtain any target triple from the multiple triples contained in the knowledge graph, and construct a fictitious triple corresponding to the target triple based on the two entities contained in the target triple; the fictitious triple is a negative sample of the target triple.

[0057] Step 302: Calculate a first distance of the target triplet using a distance function, calculate a second distance of the imaginary triplet using a distance function, and calculate a loss function value based on the first distance and the second distance.

[0058] Step 303: Adjust the relationship vector in the target triple based on the loss function value.

[0059] The relationship vector represents the relationship between the two entities contained in the target triple. The loss function is used to adjust not only the relationship vector but also the entity vector. During the optimization process, the loss function simultaneously influences the update of both through feedback.

[0060] For example, based on the entity representation in the knowledge graph, the embodiment of the present application draws on knowledge graph knowledge representation learning methods, such as TransE and TransR, to implement image reasoning in the knowledge graph. In order to fully utilize the complex association information between entity images in the knowledge graph, the embodiment of the present application constructs a loss function to optimize the distance between triples, thereby capturing the image connection between entities in the vector space. The loss function can be expressed by the following formula 2:

[0061] (Formula 2)

[0062] in, represents the distance function, and Represent the real triples and imaginary triples in the knowledge graph respectively, Represents the relationship vector between entities, is a hyperparameter.

[0063] Specifically, the step of processing the entity representation of each entity and the vector representation of the text block to which it belongs based on the attention module and the sentiment prediction module in step 202 to obtain a second optimization result of the entity representation of each entity may further include the following steps 202b1 and 202b2:

[0064] Step 202b1: Input the vector representation of the text block to which each entity belongs and the first entity representation of each entity into the attention module corresponding to the text block to which the entity belongs, to obtain the text representation of each text block.

[0065] The target text is divided into multiple text blocks, each text block contains multiple entities, and each text block corresponds to an attention module.

[0066] Step 202b2: Input the text representation of each text block into the emotion prediction module corresponding to the text block to perform emotion prediction, and obtain the second optimization result.

[0067] Specifically, the above step 202b1 may further include the following steps 202b11 and 202b12:

[0068] Step 202b11: Perform a dot product calculation on the entity identifier corresponding to any target entity and the latent vector representation of each word in the vector representation of the text block to which the target entity belongs, to obtain an attention score corresponding to each word.

[0069] Step 202b12: normalize the attention score corresponding to each word to obtain the attention weight of each word, and perform weighted averaging on the latent vector representation of each word based on the attention weight of each word to obtain the text representation of the text block to which the target entity belongs.

[0070] For example, in the embodiment of the present application, the attention mechanism is combined to represent the entity and the vector representation of the text where the entity is located To obtain the overall representation of the text. Indicates the first i The hidden vector representation of each word. This process can be expressed by the following formulas 3 and 4:

[0071] (Formula 3)

[0072] (Formula 4)

[0073] in, Represents the attention score calculation function, which converts the entity representation and the latent vector representation of the i-th word Perform dot product to calculate the attention score of the word. Then, normalize it by 𝑠𝑜𝑓𝑡𝑚𝑎𝑥 to calculate the attention weight of each word Finally, the vector representation of each word is weighted averaged with the attention weight to obtain the vector representation of the entire text , which is the textual representation of the text block.

[0074] For example, after obtaining the text representation of the text block, the text representation can be optimized through the sentiment prediction module, thereby indirectly optimizing the entity representation.

[0075] It can be understood that, since the text representation of the text block is obtained based on the entity representation of the entity, optimizing the text representation can indirectly optimize the entity representation.

[0076] Specifically, the emotion prediction module is trained by following steps 303 and 304:

[0077] Step 303: Use a normalized exponential function to calculate the text representation of the text block input into the emotion prediction module to obtain a first image probability distribution corresponding to the text block.

[0078] Step 304: Calculate a loss function value based on the first image probability distribution and a second image probability distribution corresponding to the entity identified from the text block, and optimize the emotion prediction module by minimizing the loss function value.

[0079] For example, in order to make full use of the image information in the text, the embodiment of the present application also constructs a loss function to measure the difference between the text sentiment predicted by the model and the text sentiment manually annotated. First, the text vector representation constructed based on the embodiment of the present application is , use the normalized exponential function softmax function to get the image probability distribution predicted by the emotion prediction model :

[0080] ( (Formula 5)

[0081] in, and b are weight parameters and bias terms.

[0082] For example, after obtaining the image probability distribution predicted by the emotion prediction model, a loss function can be constructed: , and optimize the representation of text by minimizing this function, thereby indirectly optimizing the representation of entities:

[0083] (Formula 6)

[0084] in, is the sentiment tendency predicted by the model (i.e., the model prediction result), It is the real entity sentiment tendency identified from the text (i.e., the manual annotation result).

[0085] Step 203: Perform a fusion calculation on the first optimization result and the second optimization result to obtain an optimized second entity representation of each entity.

[0086] Illustratively, the entity image recognition method based on knowledge reasoning provided in the embodiment of the present application optimizes the entity representation of each entity. Therefore, it is necessary to fuse the optimization results related to each entity to obtain the optimized entity representation.

[0087] Specifically, the above step 203 may further include the following step 203a:

[0088] Step 203a: Perform a fusion calculation based on the first optimization result corresponding to any target entity among the multiple entities and the second optimization result corresponding to the text block to which the target entity belongs, to obtain an optimized entity representation of the target entity.

[0089] For example, Figure 1 As shown, after the optimization results related to entity D are fused and calculated, the optimized entity representation of entity D is obtained.

[0090] The entity image recognition method based on knowledge reasoning provided in the embodiment of the present application first inputs the target text and the named entity recognition result of the target text into the language model to obtain multiple entity mentions corresponding to each entity; the named entity recognition result includes: multiple entities extracted based on the target text; then, the multiple entity mentions corresponding to each entity are processed based on the pooling layer and the knowledge graph to obtain a first optimization result of the entity representation of each entity, and the entity representation of each entity and the vector representation of the text block to which it belongs are processed based on the attention module and the sentiment prediction module to obtain a second optimization result of the entity representation of each entity; finally, the first optimization result and the second optimization result are fused and calculated to obtain the optimized second entity representation of each entity. In this way, by constructing an entity image knowledge graph and combining deep learning and knowledge graph technology for comprehensive knowledge reasoning, comprehensive and accurate modeling and recognition of complex entity images are achieved.

[0091] It should be noted that the entity image recognition method based on knowledge reasoning provided in the embodiments of the present application can be executed by a knowledge reasoning-based entity image recognition device, or a control module within the knowledge reasoning-based entity image recognition device that is used to execute the knowledge reasoning-based entity image recognition method. In the embodiments of the present application, the entity image recognition device based on knowledge reasoning, which executes the knowledge reasoning-based entity image recognition method, is used as an example to illustrate the knowledge reasoning-based entity image recognition device provided in the embodiments of the present application.

[0092] It should be noted that in the embodiments of this application, the entity image recognition method based on knowledge reasoning shown in the figures of the above-mentioned methods is described as an example in conjunction with one figure in the embodiments of this application. In specific implementation, the entity image recognition method based on knowledge reasoning shown in the figures of the above-mentioned methods can also be implemented in conjunction with any other combinable figures shown in the above-mentioned embodiments, and no further details are given here.

[0093] The entity image recognition device based on knowledge reasoning provided by the present application is described below. The entity image recognition method based on knowledge reasoning described below and the entity image recognition method based on knowledge reasoning described above can be referenced to each other.

[0094] Figure 3 A schematic diagram of the structure of the entity image recognition device based on knowledge reasoning provided in the embodiment of the present application is shown as follows: Figure 3 As shown, specifically including:

[0095] The data processing module 301 is used to input the target text and the named entity recognition result of the target text into the language model to obtain multiple entity mentions corresponding to each entity and the vector representation of the text block to which each entity belongs; the named entity recognition result includes: multiple entities extracted based on the target text; the data processing module 301 is also used to process the multiple entity mentions corresponding to each entity based on the pooling layer and the knowledge graph to obtain a first optimization result of the entity representation of each entity, and to process the entity representation of each entity and the vector representation of the text block to which it belongs based on the attention module and the sentiment prediction module to obtain a second optimization result of the entity representation of each entity; the optimization module 302 is used to perform a fusion calculation on the first optimization result and the second optimization result to obtain the optimized second entity representation of each entity.

[0096] Optionally, the data processing module 301 is specifically used to input multiple entity mentions corresponding to each entity into the pooling layer corresponding to the entity to perform a pooling operation to obtain a first entity representation of each entity; one entity corresponds to one pooling layer; the data processing module 301 is also specifically used to determine at least one associated entity with which each entity has an associated relationship based on the triples of the knowledge graph, and optimize the first entity representation of each entity based on the relationship vector between each entity and at least one associated entity to obtain the first optimization result.

[0097] Optionally, the device also includes: an acquisition module and a training module; the acquisition module is used to acquire any target triple from the multiple triples contained in the knowledge graph, and construct a fictitious triple corresponding to the target triple based on the two entities contained in the target triple; the fictitious triple is a negative sample of the target triple; the training module is used to calculate the first distance of the target triple using a distance function, and calculate the second distance of the fictitious triple using a distance function, and calculate the loss function value based on the first distance and the second distance; the training module is also used to adjust the relationship vector in the target triple based on the loss function value; wherein, the relationship vector is used to represent the association relationship between the two entities contained in the target triple.

[0098] Optionally, the data processing module 301 is specifically used to input the vector representation of the text block to which each entity belongs and the first entity representation of each entity into the attention module corresponding to the text block to which the entity belongs, so as to obtain the text representation of each text block; the target text is divided into multiple text blocks, each text block contains multiple entities, and each text block corresponds to an attention module; the data processing module 301 is also specifically used to input the text representation of each text block into the emotion prediction module corresponding to the text block for emotion prediction, so as to obtain the second optimization result.

[0099] Optionally, the data processing module 301 is specifically used to perform a dot product calculation on the entity identifier corresponding to any target entity and the latent vector representation of each word in the vector representation of the text block to which the target entity belongs, to obtain an attention score corresponding to each word; the data processing module 301 is also specifically used to normalize the attention score corresponding to each word to obtain an attention weight for each word, and perform a weighted average of the latent vector representation of each word based on the attention weight of each word to obtain a text representation of the text block to which the target entity belongs.

[0100] Optionally, the training module is also used to use a normalized exponential function to calculate the text representation of the text block input into the emotion prediction module to obtain a first image probability distribution corresponding to the text block; the training module is also used to calculate a loss function value based on the first image probability distribution and a second image probability distribution corresponding to the entity identified from the text block, and optimize the emotion prediction module by minimizing the loss function value.

[0101] Optionally, the optimization module 302 is specifically used to perform a fusion calculation based on a first optimization result corresponding to any target entity among the multiple entities and a second optimization result corresponding to the text block to which the target entity belongs, to obtain an optimized entity representation of the target entity.

[0102] The entity image recognition device based on knowledge reasoning provided by the present application first inputs the target text and the named entity recognition result of the target text into the language model to obtain multiple entity mentions corresponding to each entity; the named entity recognition result includes: multiple entities extracted based on the target text; then, the multiple entity mentions corresponding to each entity are processed based on the pooling layer and the knowledge graph to obtain a first optimization result of the entity representation of each entity, and the entity representation of each entity and the vector representation of the text block to which it belongs are processed based on the attention module and the sentiment prediction module to obtain a second optimization result of the entity representation of each entity; finally, the first optimization result and the second optimization result are fused and calculated to obtain the optimized second entity representation of each entity. In this way, by constructing an entity image knowledge graph and combining deep learning and knowledge graph technology for comprehensive knowledge reasoning, comprehensive and accurate modeling and recognition of complex entity images are achieved.

[0103] Figure 4 An example of a physical structure diagram of an electronic device is shown below. Figure 4As shown, the electronic device can include a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 complete mutual communication through the communications bus 440. The processor 410 can invoke a logical instruction in the memory 430 to execute an entity image recognition method based on knowledge reasoning, which includes: inputting a target text and a named entity recognition result of the target text into a language model to obtain a plurality of entity mentions corresponding to each entity and a vector representation of a text block to which each entity belongs; the named entity recognition result includes a plurality of entities extracted based on the target text; processing a plurality of entity mentions corresponding to each entity based on a pooling layer and a knowledge graph to obtain a first optimization result of an entity representation of each entity, and processing the entity representation of each entity and the vector representation of the text block to which each entity belongs based on an attention module and a sentiment prediction module to obtain a second optimization result of the entity representation of each entity; and performing fusion calculation on the first optimization result and the second optimization result to obtain a second entity representation of each entity after optimization.

[0104] In addition, the logical instructions in the memory 430 described above can be implemented in the form of a software functional unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0105] In another aspect, the present application also provides a computer program product, which comprises a computer program stored on a computer readable storage medium, and the computer program comprises program instructions, when the program instructions are executed by a computer, the computer can execute the knowledge reasoning-based entity image recognition method provided by the above method, and the method comprises the following steps: inputting a target text and a named entity recognition result of the target text into a language model to obtain a plurality of entity mentions corresponding to each entity and a vector representation of a text block to which each entity belongs; the named entity recognition result comprises a plurality of entities extracted based on the target text; performing processing on the plurality of entity mentions corresponding to each entity based on a pooling layer and a knowledge graph to obtain a first optimization result of an entity representation of each entity, and performing processing on the entity representation of each entity and the vector representation of the text block to which each entity belongs based on an attention module and a sentiment prediction module to obtain a second optimization result of the entity representation of each entity; and performing fusion calculation on the first optimization result and the second optimization result to obtain a second entity representation of each entity after optimization.

[0106] In another aspect, the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the knowledge reasoning-based entity image recognition method provided by the above method, and the method comprises the following steps: inputting a target text and a named entity recognition result of the target text into a language model to obtain a plurality of entity mentions corresponding to each entity and a vector representation of a text block to which each entity belongs; the named entity recognition result comprises a plurality of entities extracted based on the target text; performing processing on the plurality of entity mentions corresponding to each entity based on a pooling layer and a knowledge graph to obtain a first optimization result of an entity representation of each entity, and performing processing on the entity representation of each entity and the vector representation of the text block to which each entity belongs based on an attention module and a sentiment prediction module to obtain a second optimization result of the entity representation of each entity; and performing fusion calculation on the first optimization result and the second optimization result to obtain a second entity representation of each entity after optimization.

[0107] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment scheme. Those skilled in the art can understand and implement without creative labor.

[0108] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for entity image recognition based on knowledge reasoning, characterized in that: include: Inputting the target text and the named entity recognition results of the target text into the language model to obtain multiple entity mentions corresponding to each entity and a vector representation of the text block to which each entity belongs; The named entity recognition result includes: a plurality of entities extracted based on the target text; Based on the pooling layer and the knowledge graph, multiple entity mentions corresponding to each entity are processed to obtain a first optimization result of the entity representation of each entity. Based on the attention module and the sentiment prediction module, the entity representation of each entity and the vector representation of the corresponding text block are processed to obtain a second optimization result of the entity representation of each entity. Performing a fusion calculation on the first optimization result and the second optimization result to obtain an optimized second entity representation of each entity; The processing of multiple entity mentions corresponding to each entity based on the pooling layer and the knowledge graph to obtain a first optimization result of the entity representation of each entity includes: Input multiple entity mentions corresponding to each entity into the pooling layer corresponding to the entity to perform pooling operation to obtain the first entity representation of each entity; one entity corresponds to one pooling layer; Determining at least one associated entity with which each entity has an associated relationship based on the triples of the knowledge graph, and optimizing a first entity representation of each entity based on a relationship vector between each entity and the at least one associated entity to obtain the first optimization result; The attention module and the sentiment prediction module process the entity representation of each entity and the vector representation of the text block to which it belongs, to obtain a second optimization result of the entity representation of each entity, including: Inputting the vector representation of the text block to which each entity belongs and the first entity representation of each entity into the attention module corresponding to the text block to which the entity belongs to obtain the text representation of each text block; the target text is divided into multiple text blocks, each text block contains multiple entities, and each text block corresponds to an attention module; The text representation of each text block is input into the emotion prediction module corresponding to the text block to perform emotion prediction, thereby obtaining the second optimization result.

2. The method according to claim 1, characterized in that The relationship vectors contained in the triples in the knowledge graph are optimized by the following steps: Obtain any target triple from the multiple triples contained in the knowledge graph, and construct a fictitious triple corresponding to the target triple based on two entities contained in the target triple; the fictitious triple is a negative sample of the target triple; Calculating a first distance of the target triplet using a distance function, and calculating a second distance of the fictitious triplet using a distance function, and calculating a loss function value based on the first distance and the second distance; Adjusting the relationship vector in the target triple based on the loss function value; The relationship vector is used to represent the association relationship between the two entities included in the target triple.

3. The method according to claim 1, characterized in that The vector representation of the text block to which each entity belongs and the first entity representation of each entity are input into the attention module corresponding to the text block to which the entity belongs to obtain the text representation of each text block, including: Perform a dot product calculation on the entity identifier corresponding to any target entity and the latent vector representation of each word in the vector representation of the text block to which the target entity belongs, and obtain the attention score corresponding to each word; The attention score corresponding to each word is normalized to obtain the attention weight of each word, and the latent vector representation of each word is weighted averaged based on the attention weight of each word to obtain the text representation of the text block to which the target entity belongs.

4. The method according to claim 1 or 3, characterized in that The emotion prediction module is trained by the following steps: Using a normalized exponential function to calculate the text representation of the text block input into the emotion prediction module to obtain a first image probability distribution corresponding to the text block; A loss function value is calculated based on the first image probability distribution and a second image probability distribution corresponding to the entity identified from the text block, and the emotion prediction module is optimized by minimizing the loss function value.

5. The method according to claim 1, wherein The fusing calculation of the first optimization result and the second optimization result to obtain the optimized second entity representation of each entity includes: A fusion calculation is performed based on a first optimization result corresponding to any target entity among the multiple entities and a second optimization result corresponding to the text block to which the target entity belongs to obtain an optimized entity representation of the target entity.

6. An entity image recognition device based on knowledge reasoning, characterized in that: The device comprises: A data processing module is configured to input a target text and a named entity recognition result of the target text into a language model to obtain a plurality of entity mentions corresponding to each entity and a vector representation of the text block to which each entity belongs; the named entity recognition result includes: a plurality of entities extracted based on the target text; The data processing module is further configured to process multiple entity mentions corresponding to each entity based on the pooling layer and the knowledge graph to obtain a first optimization result of the entity representation of each entity, and to process the entity representation of each entity and the vector representation of the text block to which it belongs based on the attention module and the sentiment prediction module to obtain a second optimization result of the entity representation of each entity; an optimization module, configured to perform a fusion calculation on the first optimization result and the second optimization result to obtain an optimized second entity representation of each entity; The data processing module is specifically configured to input multiple entity mentions corresponding to each entity into a pooling layer corresponding to the entity to perform a pooling operation to obtain a first entity representation of each entity; one entity corresponds to one pooling layer; the data processing module is further specifically configured to determine, based on triples of the knowledge graph, at least one associated entity with which each entity has an associated relationship, and optimize the first entity representation of each entity based on a relationship vector between each entity and the at least one associated entity to obtain the first optimization result; The data processing module is specifically used to input the vector representation of the text block to which each entity belongs and the first entity representation of each entity into the attention module corresponding to the text block to which the entity belongs, so as to obtain the text representation of each text block; the target text is divided into multiple text blocks, each text block contains multiple entities, and each text block corresponds to an attention module; the data processing module is also specifically used to input the text representation of each text block into the emotion prediction module corresponding to the text block for emotion prediction, so as to obtain the second optimization result.

7. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the entity image recognition method based on knowledge reasoning as claimed in any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the entity image recognition method based on knowledge reasoning as claimed in any one of claims 1 to 5 are implemented.

Images