Physical image recognition method and device based on knowledge reasoning, storage medium and electronic equipment

Abstract

The invention provides an entity image recognition method and device based on knowledge reasoning, a storage medium and electronic equipment, and relates to the technical field of entity image recognition, and the method comprises the steps: inputting a target text into a language model, and obtaining a plurality of entity mentions corresponding to an entity and vector representation of a text block to which the entity belongs; processing a plurality of entity mentions corresponding to the entities based on a pooling layer and a knowledge graph to obtain a first optimization result, and processing entity representations of the entities and vector representations of text blocks to which the entities belong based on an attention module and an emotion prediction module to obtain a second optimization result; and carrying out fusion calculation on the optimization result to obtain an optimized entity representation. According to the entity image recognition method and device based on knowledge reasoning, the storage medium and the electronic equipment provided by the invention, comprehensive knowledge reasoning is carried out by constructing the entity image knowledge graph and combining deep learning and knowledge graph technologies, and comprehensive and accurate recognition of a complex entity image is realized.

Description

Technical Field

[0001] The present application relates to the technical field of entity image recognition, and in particular to a method, device, storage medium and electronic device for entity image recognition based on knowledge reasoning. Background Art

[0002] Entity image recognition aims to identify attitudes and evaluations of specific entities from public discourse. These entities can cover a wide range of categories, including individuals, organizations, and enterprises. In public cognition, the images of these entities present multi-dimensional and multi-layered characteristics. Entity image comprehensively reflects the public's overall impression and evaluation of a specific entity, which not only has a profound impact on the social status and development prospects of the entity, but also plays a key role in its strategic planning and decision-making process.

[0003] In related technologies, the identification of entity images mainly relies on manual analysis, such as content analysis and case analysis. These analysis methods identify different aspects of entity images by manually analyzing news reports, social media content, etc.

[0004] However, these traditional methods are unable to cope with the real-time analysis needs of massive data and require a lot of manpower and time. Summary of the invention

[0005] The purpose of this application is to provide a method, device, storage medium and electronic device for entity image recognition based on knowledge reasoning. By constructing an entity image knowledge graph and combining deep learning and knowledge graph technology for comprehensive knowledge reasoning, comprehensive and accurate modeling and recognition of complex entity images are achieved.

[0006] The present application provides an entity image recognition method based on knowledge reasoning, comprising: The target text and the named entity recognition result of the target text are input into the language model to obtain multiple entity mentions corresponding to each entity and a vector representation of the text block to which each entity belongs; the named entity recognition result includes: multiple entities extracted based on the target text; multiple entity mentions corresponding to each entity are processed based on the pooling layer and the knowledge graph to obtain a first optimization result of the entity representation of each entity, and the entity representation of each entity and the vector representation of the text block to which it belongs are processed based on the attention module and the sentiment prediction module to obtain a second optimization result of the entity representation of each entity; the first optimization result and the second optimization result are fused and calculated to obtain the optimized second entity representation of each entity.

[0007] Optionally, the multiple entity mentions corresponding to each entity are processed based on the pooling layer and the knowledge graph to obtain a first optimization result of the entity representation of each entity, including: inputting the multiple entity mentions corresponding to each entity into the pooling layer corresponding to the entity to perform a pooling operation to obtain a first entity representation of each entity; one entity corresponds to one pooling layer; based on the triples of the knowledge graph, at least one associated entity with which each entity has an associated relationship is determined, and the first entity representation of each entity is optimized based on the relationship vector between each entity and at least one associated entity to obtain the first optimization result.

[0008] Optionally, the relationship vector contained in the triples in the knowledge graph is optimized by the following steps: obtaining any target triple from the multiple triples contained in the knowledge graph, and constructing a fictitious triple corresponding to the target triple based on the two entities contained in the target triple; the fictitious triple is a negative sample of the target triple; using a distance function to calculate the first distance of the target triple, and using a distance function to calculate the second distance of the fictitious triple, and calculating a loss function value based on the first distance and the second distance; adjusting the relationship vector in the target triple based on the loss function value; wherein the relationship vector is used to represent the association relationship between the two entities contained in the target triple.

[0009] Optionally, the attention module and the sentiment prediction module are used to process the entity representation of each entity and the vector representation of the text block to which it belongs to obtain a second optimization result of the entity representation of each entity, including: inputting the vector representation of the text block to which each entity belongs and the first entity representation of each entity into the attention module corresponding to the text block to which the entity belongs to obtain a text representation of each text block; the target text is divided into multiple text blocks, each text block contains multiple entities, and each text block corresponds to an attention module; the text representation of each text block is input into the sentiment prediction module corresponding to the text block to perform sentiment prediction to obtain the second optimization result.

[0010] Optionally, the vector representation of the text block to which each entity belongs and the first entity representation of each entity are input into the attention module corresponding to the text block to which the entity belongs to obtain the text representation of each text block, including: performing dot product calculation on the entity identifier corresponding to any target entity and the latent vector representation of each word in the vector representation of the text block to which the target entity belongs to obtain the attention score corresponding to each word; normalizing the attention score corresponding to each word to obtain the attention weight of each word, and performing weighted averaging on the latent vector representation of each word based on the attention weight of each word to obtain the text representation of the text block to which the target entity belongs.

[0011] Optionally, the sentiment prediction module is trained through the following steps: using a normalized exponential function to calculate the text representation of the text block input into the sentiment prediction module to obtain a first image probability distribution corresponding to the text block; calculating a loss function value based on the first image probability distribution and a second image probability distribution corresponding to the entity identified from the text block, and optimizing the sentiment prediction module by minimizing the loss function value.

[0012] Optionally, the first optimization result and the second optimization result are fused to obtain the optimized second entity representation of each entity, including: fusion calculation based on the first optimization result corresponding to any target entity among the multiple entities and the second optimization result corresponding to the text block to which the target entity belongs to obtain the optimized entity representation of the target entity.

[0013] The present application also provides an entity image recognition device based on knowledge reasoning, comprising: A data processing module is used to input a target text and a named entity recognition result of the target text into a language model to obtain multiple entity mentions corresponding to each entity and a vector representation of a text block to which each entity belongs; the named entity recognition result includes: multiple entities extracted based on the target text; the data processing module is also used to process the multiple entity mentions corresponding to each entity based on a pooling layer and a knowledge graph to obtain a first optimization result of the entity representation of each entity, and to process the entity representation of each entity and the vector representation of the text block to which it belongs based on an attention module and a sentiment prediction module to obtain a second optimization result of the entity representation of each entity; an optimization module is used to perform a fusion calculation on the first optimization result and the second optimization result to obtain an optimized second entity representation of each entity.

[0014] Optionally, the data processing module is specifically used to input multiple entity mentions corresponding to each entity into the pooling layer corresponding to the entity to perform a pooling operation to obtain a first entity representation of each entity; one entity corresponds to one pooling layer; the data processing module is also specifically used to determine at least one associated entity with which each entity has an associated relationship based on the triples of the knowledge graph, and optimize the first entity representation of each entity based on the relationship vector between each entity and at least one associated entity to obtain the first optimization result.

[0015] Optionally, the device also includes: an acquisition module and a training module; the acquisition module is used to acquire any target triple from the multiple triples contained in the knowledge graph, and construct a fictitious triple corresponding to the target triple based on the two entities contained in the target triple; the fictitious triple is a negative sample of the target triple; the training module is used to calculate the first distance of the target triple using a distance function, and calculate the second distance of the fictitious triple using a distance function, and calculate the loss function value based on the first distance and the second distance; the training module is also used to adjust the relationship vector in the target triple based on the loss function value; wherein the relationship vector is used to represent the association relationship between the two entities contained in the target triple.

[0016] Optionally, the data processing module is specifically used to input the vector representation of the text block to which each entity belongs and the first entity representation of each entity into an attention module corresponding to the text block to which the entity belongs, so as to obtain a text representation of each text block; the target text is divided into multiple text blocks, each text block contains multiple entities, and each text block corresponds to an attention module; the data processing module is also specifically used to input the text representation of each text block into a sentiment prediction module corresponding to the text block to perform sentiment prediction, so as to obtain the second optimization result.

[0017] Optionally, the data processing module is specifically used to perform a dot product calculation on the entity identifier corresponding to any target entity and the latent vector representation of each word in the vector representation of the text block to which the target entity belongs, so as to obtain an attention score corresponding to each word; the data processing module is also specifically used to normalize the attention score corresponding to each word to obtain an attention weight for each word, and to perform a weighted average on the latent vector representation of each word based on the attention weight of each word to obtain a text representation of the text block to which the target entity belongs.

[0018] Optionally, the training module is also used to use a normalized exponential function to calculate the text representation of the text block input into the sentiment prediction module to obtain a first image probability distribution corresponding to the text block; the training module is also used to calculate a loss function value based on the first image probability distribution and a second image probability distribution corresponding to the entity identified from the text block, and optimize the sentiment prediction module by minimizing the loss function value.

[0019] Optionally, the optimization module is specifically used to perform a fusion calculation based on a first optimization result corresponding to any target entity among the multiple entities and a second optimization result corresponding to a text block to which the target entity belongs, so as to obtain an optimized entity representation of the target entity.

[0020] The present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of any of the above-mentioned entity image recognition methods based on knowledge reasoning.

[0021] The present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-mentioned entity image recognition methods based on knowledge reasoning are implemented.

[0022] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of any of the above-mentioned entity image recognition methods based on knowledge reasoning are implemented.

[0023] The entity image recognition method, device, storage medium and electronic device based on knowledge reasoning provided by the present application first input the target text and the named entity recognition result of the target text into the language model to obtain multiple entity mentions corresponding to each entity; the named entity recognition result includes: multiple entities extracted based on the target text; then, the multiple entity mentions corresponding to each entity are processed based on the pooling layer and the knowledge graph to obtain the first optimization result of the entity representation of each entity, and the entity representation of each entity and the vector representation of the text block to which it belongs are processed based on the attention module and the sentiment prediction module to obtain the second optimization result of the entity representation of each entity; finally, the first optimization result and the second optimization result are fused and calculated to obtain the optimized second entity representation of each entity. In this way, by constructing an entity image knowledge graph and combining deep learning and knowledge graph technology for comprehensive knowledge reasoning, comprehensive and accurate modeling and recognition of complex entity images are achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 This is the framework intention of the entity image recognition method based on knowledge reasoning provided by this application; Figure 2 It is a flowchart of the entity image recognition method based on knowledge reasoning provided by this application; Figure 3 It is a structural schematic diagram of an entity image recognition device based on knowledge reasoning provided by the present application; Figure 4 It is a structural schematic diagram of the electronic device provided by this application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0028] In response to the above-mentioned technical problems existing in the related technologies, the embodiments of the present application provide an entity image recognition method based on knowledge reasoning, which comprehensively considers the complex influences between different entities, constructs an entity image knowledge graph, and combines deep learning and knowledge graph technology for comprehensive knowledge reasoning, thereby achieving comprehensive and accurate modeling and recognition of complex entity images. This method can not only effectively process massive heterogeneous data, but also capture the complex relationships and dynamic changes between entities. This provides more comprehensive and reliable decision-making support for image shaping, communication strategy formulation, and crisis management of various complex entities.

[0029] like Figure 1 As shown, it is a schematic diagram of the framework of the entity image recognition method based on knowledge reasoning provided by the embodiment of the present application, including a large language model for encoding text and extracting text latent vector sequences, as well as a pooling layer for outputting entity representation and an attention module for generating text representation.

[0030] The entity image recognition method based on knowledge reasoning provided by the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0031] like Figure 2 As shown, an embodiment of the present application provides a method for entity image recognition based on knowledge reasoning, which may include the following steps 201 to 203: Step 201: input a target text and a named entity recognition result of the target text into a language model to obtain a plurality of entity mentions corresponding to each entity.

[0032] The named entity recognition result includes: a plurality of entities extracted based on the target text.

[0033] Exemplarily, the embodiments of the present application use entity extraction technology to accurately identify various entities from the input text material (i.e., the above-mentioned target text). In view of the fact that named entity recognition technology is relatively mature, the embodiments of the present application adopt open source tools widely recognized by the academic community, such as the Stanford Named Entity Recognizer or the Language Technology Platform (LTP). These tools can efficiently identify various named entities including names of people, places, and organizations from the text, i.e., the above-mentioned named entity recognition results.

[0034] For example, based on entity extraction, the embodiment of the present application further applies entity linking technology to accurately match the entities identified in the text with the corresponding entities in the pre-built knowledge graph. Specifically, for each entity in the text , uniquely associate it with the corresponding knowledge base entity The ID is linked to each other, and this link relationship is represented as .

[0035] Exemplarily, after obtaining the named entity recognition result of the target text, the input text can be encoded using a large language model pre-trained on a large-scale text dataset. This step ensures that the generated word representation can fully capture and encode rich linguistic features. Specifically, the embodiment of the present application uses the GPT model to encode the input text to obtain a latent vector sequence of the text, that is, multiple entity mentions corresponding to each entity.

[0036] For example, for each entity in the knowledge graph , which has k mentions , where the latent vector of each entity mention is represented as .

[0037] Step 202: Based on the pooling layer and the knowledge graph, multiple entity mentions corresponding to each entity are processed to obtain a first optimization result of the entity representation of each entity, and based on the attention module and the sentiment prediction module, the entity representation of each entity and the vector representation of the corresponding text block are processed to obtain a second optimization result of the entity representation of each entity.

[0038] For example, after obtaining multiple entity mentions of each entity, the entity representation of each entity and the text representation of each text block can be obtained through pooling operations and attention mechanisms.

[0039] Specifically, the step of processing multiple entity mentions corresponding to each entity based on the pooling layer and the knowledge graph in the above step 202 to obtain a first optimization result of the entity representation of each entity may also include the following steps 202a1 and 202a2: Step 202a1: Input multiple entity mentions corresponding to each entity into the pooling layer corresponding to the entity to perform a pooling operation to obtain a first entity representation of each entity.

[0040] Among them, one entity corresponds to one pooling layer.

[0041] Exemplarily, in the embodiment of the present application, a pooling operation is used to generate a representation of an entity from representations of multiple entity mentions, which can be specifically represented by the following formula 1: (Formula 1)

[0042] in, For Entity The entity representation of the entity image, the entity image recognition method based on knowledge reasoning provided in the embodiment of the present application is mainly optimized for the entity representation.

[0043] For example, Figure 1 As shown in the figure, after performing named entity recognition on the target text, multiple entities are identified (including entities A, B, C, D, etc.). Then, the text is encoded using a large language model to obtain entity mentions corresponding to each entity (each entity corresponds to k entity mentions). Multiple entity mentions of each entity are output to the pooling layer corresponding to the entity to perform pooling operations and obtain the first entity representation of each entity.

[0044] Step 202a2: determine at least one associated entity with which each entity has an associated relationship based on the triples of the knowledge graph, and optimize the first entity representation of each entity based on the relationship vector between each entity and at least one associated entity to obtain the first optimization result.

[0045] Exemplarily, after obtaining the first entity representation of each entity, knowledge reasoning is performed using the relationship between entities in the knowledge graph to optimize the first entity representation.

[0046] For example, Figure 2 As shown in the figure, after obtaining the entity representation of entity D through the pooling layer corresponding to entity D, it can be inferred through the knowledge graph that entity D has an association relationship with entity E and entity F. The relationship vector between entity D and entity E is obtained through The relationship vector between entity D and entity F is represented by Through the above relationship vector and relationship vector , the first entity representation of the above entity D can be optimized.

[0047] Specifically, the relationship vectors contained in the triples in the knowledge graph are optimized by following steps 301 to 303: Step 301: obtain any target triple from multiple triples contained in the knowledge graph, and construct a fictitious triple corresponding to the target triple based on two entities contained in the target triple; the fictitious triple is a negative sample of the target triple.

[0048] Step 302: Use a distance function to calculate a first distance of the target triplet, use a distance function to calculate a second distance of the imaginary triplet, and calculate a loss function value based on the first distance and the second distance.

[0049] Step 303: Adjust the relationship vector in the target triplet based on the loss function value.

[0050] The relationship vector is used to represent the association relationship between the two entities contained in the target triple. The loss function is used not only to adjust the relationship vector, but also to adjust the entity vector. During the optimization process, the loss function affects the update of both at the same time through feedback.

[0051] Exemplarily, based on the entity representation in the knowledge graph, the embodiment of the present application draws on the knowledge graph knowledge representation learning method, such as TransE, TransR, etc., to realize image reasoning in the knowledge graph. In order to make full use of the complex association information between entity images in the knowledge graph, the embodiment of the present application constructs a loss function to optimize the distance between triples, and then capture the image connection between entities in the vector space. The loss function can be expressed by the following formula 2: (Formula 2)

[0052] in, represents the distance function, and They represent real triples and imaginary triples in the knowledge graph respectively. Represents the relationship vector between entities, is a hyperparameter.

[0053] Specifically, with respect to the step of processing the entity representation of each entity and the vector representation of the text block to which it belongs based on the attention module and the sentiment prediction module in the above step 202 to obtain the second optimization result of the entity representation of each entity, the following steps 202b1 and 202b2 may also be included: Step 202b1: input the vector representation of the text block to which each entity belongs and the first entity representation of each entity into the attention module corresponding to the text block to which the entity belongs, so as to obtain the text representation of each text block.

[0054] The target text is divided into multiple text blocks, each text block contains multiple entities, and each text block corresponds to an attention module.

[0055] Step 202b2: input the text representation of each text block into the emotion prediction module corresponding to the text block to perform emotion prediction, and obtain the second optimization result.

[0056] Specifically, the above step 202b1 may further include the following steps 202b11 and 202b12: Step 202b11: perform a dot product calculation on the entity identifier corresponding to any target entity and the latent vector representation of each word in the vector representation of the text block to which the target entity belongs, to obtain an attention score corresponding to each word.

[0057] Step 202b12: normalize the attention score corresponding to each word to obtain the attention weight of each word, and perform weighted averaging on the latent vector representation of each word based on the attention weight of each word to obtain the text representation of the text block to which the target entity belongs.

[0058] For example, in the embodiment of the present application, the attention mechanism is combined to represent the entity and the vector representation of the text where the entity is located To obtain the overall representation of the text. Indicates that the text i The hidden vector representation of each word. This process can be expressed by the following formulas 3 and 4: (Formula 3)

[0059] (Formula 4)

[0060] in, represents the attention score calculation function, which converts the entity representation and the latent vector representation of the i-th word Dot product is performed to calculate the attention score of the word. Then, the attention weight of each word is calculated by normalizing it through 𝑠𝑜𝑓𝑡𝑚𝑎𝑥 Finally, the vector representation of each word is weighted averaged with the attention weight to obtain the vector representation of the entire text. , which is the textual representation of the text block.

[0061] For example, after obtaining the text representation of the text block, the text representation can be optimized through the sentiment prediction module, thereby achieving indirect optimization of the entity representation.

[0062] It can be understood that, since the text representation of the text block is obtained based on the entity representation of the entity, optimizing the text representation can indirectly optimize the entity representation.

[0063] Specifically, the emotion prediction module is trained by following steps 303 and 304: Step 303: Use a normalized exponential function to calculate the text representation of the text block input into the emotion prediction module to obtain a first image probability distribution corresponding to the text block.

[0064] Step 304: Calculate a loss function value based on the first image probability distribution and a second image probability distribution corresponding to the entity identified from the text block, and optimize the sentiment prediction module by minimizing the loss function value.

[0065] For example, in order to make full use of the image information in the text, the embodiment of the present application also constructs a loss function to measure the difference between the text sentiment predicted by the model and the text sentiment annotated manually. First, based on the text vector representation constructed in the embodiment of the present application , use the normalized exponential function softmax function to get the image probability distribution predicted by the emotion prediction model : ( (Formula 5)

[0066] in, and b are weight parameters and bias terms.

[0067] For example, after obtaining the image probability distribution predicted by the emotion prediction model, a loss function can also be constructed: , and optimize the representation of text by minimizing this function, thereby indirectly optimizing the representation of entities: (Formula 6)

[0068] in, is the sentiment tendency predicted by the model (i.e., the model prediction result), It is the real entity sentiment tendency identified from the text (i.e., the manual annotation result).

[0069] Step 203: perform a fusion calculation on the first optimization result and the second optimization result to obtain a second entity representation of each entity after optimization.

[0070] Exemplarily, the entity image recognition method based on knowledge reasoning provided in the embodiment of the present application optimizes the entity representation of each entity. Therefore, it is necessary to fuse the optimization results related to each entity to obtain the optimized entity representation.

[0071] Specifically, the above step 203 may further include the following step 203a: Step 203a: perform a fusion calculation based on the first optimization result corresponding to any target entity among the multiple entities and the second optimization result corresponding to the text block to which the target entity belongs, to obtain an optimized entity representation of the target entity.

[0072] For example, Figure 1 As shown, after the optimization results related to entity D are fused and calculated, the optimized entity representation of entity D is obtained.

[0073] The entity image recognition method based on knowledge reasoning provided in the embodiment of the present application first inputs the target text and the named entity recognition result of the target text into the language model to obtain multiple entity mentions corresponding to each entity; the named entity recognition result includes: multiple entities extracted based on the target text; then, the multiple entity mentions corresponding to each entity are processed based on the pooling layer and the knowledge graph to obtain the first optimization result of the entity representation of each entity, and the entity representation of each entity and the vector representation of the text block to which it belongs are processed based on the attention module and the sentiment prediction module to obtain the second optimization result of the entity representation of each entity; finally, the first optimization result and the second optimization result are fused and calculated to obtain the optimized second entity representation of each entity. In this way, by constructing an entity image knowledge graph and combining deep learning and knowledge graph technology for comprehensive knowledge reasoning, comprehensive and accurate modeling and recognition of complex entity images are achieved.

[0074] It should be noted that the entity image recognition method based on knowledge reasoning provided in the embodiment of the present application can be executed by an entity image recognition device based on knowledge reasoning, or a control module in the entity image recognition device based on knowledge reasoning for executing the entity image recognition method based on knowledge reasoning. In the embodiment of the present application, an entity image recognition device based on knowledge reasoning executing the entity image recognition method based on knowledge reasoning is taken as an example to illustrate the entity image recognition device based on knowledge reasoning provided in the embodiment of the present application.

[0075] It should be noted that, in the embodiments of the present application, the entity image recognition methods based on knowledge reasoning shown in the above-mentioned method drawings are all illustrated by combining one of the drawings in the embodiments of the present application as an example. In specific implementation, the entity image recognition methods based on knowledge reasoning shown in the above-mentioned method drawings can also be implemented in combination with any other drawings that can be combined as shown in the above-mentioned embodiments, which will not be repeated here.

[0076] The following is a description of the entity image recognition device based on knowledge reasoning provided by the present application. The entity image recognition method based on knowledge reasoning described below and described above can refer to each other.

[0077] Figure 3 A schematic diagram of the structure of an entity image recognition device based on knowledge reasoning provided in an embodiment of the present application, such as Figure 3 As shown, specifically including: The data processing module 301 is used to input the target text and the named entity recognition result of the target text into the language model to obtain multiple entity mentions corresponding to each entity and the vector representation of the text block to which each entity belongs; the named entity recognition result includes: multiple entities extracted based on the target text; the data processing module 301 is also used to process the multiple entity mentions corresponding to each entity based on the pooling layer and the knowledge graph to obtain a first optimization result of the entity representation of each entity, and to process the entity representation of each entity and the vector representation of the text block to which it belongs based on the attention module and the sentiment prediction module to obtain a second optimization result of the entity representation of each entity; the optimization module 302 is used to fuse the first optimization result and the second optimization result to obtain the optimized second entity representation of each entity.

[0078] Optionally, the data processing module 301 is specifically used to input multiple entity mentions corresponding to each entity into the pooling layer corresponding to the entity to perform a pooling operation to obtain a first entity representation of each entity; one entity corresponds to one pooling layer; the data processing module 301 is also specifically used to determine at least one associated entity with which each entity has an associated relationship based on the triples of the knowledge graph, and optimize the first entity representation of each entity based on the relationship vector between each entity and at least one associated entity to obtain the first optimization result.

[0079] Optionally, the device also includes: an acquisition module and a training module; the acquisition module is used to acquire any target triple from the multiple triples contained in the knowledge graph, and construct a fictitious triple corresponding to the target triple based on the two entities contained in the target triple; the fictitious triple is a negative sample of the target triple; the training module is used to calculate the first distance of the target triple using a distance function, and calculate the second distance of the fictitious triple using a distance function, and calculate the loss function value based on the first distance and the second distance; the training module is also used to adjust the relationship vector in the target triple based on the loss function value; wherein the relationship vector is used to represent the association relationship between the two entities contained in the target triple.

[0080] Optionally, the data processing module 301 is specifically used to input the vector representation of the text block to which each entity belongs and the first entity representation of each entity into the attention module corresponding to the text block to which the entity belongs, so as to obtain the text representation of each text block; the target text is divided into multiple text blocks, each text block contains multiple entities, and each text block corresponds to an attention module; the data processing module 301 is also specifically used to input the text representation of each text block into the emotion prediction module corresponding to the text block to perform emotion prediction, so as to obtain the second optimization result.

[0081] Optionally, the data processing module 301 is specifically used to perform a dot product calculation on the entity identifier corresponding to any target entity and the latent vector representation of each word in the vector representation of the text block to which the target entity belongs, so as to obtain an attention score corresponding to each word; the data processing module 301 is also specifically used to normalize the attention score corresponding to each word to obtain an attention weight for each word, and to perform a weighted average on the latent vector representation of each word based on the attention weight of each word to obtain a text representation of the text block to which the target entity belongs.

[0082] Optionally, the training module is also used to use a normalized exponential function to calculate the text representation of the text block input into the sentiment prediction module to obtain a first image probability distribution corresponding to the text block; the training module is also used to calculate a loss function value based on the first image probability distribution and a second image probability distribution corresponding to the entity identified from the text block, and optimize the sentiment prediction module by minimizing the loss function value.

[0083] Optionally, the optimization module 302 is specifically used to perform a fusion calculation based on a first optimization result corresponding to any target entity among the multiple entities and a second optimization result corresponding to a text block to which the target entity belongs, so as to obtain an optimized entity representation of the target entity.

[0084] The entity image recognition device based on knowledge reasoning provided by the present application first inputs the target text and the named entity recognition result of the target text into the language model to obtain multiple entity mentions corresponding to each entity; the named entity recognition result includes: multiple entities extracted based on the target text; then, based on the pooling layer and the knowledge graph, the multiple entity mentions corresponding to each entity are processed to obtain the first optimization result of the entity representation of each entity, and based on the attention module and the sentiment prediction module, the entity representation of each entity and the vector representation of the text block to which it belongs are processed to obtain the second optimization result of the entity representation of each entity; finally, the first optimization result and the second optimization result are fused and calculated to obtain the optimized second entity representation of each entity. In this way, by constructing an entity image knowledge graph and combining deep learning and knowledge graph technology for comprehensive knowledge reasoning, comprehensive and accurate modeling and recognition of complex entity images are achieved.

[0085] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the entity image recognition method based on knowledge reasoning, the method comprising: inputting the target text and the named entity recognition result of the target text into the language model, obtaining multiple entity mentions corresponding to each entity and the vector representation of the text block to which each entity belongs; the named entity recognition result comprises: multiple entities extracted based on the target text; processing the multiple entity mentions corresponding to each entity based on the pooling layer and the knowledge graph to obtain the first optimization result of the entity representation of each entity, and processing the entity representation of each entity and the vector representation of the text block to which it belongs based on the attention module and the sentiment prediction module to obtain the second optimization result of the entity representation of each entity; performing fusion calculation on the first optimization result and the second optimization result to obtain the optimized second entity representation of each entity.

[0086] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0087] On the other hand, the present application also provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the entity image recognition method based on knowledge reasoning provided by the above-mentioned methods, and the method includes: inputting the target text and the named entity recognition result of the target text into a language model to obtain multiple entity mentions corresponding to each entity and a vector representation of the text block to which each entity belongs; the named entity recognition result includes: multiple entities extracted based on the target text; based on the pooling layer and the knowledge graph, the multiple entity mentions corresponding to each entity are processed to obtain a first optimization result of the entity representation of each entity, and based on the attention module and the sentiment prediction module, the entity representation of each entity and the vector representation of the text block to which it belongs are processed to obtain a second optimization result of the entity representation of each entity; the first optimization result and the second optimization result are fused and calculated to obtain the optimized second entity representation of each entity.

[0088] On the other hand, the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the above-mentioned entity image recognition method based on knowledge reasoning, the method comprising: inputting a target text and a named entity recognition result of the target text into a language model to obtain multiple entity mentions corresponding to each entity and a vector representation of a text block to which each entity belongs; the named entity recognition result comprises: multiple entities extracted based on the target text; processing the multiple entity mentions corresponding to each entity based on a pooling layer and a knowledge graph to obtain a first optimization result of the entity representation of each entity, and processing the entity representation of each entity and the vector representation of the text block to which it belongs based on an attention module and a sentiment prediction module to obtain a second optimization result of the entity representation of each entity; performing a fusion calculation on the first optimization result and the second optimization result to obtain an optimized second entity representation of each entity.

[0089] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0090] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for entity image recognition based on knowledge reasoning, characterized in that: include: Inputting a target text and a named entity recognition result of the target text into a language model to obtain a plurality of entity mentions corresponding to each entity and a vector representation of a text block to which each entity belongs; The named entity recognition result includes: a plurality of entities extracted based on the target text; Based on the pooling layer and the knowledge graph, multiple entity mentions corresponding to each entity are processed to obtain a first optimization result of the entity representation of each entity, and based on the attention module and the sentiment prediction module, the entity representation of each entity and the vector representation of the text block to which it belongs are processed to obtain a second optimization result of the entity representation of each entity; The first optimization result and the second optimization result are fused and calculated to obtain a second entity representation after optimization of each entity.

2. The method according to claim 1, characterized in that The method processes multiple entity mentions corresponding to each entity based on the pooling layer and the knowledge graph to obtain a first optimization result of the entity representation of each entity, including: Input multiple entity mentions corresponding to each entity into the pooling layer corresponding to the entity to perform pooling operation to obtain the first entity representation of each entity; one entity corresponds to one pooling layer; At least one associated entity with which each entity has an associated relationship is determined based on the triples of the knowledge graph, and the first entity representation of each entity is optimized based on the relationship vector between each entity and at least one associated entity to obtain the first optimization result.

3. The method according to claim 2, characterized in that The relationship vectors contained in the triples in the knowledge graph are optimized by the following steps: Obtain any target triple from among the multiple triples contained in the knowledge graph, and construct a fictitious triple corresponding to the target triple based on two entities contained in the target triple; the fictitious triple is a negative sample of the target triple; Calculating a first distance of the target triplet using a distance function, calculating a second distance of the imaginary triplet using a distance function, and calculating a loss function value based on the first distance and the second distance; Adjusting the relationship vector in the target triplet based on the loss function value; The relationship vector is used to represent the association relationship between two entities included in the target triple.

4. The method according to claim 1, characterized in that: The attention module and the sentiment prediction module are used to process the entity representation of each entity and the vector representation of the text block to which it belongs, to obtain a second optimization result of the entity representation of each entity, including: Inputting the vector representation of the text block to which each entity belongs and the first entity representation of each entity into the attention module corresponding to the text block to which the entity belongs, to obtain the text representation of each text block; the target text is divided into a plurality of text blocks, each text block contains a plurality of entities, and each text block corresponds to an attention module; The text representation of each text block is input into the sentiment prediction module corresponding to the text block to perform sentiment prediction, thereby obtaining the second optimization result.

5. The method according to claim 4, characterized in that The step of inputting the vector representation of the text block to which each entity belongs and the first entity representation of each entity into the attention module corresponding to the text block to which the entity belongs, and obtaining the text representation of each text block, comprises: Perform a dot product calculation on the entity identifier corresponding to any target entity and the latent vector representation of each word in the vector representation of the text block to which the target entity belongs, to obtain an attention score corresponding to each word; The attention score corresponding to each word is normalized to obtain the attention weight of each word, and the latent vector representation of each word is weighted averaged based on the attention weight of each word to obtain the text representation of the text block to which the target entity belongs.

6. The method according to claim 4 or 5, characterized in that: The emotion prediction module is trained by the following steps: Using a normalized exponential function to calculate the text representation of the text block input into the sentiment prediction module, to obtain a first image probability distribution corresponding to the text block; A loss function value is calculated based on the first image probability distribution and a second image probability distribution corresponding to the entity identified from the text block, and the sentiment prediction module is optimized by minimizing the loss function value.

7. The method according to claim 1, characterized in that The fusing calculation of the first optimization result and the second optimization result to obtain the optimized second entity representation of each entity includes: A fusion calculation is performed based on a first optimization result corresponding to any target entity among the multiple entities and a second optimization result corresponding to the text block to which the target entity belongs to obtain an optimized entity representation of the target entity.

8. An entity image recognition device based on knowledge reasoning, characterized in that: The device comprises: A data processing module, used for inputting a target text and a named entity recognition result of the target text into a language model to obtain a plurality of entity mentions corresponding to each entity and a vector representation of a text block to which each entity belongs; the named entity recognition result includes: a plurality of entities extracted based on the target text; The data processing module is further used to process multiple entity mentions corresponding to each entity based on the pooling layer and the knowledge graph to obtain a first optimization result of the entity representation of each entity, and to process the entity representation of each entity and the vector representation of the text block to which it belongs based on the attention module and the sentiment prediction module to obtain a second optimization result of the entity representation of each entity; The optimization module is used to perform a fusion calculation on the first optimization result and the second optimization result to obtain a second entity representation after optimization of each entity.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the entity image recognition method based on knowledge reasoning as claimed in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the entity image recognition method based on knowledge reasoning as described in any one of claims 1 to 7 are implemented.

Images