Entity Infographic Generation Method, Apparatus, Equipment and Medium

Abstract

This invention relates to the field of information extraction, disclosing a method, apparatus, device, and medium for generating entity information graphs, comprising: acquiring a text corpus and an image corpus; selecting text entity reference itemsets from the text corpus, performing entity disambiguation on the text entity reference itemsets to obtain a disambiguated entity set, and extracting key components from the disambiguated entity set to obtain a target text entity set of the text corpus; acquiring text entity relationships between the target text entity sets, and generating a text information graph based on the text entity relationships and the target text entity sets; extracting visual features from the image corpus to obtain an image feature set, and extracting entity features from the image feature set to obtain an image entity set of the image corpus; and fusing the image entity set with the text information graph to obtain a target information graph containing image information. This invention can improve the accuracy of entity information graph generation.

Description

Technical Field

[0001] This invention relates to the field of information extraction, and in particular to a method, apparatus, device and storage medium for generating entity information graphs. Background Technology

[0002] With the rapid development of entity graph generation technology, information retrieval based on entity graphs has been widely used in many fields and positions due to its accurate and efficient retrieval capabilities. For example, when a company needs to acquire new customers, it can build an information graph of relevant employees of the partner company to understand in advance which department and which employee of the partner company it will be negotiating with. This makes it easier to understand the partner company's needs in advance and thus achieve cooperation.

[0003] Traditional entity graph generation methods rely on supervised deep learning to extract implicit semantic relationships between different entities from unstructured text, thereby constructing an entity graph. However, this method suffers from several drawbacks. First, it extracts many irrelevant words during the entity relationship extraction process, resulting in low accuracy. Second, it only generates entity graphs from text files, ignoring image information associated with the text, leading to low accuracy in the generated entity graphs. Summary of the Invention

[0004] This invention provides a method, apparatus, device, and storage medium for generating entity information maps, the main purpose of which is to improve the accuracy of entity information map generation.

[0005] To achieve the above objectives, the present invention provides a method for generating entity information graphs, comprising:

[0006] Acquire text and image corpora;

[0007] Select a set of text entity references from the text corpus, perform entity disambiguation on the set of text entity references to obtain a disambiguated entity set, and extract key components from the disambiguated entity set to obtain the target text entity set of the text corpus.

[0008] Obtain the text entity relationships between the target text entity sets, and generate a text information graph based on the text entity relationships and the target text entity sets;

[0009] Visual features are extracted from the image corpus to obtain an image feature set, and entity features are extracted from the image feature set to obtain an image entity set of the image corpus;

[0010] The image entity set is fused with the text information graph to obtain a target information graph containing image information.

[0011] Optionally, fusing the image entity set with the text information graph to obtain a target information graph containing image information includes:

[0012] The relationship between text entities and corresponding image entities in the text information graph is located to obtain the text-image entity relationship;

[0013] By using preset relationship rules, the relationships between different image entities in the image entity set are identified, and the entity relationships between images are obtained;

[0014] Based on the relationship between text and image entities and the relationship between image entities, the text entities and the image entities are linked to obtain the target information map.

[0015] Optionally, the step of linking the text entities and image entities based on the graph-text entity relationship and the image entity relationship to obtain the target information graph includes:

[0016] Based on the entity relationships between the text and images, an initial information graph is constructed with entities as nodes and entity relationships as edges, and an information dual graph is constructed with entity relationships as nodes and entities as edges.

[0017] All triplet structures are scored based on the initial information graph and the information dual graph to obtain the triplet score for each triplet structure;

[0018] The text entities and image entities are relinked based on the triplet scores to obtain the target information map.

[0019] Optionally, the step of disambiguating the text entity reference set to obtain a disambiguated entity set includes:

[0020] Select a candidate text entity set corresponding to the text entity reference set, and identify the degree of entity influence of each candidate entity in the corresponding text entity reference in the candidate entity set.

[0021] Candidate entities with an importance greater than a preset value are selected from the entity influence levels as the target candidate entity set corresponding to the text entity reference item set, and the remaining candidate entity sets corresponding to the text entity reference item set are removed to obtain the disambiguation entity set.

[0022] Optionally, the step of extracting entity features from the image feature set to obtain the image entity set of the image corpus includes:

[0023] Determine whether the image features in the image feature set contain text content;

[0024] When image features contain text content, the text content is extracted and used as the image label of the image corresponding to the image feature. The image label is named the image entity set.

[0025] When the image features do not contain text content, an image label is assigned to the image corresponding to the image features based on the image features, and the image label is named an image entity set.

[0026] Optionally, generating a text information graph based on the text entity relationships and the target text entity set includes:

[0027] Knowledge is extracted from the target text entity set and the text entity relationship to obtain a text triplet structure;

[0028] The text triple structure is fused with a preset knowledge base to obtain a text information graph.

[0029] Optionally, the step of extracting visual features from the image corpus to obtain an image feature set includes:

[0030] The image corpus is convolved using the convolutional layers in a pre-trained visual feature extraction network to obtain an image feature vector set.

[0031] The image feature set is obtained by performing max pooling on the image feature vector set using the fully connected layer in the visual feature extraction network.

[0032] To address the above problems, the present invention also provides an entity information map generation apparatus, the apparatus comprising:

[0033] The acquisition module is used to acquire text corpora and image corpora.

[0034] The text entity extraction module is used to select a set of text entity reference items from the text corpus, perform entity disambiguation on the set of text entity reference items to obtain a disambiguated entity set, and extract key components from the disambiguated entity set to obtain the target text entity set of the text corpus.

[0035] The text graph generation module is used to obtain the text entity relationships between the target text entity sets and generate a text information graph based on the text entity relationships and the target text entity sets.

[0036] The image entity extraction module is used to extract visual features from the image corpus to obtain an image feature set, and to extract entity features from the image feature set to obtain an image entity set of the image corpus.

[0037] The target information graph generation module is used to fuse the image entity set with the text information graph to obtain a target information graph containing image information.

[0038] To address the above problems, the present invention also provides an electronic device, the electronic device comprising:

[0039] Memory, storing at least one computer program; and

[0040] The processor executes the computer program stored in the memory to implement the entity information map generation method described above.

[0041] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the entity information map generation method described above.

[0042] In this embodiment of the invention, firstly, by selecting text entity reference itemsets from a text corpus and performing entity disambiguation on these itemsets, the true entities corresponding to the reference itemsets can be identified, and entity noise unrelated to the true entities can be removed. Furthermore, key components are extracted from the disambiguated entity set, removing some irrelevant entity components and improving the accuracy of subsequent text infographic generation. Secondly, a text infographic is generated based on text entity relationships and a target text entity set, which improves the accuracy of text infographic generation and facilitates subsequent improvements in entity infographic generation accuracy. Finally, by fusing image information with the text infographic, a target infographic containing image information is obtained. This allows for the addition of corresponding image information to text entities in the text infographic based on existing text, achieving the fusion of image information associated with text information into the infographic and improving the accuracy of entity infographic generation. Therefore, the entity infographic generation method, apparatus, device, and storage medium proposed in this embodiment of the invention can improve the accuracy of entity infographic generation. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating an entity information map generation method according to an embodiment of the present invention.

[0044] Figure 2 This is a detailed flowchart illustrating one step of an entity information map generation method according to an embodiment of the present invention.

[0045] Figure 3 This is a detailed flowchart illustrating one step of an entity information map generation method according to an embodiment of the present invention.

[0046] Figure 4This is a schematic diagram of a module of an entity information map generation device provided in an embodiment of the present invention;

[0047] Figure 5 This is a schematic diagram of the internal structure of an electronic device that implements the entity information map generation method according to an embodiment of the present invention.

[0048] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0049] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0050] This invention provides a method for generating an entity information graph. The execution entity of the entity information graph generation method includes, but is not limited to, at least one of the following: a server, a terminal, or other electronic devices that can be configured to execute the method provided in this application embodiment. In other words, the entity information graph generation method can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster.

[0051] Reference Figure 1 The flowchart shown is a schematic diagram of an entity information graph generation method provided in an embodiment of the present invention. In this embodiment, the entity information graph generation method includes the following steps:

[0052] S1. Obtain the text corpus and image corpus.

[0053] In this embodiment of the invention, the specific content of the text corpus and image corpus can be determined based on actual business scenarios. For example, the text corpus may include the business information of various enterprises and the corresponding employee information for each business; the image corpus may include images of employees of various enterprises.

[0054] In one embodiment of the present invention, the text corpus and image corpus can be obtained by crawling social media platforms (such as LinkedIn, Twitter, and Qichacha).

[0055] S2. Select a set of text entity reference items from the text corpus, perform entity disambiguation on the set of text entity reference items to obtain a disambiguated entity set, and extract key components from the disambiguated entity set to obtain the target text entity set of the text corpus.

[0056] In this embodiment of the invention, the text entity referential set refers to the set of all text entities in the text corpus. This is mainly manifested in the fact that the same text entity may have different entity referentials in the text corpus. For example, in company X, employee A has the positions of deputy manager and group leader of group B. Therefore, employee A's entity referentials can be deputy manager and group leader of group B. It can also be manifested in the fact that the same entity referential can refer to different entities in different contexts. For example, since there are three deputy managers, the deputy manager can refer to employee A or the other two employees, etc. The disambiguation entity set refers to the set of text entities after entity disambiguation; the target text entity set refers to the set of all text entities after removing text noise.

[0057] This invention disambiguates text entity reference sets to identify the true entities corresponding to the reference sets, removes entity noise unrelated to the true entities, and extracts key components from the disambiguated entity set to further remove useless entities, thereby improving the accuracy of subsequent text infographic generation.

[0058] As an embodiment of the present invention, reference Figure 2 As shown, the entity disambiguation of the text entity reference itemset to obtain the disambiguated entity set includes:

[0059] S21. Select the candidate text entity set corresponding to the text entity reference item set, and identify the degree of entity influence of each candidate entity in the corresponding text entity reference item in the candidate entity set.

[0060] S22. Select candidate entities with an importance greater than a preset value from the entity influence degree as the target candidate entity set corresponding to the text entity reference item set, and remove the remaining candidate entity sets corresponding to the text entity reference item set to obtain the disambiguation entity set.

[0061] The candidate text entity set refers to the set of entities in the text corpus that have the same name as the entity reference. The candidate text entity set can be represented by calculating the similarity between the entity reference and the candidate text entity using a clustering algorithm, and the final candidate text entities are selected by ranking the similarity.

[0062] In one embodiment of the present invention, the degree of entity influence can be calculated by using the PageRank algorithm to calculate the proportion weight of each candidate text entity in the corresponding entity designation in the candidate text entity set, and the proportion weight represents the degree of entity influence of each candidate text entity; further, in this embodiment of the present invention, the preset value can be 0.7.

[0063] Furthermore, in this embodiment of the invention, the Hanlp tool can be used to identify the part-of-speech of the disambiguation entity set. Based on the part-of-speech, useless words (such as adjectives, adverbs and other modifiers) identified in the disambiguation entity set can be removed, thereby enabling the extraction of key components from the disambiguation entity set to obtain the target text entity set.

[0064] S3. Obtain the text entity relationships between the target text entity sets, and generate a text information graph based on the text entity relationships and the target text entity sets.

[0065] In this embodiment of the invention, the text entity relationship refers to the relationship between text entities. For example, in a company, employee B and employee C in the marketing department have a superior-subordinate relationship.

[0066] In this embodiment of the invention, the text infographic refers to a new text entity and its corresponding relationship obtained by linking the text entities in the text triple structure with text in a preset knowledge base. The text infographic can also be viewed as a knowledge base that enhances information retrieval functionality. For example, if the knowledge base contains an entity named "Employee A" and the text triple also contains an entity named "Employee A," knowledge fusion can be used to merge the triple with related content of the same entity "Employee A" in the knowledge base to obtain a text infographic related to the "Employee A" entity. The preset knowledge base can be a corporate relationship database storing business information of various enterprises and corresponding employee information for each business.

[0067] This invention generates a text information graph based on the text entity relationships and the target text entity set, thereby achieving accurate fusion of the text entity relationships and the target text entity set and improving the accuracy of text information graph generation.

[0068] As an embodiment of the present invention, the step of generating a text information graph based on the text entity relationship and the target text entity set includes:

[0069] Knowledge is extracted from the target text entity set and the text entity relationships to obtain a text triple structure; the text triple structure is then fused with a preset knowledge base to obtain a text information graph.

[0070] The knowledge extraction can be achieved by using a deep learning model (such as the BERT model) to transform the target text entities and text entity relationships into a multi-classification problem, so as to associate each target entity in the target text entity set with the corresponding text entity relationship, and output a text triplet structure of target text entity-text entity relationship-target text entity.

[0071] In one embodiment of the present invention, the similarity between the text entity of each triple in the text triple structure and the text entity contained in the knowledge base can be calculated by cosine similarity. When the similarity is greater than a preset threshold, the text entities in the text triple structure and the text entities in the knowledge base are merged to obtain a text information graph.

[0072] S4. Extract visual features from the image corpus to obtain an image feature set, and extract entity features from the image feature set to obtain an image entity set of the image corpus.

[0073] In this embodiment of the invention, the image feature set refers to the set of feature vectors describing the image corpus. The image entity set refers to the set of entities describing the image content; for example, if an image is an employee A's onboarding photo, then the corresponding image entity is employee A; or, if an image is a sales process framework diagram for product D, then the corresponding image entity is the sales process framework for product D.

[0074] This invention, through extracting visual features from the image corpus, can extract image features that accurately represent the image corpus. Furthermore, it can extract entity features from the image feature set, thereby enabling the identification of image labels from the image feature set and using the identified image labels as entity features to improve the accuracy of entity feature identification.

[0075] As an embodiment of the present invention, the step of extracting visual features from the image corpus to obtain an image feature set includes:

[0076] The image corpus is convolved using the convolutional layers in the pre-trained visual feature extraction network to obtain an image feature vector set; the image feature vector set is then max-pooled using the fully connected layers in the visual feature extraction network to obtain the image feature set.

[0077] The visual feature extraction network can be VGG (Visual Geometry Group Network)-16, where 16 means that the network contains 13 convolutional layers and 3 fully connected layers.

[0078] In one embodiment of the present invention, the convolutional layer uses stacked convolutional kernels of 3x3 per layer to convolve the image corpus. Every three consecutive stacked 3x3 convolutions are equivalent to a 7x7 convolution, which can expand the receptive field of the convolutional kernel to extract deeper and more detailed image features from the image corpus, thereby improving the accuracy of image feature extraction. Furthermore, by performing max pooling on the image feature set through a fully connected layer, necessary features in the image feature set can be retained while unnecessary image features are removed, reducing the amount of computation and improving the efficiency of feature extraction while ensuring the accuracy of image features.

[0079] Further, the step of extracting entity features from the image feature set to obtain the image entity set of the image corpus includes:

[0080] Determine whether there is text content in the image features in the image feature set; when there is text content in the image features, extract the text content and use the text content as the image label of the image corresponding to the image feature, and name the image label as an image entity set; when there is no text content in the image features, assign an image label to the image corresponding to the image feature according to the image features, and name the image label as an image entity set.

[0081] The text content refers to all text information included in the image features, and the presence of text content in the image features can be identified through optical character recognition technology (such as PaddleOCR technology).

[0082] Specifically, in this embodiment of the invention, PaddleOCR can be used to perform text detection and recognition on image features. During detection, PaddleOCR uses the EAST algorithm to detect whether there is text content in the image features. When text content is detected, the CRNN algorithm is used to predict the character label set of the image features and obtain all path probabilities of the character label set. The maximum path probability corresponding to each character label is found from multiple path probabilities, and each maximum path probability is merged to integrate the character label set and obtain the final text content. This improves the accuracy of text content recognition and facilitates the improvement of the accuracy of subsequent image entity recognition.

[0083] S5. The image entity set is fused with the text information graph to obtain a target information graph containing image information.

[0084] In this embodiment of the invention, the target information graph refers to a multimodal information graph that includes text entities, image entities, relationships between text entities, relationships between image entities, and relationships between text entities and image entities.

[0085] For example, given an image entity representing employee A's onboarding photo and text entities representing employee A's name, company, position, age, length of service, department, and responsibilities, constructing an information graph of employee A's target information can provide all information about employee A within the company. This allows for better preparation of negotiation information with employee A during business collaborations, enabling a better understanding of the partner's needs and increasing the probability of successful business partnerships.

[0086] This invention integrates the image entity set with the text information graph to obtain a target information graph containing image information. It can add corresponding image information to text entities in the text information graph based on existing text, thereby integrating image information associated with text information into the information graph and improving the accuracy of entity information graph generation.

[0087] As an embodiment of the present invention, reference Figure 3 As shown, the process of fusing the image entity set with the text information graph to obtain a target information graph containing image information includes:

[0088] S51. Locate the relationship between the text entities and the corresponding image entities in the text information graph to obtain the text-image entity relationship;

[0089] S52. Using preset relationship rules, identify the relationships between different image entities in the image entity set to obtain the entity relationships between images;

[0090] S53. Based on the relationship between the text and image entities and the relationship between the entities in the images, link the text entities and the image entities to obtain the target information map.

[0091] The relationship localization can be achieved by matching text entities with image entities to determine the relationship between them. For example, if there is an image entity called "Employee A's Image" and a text entity called "Employee A's Name", then by concatenating the features of "Employee A's Name" and "Employee A's Image", a fused feature of the text-image entity can be obtained. Then, by performing a fully connected operation on this fused feature, the image entity (Employee A's Image) can be matched with the text entity (such as "Employee A's Name") to obtain the image-text entity relationship between "Employee A's Image" and "Employee A's Name".

[0092] In one embodiment of the present invention, the preset relationship rules include sameness, inclusion, and correlation between two image entities. For example, if image entity E1 and image entity E2 have a "Same As" relationship, then E1 and E2 have the sameness relationship; if image entity E3 and image E4 have a "Contain" relationship, then E3 and E4 have the inclusion relationship; if image entity E5 and image E6 have a "Relate To" relationship, then E5 and E6 have the correlation relationship. Through the above three relationship rules, the entity relationships between all image entities in the image corpus can be summarized.

[0093] Further, the step of linking the text entities and image entities based on the graph-text entity relationship and the image entity relationship to obtain the target information graph includes:

[0094] Based on the entity relationships between the text and images, an initial information graph is constructed with entities as nodes and entity relationships as edges. A dual information graph is then constructed with entity relationships as nodes and entities as edges. All triplet structures are scored according to the initial and dual information graphs to obtain a triplet score for each structure. Based on the triplet scores, the text entities and image entities are relinked to obtain the target information graph.

[0095] Among them, a relation-aware dual-graph convolutional network can be used to construct an information dual graph with entity relations as nodes and entities as edges based on the initial information graph.

[0096] In one embodiment of the present invention, the relation score of each entity relation can be calculated using the following formula:

[0097]

[0098]

[0099] Where PR(r) represents the importance of entity relation r; entity relation v belongs to B. r B r L(v) represents the set of neighbor relations of entity relation r; L(v) represents the number of relations connected to entity relation v; Score represents the relation score of entity relation r; d h This represents the number of entity relations associated with entity h (the head entity in the triple) that forms a triple with entity relation r; d t This represents the number of entity relations associated with entity t (i.e., the tail entity in the triple) that form a triple with entity relation r.

[0100] Furthermore, in this embodiment of the invention, the triple score is compared with a preset threshold α, triples with a score > α are retained, and the associated text entities and image entities are linked based on the retained triples to obtain the target information map. This can eliminate redundant or invalid relationships between text entities and image entities, ensuring the accuracy of entity information map generation.

[0101] In this embodiment of the invention, firstly, by selecting text entity reference itemsets from a text corpus and performing entity disambiguation on these itemsets, the true entities corresponding to the reference itemsets can be identified, and entity noise unrelated to the true entities can be removed. Furthermore, key components are extracted from the disambiguated entity set, removing some irrelevant entity components and improving the accuracy of subsequent text infographic generation. Secondly, a text infographic is generated based on text entity relationships and a target text entity set, which improves the accuracy of text infographic generation and facilitates subsequent improvements in entity infographic generation accuracy. Finally, by fusing image information with the text infographic, a target infographic containing image information is obtained. This allows for the addition of corresponding image information to text entities in the text infographic based on existing text, achieving the fusion of image information associated with text information into the infographic and improving the accuracy of entity infographic generation. Therefore, the entity infographic generation method proposed in this embodiment of the invention can improve the accuracy of entity infographic generation.

[0102] like Figure 4 As shown, the entity information map generation device 100 of the present invention can be installed in an electronic device. Depending on the functions implemented, the entity information map generation device may include an acquisition module 101, a text entity extraction module 102, a text map generation module 103, an image entity extraction module 104, and a target map generation module 105. The modules described in this invention can also be referred to as units, which are a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and are stored in the memory of the electronic device.

[0103] In this embodiment, the functions of each module / unit are as follows:

[0104] In detail, the modules in the entity information map generation device 100 described in this embodiment of the invention employ the same methods as described above during use. Figures 1 to 3 The method used is the same as the entity information graph generation method described in the previous section and can produce the same technical effect, so it will not be repeated here.

[0105] like Figure 5 The diagram shown is a schematic representation of the structure of an electronic device that implements the entity information map generation method of the present invention.

[0106] The electronic device may include a processor 10, a memory 11, a communication bus 12 and a communication interface 13, and may also include a computer program, such as an entity information map generation program, stored in the memory 11 and capable of running on the processor 10.

[0107] The memory 11 includes at least one type of medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, local disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. Furthermore, the memory 11 can include both internal and external storage units of the electronic device. The memory 11 can be used not only to store application software and various types of data installed on the electronic device, such as the code of an entity information map generation program, but also to temporarily store data that has been output or will be output.

[0108] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules (such as entity information map generation programs) stored in the memory 11, and calls data stored in the memory 11 to perform various functions of the electronic device and process data.

[0109] The communication bus 12 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The communication bus 12 is configured to enable communication between the memory 11 and at least one processor 10, etc. For ease of illustration, only one thick line is used in the figure, but this does not indicate that there is only one bus or one type of bus.

[0110] Figure 5 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 5The structure shown does not constitute a limitation on the electronic device and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0111] For example, although not shown, the electronic device may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0112] Optionally, the communication interface 13 may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device and other electronic devices.

[0113] Optionally, the communication interface 13 may further include a user interface, which may be a display, an input unit (such as a keyboard), or, optionally, a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.

[0114] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0115] The entity information map generation program stored in the memory 11 of the electronic device is a combination of multiple computer programs. When run in the processor 10, it can achieve the following:

[0116] Acquire text and image corpora;

[0117] Select a set of text entity references from the text corpus, perform entity disambiguation on the set of text entity references to obtain a disambiguated entity set, and extract key components from the disambiguated entity set to obtain the target text entity set of the text corpus.

[0118] Obtain the text entity relationships between the target text entity sets, and generate a text information graph based on the text entity relationships and the target text entity sets;

[0119] Visual features are extracted from the image corpus to obtain an image feature set, and entity features are extracted from the image feature set to obtain an image entity set of the image corpus;

[0120] The image entity set is fused with the text information graph to obtain a target information graph containing image information.

[0121] Specifically, the processor 10's implementation method of the above-mentioned computer program can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0122] Furthermore, if the modules / units integrated into the electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable medium. The computer-readable medium can be non-volatile or volatile. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0123] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following:

[0124] Acquire text and image corpora;

[0125] Select a set of text entity references from the text corpus, perform entity disambiguation on the set of text entity references to obtain a disambiguated entity set, and extract key components from the disambiguated entity set to obtain the target text entity set of the text corpus.

[0126] Obtain the text entity relationships between the target text entity sets, and generate a text information graph based on the text entity relationships and the target text entity sets;

[0127] Visual features are extracted from the image corpus to obtain an image feature set, and entity features are extracted from the image feature set to obtain an image entity set of the image corpus;

[0128] The image entity set is fused with the text information graph to obtain a target information graph containing image information. Further, the computer-readable storage medium may primarily include a program storage area and a data storage area, wherein the program storage area may store the operating system, at least one application program required for a function, etc.; the data storage area may store data created based on the use of blockchain nodes, etc.

[0129] In the several embodiments provided by this invention, it should be understood that the disclosed media, devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0130] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0131] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0132] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0133] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0134] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0135] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. The term "second class" is used to indicate names and does not indicate any specific order.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for generating entity information graphs, characterized in that, The method comprises: Acquire text and image corpora related to the target company; From the text corpus, select a set of text entity reference items, select a set of candidate text entities corresponding to the set of text entity reference items, identify the degree of entity influence of each candidate entity in the corresponding text entity reference item, select candidate entities with an importance greater than a preset value from the degree of entity influence as the target candidate entity set corresponding to the set of text entity reference items, remove the remaining candidate entity sets corresponding to the set of text entity reference items to obtain a disambiguation entity set, and extract key components from the disambiguation entity set to obtain the target text entity set of the text corpus; Obtain the text entity relationships between the target text entity sets, and generate a text information graph based on the text entity relationships and the target text entity sets; Visual features are extracted from the image corpus to obtain an image feature set. It is determined whether the image features in the image feature set contain text content. When the image features contain text content, a preset algorithm is used to predict the character tag set of the image features. All path probabilities of the character tag set are obtained. The maximum path probability corresponding to each character tag is found from each path probability. Each maximum path probability is merged to obtain the image tag of the image corresponding to the image feature and named it as an image entity set. When the image features do not contain text content, the image tag of the image corresponding to the image feature is assigned according to the image features and named as an image entity set. The relationship between text entities and corresponding image entities in the text information graph is located to obtain the text-image entity relationship. The relationship between different image entities in the image entity set is identified using preset relationship rules to obtain the inter-image entity relationship. Based on the text-image entity relationship and the inter-image entity relationship, the text entities and the image entities are linked to obtain the target information graph.

2. The entity information map generation method as described in claim 1, characterized in that, The step of linking text entities and image entities based on the graph-text entity relationship and the image entity relationship to obtain the target information graph includes: Based on the entity relationships between the text and images, an initial information graph is constructed with entities as nodes and entity relationships as edges, and an information dual graph is constructed with entity relationships as nodes and entities as edges. All triplet structures are scored based on the initial information graph and the information dual graph to obtain the triplet score for each triplet structure; The text entities and image entities are relinked based on the triplet scores to obtain the target information map.

3. The entity information map generation method as described in any one of claims 1-2, characterized in that, The process of generating a text information graph based on the text entity relationships and the target text entity set includes: Knowledge extraction is performed on the target text entity set and the text entity relationship to obtain a text triplet structure; The text triple structure is fused with a preset knowledge base to obtain a text information graph.

4. The entity information map generation method as described in any one of claims 1-2, characterized in that, The step of extracting visual features from the image corpus to obtain an image feature set includes: The image corpus is convolved using the convolutional layers in a pre-trained visual feature extraction network to obtain an image feature vector set. The image feature set is obtained by performing max pooling on the image feature vector set using the fully connected layer in the visual feature extraction network.

5. An entity information map generation device, characterized in that, The device comprises: The acquisition module is used to acquire text and image corpora related to the target company. The text entity extraction module is used to select a set of text entity reference items from the text corpus, select a set of candidate text entities corresponding to the set of text entity reference items, identify the degree of entity influence of each candidate entity in the corresponding text entity reference item, select candidate entities with an importance greater than a preset value from the degree of entity influence as the target candidate entity set corresponding to the set of text entity reference items, remove the remaining candidate entity sets corresponding to the set of text entity reference items to obtain a disambiguation entity set, and extract key components from the disambiguation entity set to obtain the target text entity set of the text corpus. The text graph generation module is used to obtain the text entity relationships between the target text entity sets and generate a text information graph based on the text entity relationships and the target text entity sets. The image entity extraction module is used to extract visual features from the image corpus to obtain an image feature set, determine whether the image features in the image feature set contain text content, and when the image features contain text content, use a preset algorithm to predict the character tag set of the image features, obtain all path probabilities of the character tag set, find the maximum path probability corresponding to each character tag from each path probability, merge each maximum path probability to obtain the image tag of the image corresponding to the image feature and name it an image entity set; when the image features do not contain text content, assign the image tag of the image corresponding to the image feature according to the image features and name it an image entity set. The target information graph generation module is used to locate the relationship between text entities and corresponding image entities in the text information graph to obtain the text-image entity relationship, identify the relationship between different image entities in the image entity set using preset relationship rules to obtain the image entity relationship, and link the text entities and image entities based on the text-image entity relationship and the image entity relationship to obtain the target information graph.

6. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the entity information map generation method as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the entity information map generation method as described in any one of claims 1 to 4.

Images