Data processing method, device and equipment
The triple sub-graph and attribute structure diagram are adjusted through the multimodal optimization model, which solves the problem of inaccurate triple extraction in the existing technology, and achieves a higher quality and complete triple extraction effect.
Patent Information
- Application Number
- CN202411366361.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-27
- Publication Date
- 2025-02-18
AI Technical Summary
Existing triple extraction methods are usually predicted in units of single triples, failing to adequately consider the relationship between multiple triples in paragraphs or articles, resulting in the extracted triples being inaccurate enough.
By inputting a multimodal optimization model based on the triple subgraph and attribute structure diagram of the pending text, an optimized triple set is generated. The method includes text encoding, visual feature extraction, and multimodal alignment to adjust the accuracy of the triple.
It improves the accuracy and completeness of triple extraction, reduces unreasonable triple groups, and supplements the implicit relationship between entities, making the final triple quality higher.
Smart Images
Figure CN119476285A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of data processing, and in particular to a data processing method, device and equipment. Background Art
[0002] Triple extraction is an important step in automatically building a knowledge graph. Current information extraction methods are generally divided into extractive and generative methods. Most existing methods use a single triple as a unit. Each triple includes a predicted head entity and a tail entity, and the relationship between entities is judged one by one. The predicted relationship is predicted by extracting the head and tail entities of the corresponding relationship. However, there are usually multiple triplets in a paragraph or article. When predicting a single triple, the relationship between this triple and other triplets in the paragraph is often not taken into account, which will lead to inaccurate triples in the paragraph or article. Summary of the invention
[0003] The present disclosure provides a data processing method, device and equipment to at least solve the above technical problems existing in the prior art.
[0004] According to a first aspect of the present disclosure, a data processing method is provided, the method comprising:
[0005] Determine an initial triple set based on the triples corresponding to the text to be processed; the triples are used to characterize the relationship between at least two entities in the text to be processed;
[0006] Generate a triple subgraph with entities in the initial triple set as nodes and the relationship between two entities as edges;
[0007] The text to be processed, the triple subgraph and the attribute structure graph generated based on the preset knowledge base are input into the multimodal optimization model to adjust the entities and the relationships between entities in the initial triple set to obtain an optimized triple set; wherein the attribute structure graph represents the first attribute type of each entity in the triple set and the second attribute type of the relationship between entities.
[0008] In one possible implementation, the step of inputting the to-be-processed text, the triple subgraph, and the attribute structure graph generated based on the preset knowledge base into a multimodal optimization model to adjust the entities and the relationships between the entities in the initial triple set includes:
[0009] Performing text encoding on the text to be processed to obtain a first feature code;
[0010] Extracting visual features of the triplet sub-graph using an encoder to obtain a first visual feature;
[0011] Extracting visual features of the attribute structure graph generated based on the preset knowledge base using an encoder to obtain a second visual feature;
[0012] Determining a fourth feature code characterizing the set of optimized triples according to the first feature code, the first visual feature, the second visual feature and the multimodal optimization model;
[0013] The fourth characteristic code is decoded to obtain the optimized triple set.
[0014] In one possible implementation, determining a fourth feature code characterizing the set of optimized triples according to the first feature code, the first visual feature, the second visual feature and the multimodal optimization model includes:
[0015] The first visual feature and the second visual feature are mapped to the text space encoded by the first feature to perform a multimodal alignment operation, and the entities and the relationships between the entities in the initial triple set are adjusted according to the alignment result.
[0016] In one possible implementation, adjusting the entities and the relationships between the entities in the initial triple set according to the alignment result includes:
[0017] Detecting whether the entities or relationships between entities in the text to be processed conform to the attribute structure diagram generated based on the preset knowledge base;
[0018] If it does not match, the entities and the relationships between the entities in the initial triple set are adjusted to output an optimized triple set; if it matches, the optimized triple set is directly output.
[0019] In one possible implementation, the detecting whether the entities or the relationships between entities in the text to be processed conform to the attribute structure diagram generated based on the preset knowledge base includes at least one of the following:
[0020] Detecting whether the first attribute type of the entity in the initial triple set conforms to the first attribute type of the corresponding entity in the preset knowledge base represented by the attribute structure graph corresponding to the attribute structure graph generated based on the preset knowledge base;
[0021] It is detected whether the entity relationship in the initial triple combination conforms to the second attribute type of the corresponding entity relationship in the preset knowledge base represented by the attribute structure graph corresponding to the attribute structure graph generated based on the preset knowledge base.
[0022] In one possible implementation, the adjusting the entities and the relationships between the entities in the initial triple set includes at least one of the following adjustment methods:
[0023] If it is determined that a single triple in the initial triple set has a semantic error, deleting or adjusting the single triple;
[0024] If it is determined that there are multiple triples with conflicting relationships in the initial triple set, determine a triple to be retained from the multiple triples according to a predefined conflict resolution strategy, and delete the other triples in the multiple triples except the triple to be retained;
[0025] If it is determined that multiple triples in the initial triple set meet the similarity condition, merging the multiple triples into the same triple;
[0026] If it is determined that there are missing entity relationships in the initial triple set, convert the missing entity relationships into newly added triples, and add the newly added triples to the initial triple set;
[0027] If it is determined that there are triples in the initial triple set that do not conform to the attribute structure of the target domain corresponding to the to-be-processed text, the triples are deleted or adjusted.
[0028] In one possible implementation, the multimodal optimization model includes:
[0029] The triple optimization sub-model is used to adjust the entities and the relationships between entities in the initial triple set based on the text to be processed, the attribute structure graph generated based on the preset knowledge base, and the triple sub-graph to obtain an optimized triple set.
[0030] In one possible implementation, based on the text to be processed, the attribute structure graph generated based on the preset knowledge base, and the triple subgraph, the entities and the relationships between the entities in the initial triple set are adjusted to obtain an optimized triple set, including:
[0031] Performing text encoding on the text to be processed to obtain a first feature code;
[0032] Extracting visual features of the triplet sub-graph using an encoder to obtain a first visual feature;
[0033] Extracting visual features of the attribute structure graph generated based on the preset knowledge base using an encoder to obtain a second visual feature;
[0034] The first visual feature and the second visual feature are mapped to the text space encoded by the first feature to perform a multimodal alignment operation, and the entities and the relationships between the entities in the initial triple set are adjusted according to the alignment result.
[0035] According to a second aspect of the present disclosure, a data processing device is provided, the device comprising:
[0036] A determination module, used to determine an initial triple set based on triples corresponding to the text to be processed; the triples are used to characterize the relationship between at least two entities in the text to be processed;
[0037] A generating module, used for generating a triple subgraph with entities in the initial triple set as nodes and the relationship between two entities as edges;
[0038] An optimization module is used to input the text to be processed, the triple subgraph and the attribute structure graph generated based on the preset knowledge base into a multimodal optimization model to adjust the entities and the relationships between the entities in the initial triple set to obtain an optimized triple set; wherein the attribute structure graph represents the first attribute type of each entity in the triple set and the second attribute type of the relationship between the entities.
[0039] According to a third aspect of the present disclosure, there is provided an electronic device, including:
[0040] at least one processor; and
[0041] a memory communicatively connected to the at least one processor; wherein,
[0042] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the present disclosure.
[0043] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the present disclosure.
[0044] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:
[0046] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.
[0047] Figure 1 The implementation process of the data processing method of the present disclosure is shown in FIG. Figure 1 ;
[0048] Figure 2 A triplet subgraph of an embodiment of the present disclosure is shown;
[0049] Figure 3 The attribute structure diagram generated based on the preset knowledge base in the embodiment of the present disclosure is shown;
[0050] Figure 4 A schematic diagram showing the structure of a data processing device according to an embodiment of the present disclosure is shown;
[0051] Figure 5 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0052] In order to make the purpose, features, and advantages of the present disclosure more obvious and easy to understand, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.
[0053] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.
[0054] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0055] The data processing method, device and equipment provided by the present application are described below in conjunction with the accompanying drawings.
[0056] like Figure 1 As shown, the present application provides a data processing method, the method comprising:
[0057] S101, determining an initial triple set based on triples corresponding to the text to be processed; the triples are used to represent the relationship between at least two entities in the text to be processed;
[0058] It is understandable that in this application, the initial triple set is obtained through the entities in the text to be processed and the relationship between the entities. The entities in this application can be objective things, for example, they can be people, organizations, movies, sentences, etc.; the relationship between entities refers to the relationship between entities, including the relationship between people, the relationship between subject, predicate and object, the belonging relationship between objects and categories, etc. A triple is composed of two entities and the relationship between the two entities, indicating that the first entity has a first relationship with the second entity.
[0059] In this application, the text to be processed is a paragraph or article, which consists of multiple sentences, and has multiple triples. All triples extracted from the text to be processed constitute the initial triple set. For example, the initial triple set is (first entity, first relationship, second entity), (third entity, second relationship, fourth entity), ... (Nth entity, Mth relationship, N+1th entity).
[0060] S102, generating a triple subgraph with entities in the initial triple set as nodes and the relationship between two entities as edges;
[0061] It is understandable that in this application, entities are used as nodes and the relationships between entities are used as edges, thereby obtaining a triple subgraph. That is, the extracted triple set is converted into a graph structure, and a graph layout algorithm is used in this application, such as a force-directed layout algorithm, to convert the extracted triple set into a visual image.
[0062] For example, the exemplary text is "According to news reports, in 2005, the husband of long-distance runner Sun, Wang, was hired as a coach by Institution 1. The two athletes he coached won 3 gold medals and 1 silver medal at the Paralympics. In October 2011, Institution 1 rewarded Wang with RMB X million. After learning the news, his unit, Institution 2, asked him to hand over the bonus to be distributed by the organization, but Wang disagreed. On December 28, 2011, Institution 2 suspended Wang from his job." The triples extracted from it are as follows:
[0063] Table 1 Triples
[0064] Head Entity relation Tail Entity Wang husband Sun Wang Working on Institution 1 Wang Working on Institution 2 Institution 1 Give Rewards Wang Institution 1 Give Rewards X million yuan
[0065] The entities in Table 1 are used as nodes and the relationships between entities are used as edges to generate Figure 2 The triplet subgraph shown.
[0066] S103, inputting the text to be processed, the triple subgraph and the attribute structure graph generated based on the preset knowledge base into the multimodal optimization model to adjust the entities and the relationships between the entities in the initial triple set to obtain an optimized triple set; wherein the attribute structure graph represents the first attribute type of each entity in the triple set and the second attribute type of the relationship between the entities.
[0067] It is understandable that the present application pre-sets a text knowledge base and generates an attribute structure diagram based on the text knowledge base. It should be noted that the attribute structure diagram generated by the text knowledge base includes the attribute types that may be possessed by the text entities to be processed and the relationships between entities.
[0068] Exemplarily, the attribute type of an entity can be the entity type to which the entity belongs, such as a person, an organization, an item, etc.; it can also be the role type of the entity in a relationship, such as the head entity of a relationship or the tail entity of a relationship; it can also be the type of a certain attribute of an entity, such as the category of the attribute "gender" of the entity "Wang" is "male" or "female", etc. The attribute type of a relationship can be the relationship itself, such as "wife", "give reward", "the alias of..."; it can also be the type of a relationship, such as the relationship "give reward" belongs to "action relationship" and the relationship "wife" belongs to "person relationship"; it can also be the type of a relationship attribute, such as "directed relationship" and "undirected relationship" according to the directionality of the relationship, "the alias of..." belongs to "undirected relationship", and "wife" belongs to "directed relationship". The attribute type of an entity realizes the constraint relationship through the attribute type of a relationship, including constraining the attribute type of a relationship through the attribute type of an entity, and constraining the attribute type of an entity through the attribute type of a relationship, so as to ensure that the extracted triples are reasonable and accurate.
[0069] For example, when the attribute type of the relationship is "wife", the entity type of both entities must be "person"; further constraints can be added, where the gender of the head entity is "female". For example, when the entity type of both entities is "person", the relationship type can be "wife" and "the alias of... is...", but not "accepting prizes".
[0070] It should be noted that the attribute structure diagram in this application constrains the first attribute type of the entity through the second attribute type of the relationship, and supplements the implicit relationship that is difficult to analyze through semantic analysis. For example, when there is a relationship between the first entity and the second entity that the first entity is the wife of the second entity, it can be supplemented with the implicit relationship that the second entity is the husband of the first entity. The resulting attribute structure diagram is as follows Figure 3 shown.
[0071] It can be understood that multimodality or multimodal biometrics refers to the integration or fusion of two or more biometric technologies, using the unique advantages of multiple biometric technologies, and combining data fusion technology to make the authentication and identification process more accurate and secure. The main difference from the traditional single biometric method is that multimodal biometric technology can collect different biometric features (such as fingerprints, finger veins, faces, iris images, etc.) through independent or multiple collection methods. Collectors in one collector, and identify and authenticate by analyzing and judging the feature values of multiple biometric methods. In this application, a multimodal optimization model is used to receive and process the text to be processed, the triple subgraph and the attribute structure diagram, so as to adjust the entities and the relationship between entities in the initial triple set to obtain an optimized triple set.
[0072] It is understandable that, since the triples in the initial triple set are extracted from the same paragraph or the same article, they are often used to express the same topic with the same context and semantic scope, and they are mutually contextual, and often have order, cause and effect and other related relationships. Some triples have no semantic or logical problems when viewed alone, but when placed in the same context and semantic scope, conflicts or contradictions may arise. In this application, the entities and the relationships between entities in the initial triple set are adjusted, including deleting duplicate entities, deleting duplicate relationships, adding or updating entities and / or relationships, etc.
[0073] The data processing method provided by the present application determines the initial triple set, obtains the triple subgraph, and then uses the multimodal optimization model to align the triple subgraph, the attribute structure graph generated by the preset knowledge base, and the text to be processed to adjust the entities and the relationship between the entities in the initial triple set, thereby obtaining an optimized triple set. The present application can reduce unreasonable triples and supplement the implicit relationship between the entities of the triples, so that the quality of the final triples is higher.
[0074] In some optional embodiments, the step of inputting the to-be-processed text, the triple subgraph, and the attribute structure graph generated based on the preset knowledge base into the multimodal optimization model to adjust the entities and the relationships between the entities in the initial triple set includes:
[0075] Performing text encoding on the text to be processed to obtain a first feature code;
[0076] Extracting visual features of the triplet sub-graph using an encoder to obtain a first visual feature;
[0077] Extracting visual features of the attribute structure graph generated based on the preset knowledge base using an encoder to obtain a second visual feature;
[0078] Determining a fourth feature code characterizing the set of optimized triples according to the first feature code, the first visual feature, the second visual feature and the multimodal optimization model;
[0079] The fourth characteristic code is decoded to obtain the optimized triple set.
[0080] Through the multimodal optimization model, the present application can better understand and process the features of the text to be processed, the triple subgraph and the attribute structure graph, and align the first feature coding of the text to be processed by mapping the first visual feature and the second visual feature to the text space to obtain an optimized fourth feature coding, and obtain an optimized triple set after decoding. Through the alignment of the first feature coding, the first visual feature and the second visual feature, the present application can adjust the initial triple set graph, delete the edges representing the conflicting triples, and add the missing entity relationships newly extracted from the text to be processed according to the attribute structure graph, so that the final optimized triple set is more complete and accurate and avoids the appearance of multiple triples that may conflict.
[0081] In the present application, a text encoder may be used to encode the text to be processed. The text encoder is pre-trained for encoding. The pre-trained text encoder may use a spanBERT model.
[0082] In some optional embodiments, determining a fourth feature code characterizing the set of optimized triples according to the first feature code, the first visual feature, the second visual feature and the multimodal optimization model comprises:
[0083] The first visual feature and the second visual feature are mapped to the text space encoded by the first feature to perform a multimodal alignment operation, and the entities and the relationships between the entities in the initial triple set are adjusted according to the alignment result.
[0084] It can be understood that the multimodal feature alignment parameters are pre-determined in the multimodal optimization model in the present application, and the multimodal feature alignment parameters are used to perform multimodal feature alignment on the encoded first visual feature and the second visual feature with the first feature encoding in the text space, thereby obtaining the modal alignment features corresponding to the modal data. Among them, the multimodal feature alignment parameters in this embodiment are determined in advance by performing feature type discrimination training on the sample features corresponding to the sample data using generative adversarial technology.
[0085] In this embodiment, before determining the multimodal feature alignment parameters, it is necessary to determine the sample features of the sample to be processed, the first visual features of the triple subgraph, and the second visual features of the attribute structure graph. Then, the first visual features, the second visual features, and the first feature encoding are aligned using the current multimodal feature alignment parameters to obtain aligned modal alignment features. Since the alignment of the modal features by the multimodal feature alignment parameters is only the alignment of the feature representation text space, that is, the modal features are represented in the same way (that is, the representation method can have the ability to represent text-related contextual information), therefore, in this embodiment, the sample data for training the multimodal feature alignment parameters can use unpaired triple subgraphs and attribute structure graphs. In this embodiment, unpaired sample data, triple subgraphs, and attribute structure graphs can be used to train the multimodal feature alignment parameters, which can improve the training effect of the multimodal feature alignment parameters, thereby improving the alignment effect of the multimodal features.
[0086] In some optional embodiments, adjusting the entities and the relationships between the entities in the initial triple set according to the alignment result includes:
[0087] Detecting whether the entities or relationships between entities in the text to be processed conform to the attribute structure diagram generated based on the preset knowledge base;
[0088] If it does not match, the entities and the relationships between the entities in the initial triple set are adjusted to output an optimized triple set; if it matches, the optimized triple set is directly output.
[0089] It is understandable that the attribute structure diagram in this application is pre-set, which defines the entity type, relationship type, head entity type and tail entity type of each relationship in the text to be processed. Among them, the entity type is used as the first attribute type of the entity; the relationship is used as the second attribute type of the relationship.
[0090] For example, in this application, taking the exemplary text as an example, firstly, the initial triple set extracted from the above-mentioned text to be processed by semantic analysis is as follows: {(Wang, wife, Sun), (Wang, works at, institution 2), (institution 2, gives reward, Wang), (institution 2, gives prize, prize), (Wang, works at, institution 1). Then the above triple set is converted into Figure 2 The triple subgraph shown. When one of the triples is obtained, for example (Wang, wife, Sun), the triple is tested according to the attribute structure graph. Then, the entities and the relationships between the entities in the initial triple set are adjusted according to the test results, and the optimized triple set is output.
[0091] The detecting whether the entities or the relationships between entities in the to-be-processed text conform to the attribute structure diagram generated based on the preset knowledge base includes at least one of the following:
[0092] Detecting whether the first attribute type of the entity in the initial triple set conforms to the first attribute type of the corresponding entity in the preset knowledge base represented by the attribute structure graph corresponding to the attribute structure graph generated based on the preset knowledge base;
[0093] It is detected whether the entity relationship in the initial triple combination conforms to the second attribute type of the corresponding entity relationship in the preset knowledge base represented by the attribute structure graph corresponding to the attribute structure graph generated based on the preset knowledge base.
[0094] Specifically, when the present application detects a triplet, the detection includes:
[0095] Whether the entity type conforms to the entity type defined in the attribute structure diagram;
[0096] Whether the relationship between entities conforms to the relationship defined by the attribute schema;
[0097] Whether the head entity type of the relationship between entities conforms to the head entity type of the corresponding edge definition;
[0098] Whether the tail entity type of the relationship between entities matches the tail entity type of the corresponding edge definition.
[0099] In some optional embodiments, the adjusting the entities and the relationships between the entities in the initial triple set includes at least one of the following adjustment methods:
[0100] If it is determined that a single triple in the initial triple set has a semantic error, deleting or adjusting the single triple;
[0101] If it is determined that there are multiple triples with conflicting relationships in the initial triple set, determine a triple to be retained from the multiple triples according to a predefined conflict resolution strategy, and delete the other triples in the multiple triples except the triple to be retained;
[0102] If it is determined that multiple triples in the initial triple set meet the similarity condition, merging the multiple triples into the same triple;
[0103] If it is determined that there are missing entity relationships in the initial triple set, convert the missing entity relationships into newly added triples, and add the newly added triples to the initial triple set;
[0104] If it is determined that there are triples in the initial triple set that do not conform to the attribute structure of the target domain corresponding to the to-be-processed text, the triples are deleted or adjusted.
[0105] Specifically, in the present application, triples are adjusted according to specific circumstances of the existence of triples in the triple set.
[0106] For example, if the entity type in the triple does not conform to the entity type defined in the attribute schema, the triple is deleted; if the relationship in the triple does not conform to the relationship defined in the attribute schema, it is further determined whether there is a relationship defined in the attribute schema whose approximation to the corresponding relationship is higher than a threshold. If so, the corresponding relationship is replaced with the relationship with the highest approximation defined in the attribute schema; if the head entity type of the relationship in the triple conforms to the head entity type of the corresponding relationship, but the tail entity type of the relationship in the triple does not conform to the tail entity type of the corresponding relationship, the tail entity that conforms to the tail entity type of the corresponding relationship is re-extracted based on the text to be processed, etc.
[0107] For example, in the initial triple set, the following two triplets do not conform to the constraint relationship that when the relationship is "working at", the same head entity can only correspond to the same tail entity: (Wang, working at, organization 2) and (Wang, working at, organization 1). This application determines the triples to be retained from multiple triples according to a pre-defined conflict resolution strategy, and deletes other triples except the triples to be retained from the multiple triples; it is assumed that the pre-defined conflict resolution strategy in this embodiment is: in combination with the file to be processed, check whether the head entity has a "suspension" relationship with multiple conflicting "organization" type entities. According to the above conflict resolution strategy, this embodiment re-extracts the following triples from the file to be processed: (Wang, suspended, organization 2), adds the triple to the original initial triple set, and deletes the triple (Wang, working at, organization 2).
[0108] For example, if multiple triples in the initial triple set meet similar conditions, the multiple triples are merged into the same triple.
[0109] For example, according to the following triple relationships in the initial triple set: (Organization 1, Give Reward, Wang), (Organization 1, Give Prize, Prize 1), according to the attribute structure diagram, when a certain "work organization" type entity has a "give reward" relationship with a certain "person" type entity, and the same "organization" type entity has a "give prize" relationship with an "item" type entity, there is often an implicit "accept prize" relationship between a certain "person" type entity and an "item" type entity. It can be determined that the following triple relationship is missing in the initial triple set: (Wang, accept prize, prize 1). Convert the missing entity relationship into a new triple, and add the new triple to the initial triple set.
[0110] For example, the triples (organization 2, suspended, Wang) and (Wang, accepting prize, prize 1) are added, and the triple (Zhang, working at, organization 1) is deleted to obtain an optimized triple set. In this embodiment, the initial triple set graph is adjusted according to the attribute structure graph corresponding to the text to be processed, the edges representing the conflicting triples are deleted, and the missing entity relationships newly extracted from the text to be processed according to the attribute structure graph are added, so that the final optimized triple set is more complete and accurate and avoids the occurrence of multiple triples that may conflict.
[0111] In some optional embodiments, the multimodal optimization model includes:
[0112] The triple optimization sub-model is used to adjust the entities and the relationships between entities in the initial triple set based on the text to be processed, the attribute structure graph generated based on the preset knowledge base, and the triple sub-graph to obtain an optimized triple set.
[0113] Specifically, in the present application, a triple optimization sub-model is provided in the multimodal optimization model, and the triple optimization sub-model can adjust triples based on the attribute structure graph, the triple sub-graph and the text to be processed, and finally obtain an optimized triple set.
[0114] In some optional embodiments, based on the text to be processed, the attribute structure graph generated based on the preset knowledge base, and the triple subgraph, the entities and the relationships between the entities in the initial triple set are adjusted to obtain an optimized triple set, including:
[0115] Performing text encoding on the text to be processed to obtain a first feature code;
[0116] Extracting visual features of the triplet sub-graph using an encoder to obtain a first visual feature;
[0117] Extracting visual features of the attribute structure graph generated based on the preset knowledge base using an encoder to obtain a second visual feature;
[0118] The first visual feature and the second visual feature are mapped to the text space encoded by the first feature to perform a multimodal alignment operation, and the entities and the relationships between the entities in the initial triple set are adjusted according to the alignment result.
[0119] Specifically, the triple optimization sub-model in the present application is to obtain a first feature code by performing text encoding on the processed text, extract the visual features of the triple sub-graph using the encoder to obtain a first visual feature, and extract the visual features of the attribute structure graph generated based on a preset knowledge base using the encoder to obtain a second visual feature; the first visual feature and the second visual feature are mapped to the text space of the first feature code for a multimodal alignment operation, and the entities and the relationships between the entities in the initial triple set are adjusted according to the alignment results.
[0120] like Figure 4 As shown, the present application provides a data processing device, the device comprising:
[0121] A determination module 401 is used to determine an initial triple set based on the triples corresponding to the text to be processed; the triples are used to characterize the relationship between at least two entities in the text to be processed;
[0122] A generating module 402, configured to generate a triple subgraph with entities in the initial triple set as nodes and the relationship between two entities as edges;
[0123] The optimization module 403 is used to input the text to be processed, the triple subgraph and the attribute structure diagram generated based on the preset knowledge base into the multimodal optimization model to adjust the entities and the relationships between the entities in the initial triple set to obtain an optimized triple set; wherein the attribute structure diagram represents the first attribute type of each entity in the triple set and the second attribute type of the relationship between the entities.
[0124] The data processing device provided by the present application determines an initial triple set based on the triples corresponding to the text to be processed by a determination module 401; the triples are used to represent the relationship between at least two entities in the text to be processed; the generation module 402 generates a triple subgraph using the entities in the initial triple set as nodes and the relationship between the two entities as edges; the optimization module 403 inputs the text to be processed, the triple subgraph and the attribute structure graph generated based on the preset knowledge base into a multimodal optimization model to adjust the entities and the relationships between the entities in the initial triple set to obtain an optimized triple set; wherein the attribute structure graph represents the first attribute type of each entity in the triple set and the second attribute type of the relationship between the entities.
[0125] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.
[0126] Figure 5A schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0127] like Figure 5 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.
[0128] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0129] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as data processing methods. For example, in some embodiments, the data processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the data processing method in any other appropriate manner (e.g., by means of firmware).
[0130] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0131] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0132] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0133] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0134] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0135] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0136] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0137] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0138] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.
Claims
1. A data processing method, the method comprising: Determine an initial triple set based on the triples corresponding to the text to be processed; The triple is used to represent the relationship between at least two entities in the text to be processed; Generate a triple subgraph with entities in the initial triple set as nodes and the relationship between two entities as edges; The text to be processed, the triple subgraph and the attribute structure graph generated based on the preset knowledge base are input into the multimodal optimization model to adjust the entities and the relationships between entities in the initial triple set to obtain an optimized triple set; wherein the attribute structure graph represents the first attribute type of each entity in the triple set and the second attribute type of the relationship between entities.
2. The method according to claim 1, wherein the text to be processed, the triple subgraph, and the attribute structure graph generated based on the preset knowledge base are input into the multimodal optimization model to adjust the entities and the relationships between the entities in the initial triple set, comprising: Performing text encoding on the text to be processed to obtain a first feature code; Extracting visual features of the triplet sub-graph using an encoder to obtain a first visual feature; Extracting visual features of the attribute structure graph generated based on the preset knowledge base using an encoder to obtain a second visual feature; Determining a fourth feature code characterizing the set of optimized triples according to the first feature code, the first visual feature, the second visual feature and the multimodal optimization model; The fourth characteristic code is decoded to obtain the optimized triple set.
3. The method according to claim 2, wherein determining a fourth feature code characterizing the set of optimized triples according to the first feature code, the first visual feature, the second visual feature and the multimodal optimization model comprises: The first visual feature and the second visual feature are mapped to the text space encoded by the first feature to perform a multimodal alignment operation, and the entities and the relationships between the entities in the initial triple set are adjusted according to the alignment result.
4. The method according to claim 3, wherein adjusting the entities and the relationships between the entities in the initial triple set according to the alignment result comprises: Detecting whether the entities or relationships between entities in the text to be processed conform to the attribute structure diagram generated based on the preset knowledge base; If it does not match, the entities and the relationships between the entities in the initial triple set are adjusted to output an optimized triple set; if it matches, the optimized triple set is directly output.
5. According to the method of claim 4, the detecting whether the entities or the relationships between entities in the to-be-processed text conform to the attribute structure diagram generated based on the preset knowledge base comprises at least one of the following: Detecting whether the first attribute type of the entity in the initial triple set conforms to the first attribute type of the corresponding entity in the preset knowledge base represented by the attribute structure graph corresponding to the attribute structure graph generated based on the preset knowledge base; It is detected whether the entity relationship in the initial triple combination conforms to the second attribute type of the corresponding entity relationship in the preset knowledge base represented by the attribute structure graph corresponding to the attribute structure graph generated based on the preset knowledge base.
6. The method according to claim 1, wherein the adjusting the entities and the relationships between the entities in the initial triple set comprises at least one of the following adjustment methods: If it is determined that a single triple in the initial triple set has a semantic error, deleting or adjusting the single triple; If it is determined that there are multiple triples with conflicting relationships in the initial triple set, determine a triple to be retained from the multiple triples according to a predefined conflict resolution strategy, and delete the other triples in the multiple triples except the triple to be retained; If it is determined that multiple triples in the initial triple set meet the similarity condition, merging the multiple triples into the same triple; If it is determined that there are missing entity relationships in the initial triple set, convert the missing entity relationships into newly added triples, and add the newly added triples to the initial triple set; If it is determined that there are triples in the initial triple set that do not conform to the attribute structure of the target domain corresponding to the to-be-processed text, the triples are deleted or adjusted.
7. The method according to claim 1, wherein the multimodal optimization model comprises: The triple optimization sub-model is used to adjust the entities and the relationships between entities in the initial triple set based on the text to be processed, the attribute structure graph generated based on the preset knowledge base, and the triple sub-graph to obtain an optimized triple set.
8. The method according to claim 7, adjusting the entities and the relationships between the entities in the initial triple set based on the text to be processed, the attribute structure graph generated based on the preset knowledge base, and the triple subgraph to obtain an optimized triple set, comprising: Performing text encoding on the text to be processed to obtain a first feature code; Extracting visual features of the triplet sub-graph using an encoder to obtain a first visual feature; Extracting visual features of the attribute structure graph generated based on the preset knowledge base using an encoder to obtain a second visual feature; The first visual feature and the second visual feature are mapped to the text space encoded by the first feature to perform a multimodal alignment operation, and the entities and the relationships between the entities in the initial triple set are adjusted according to the alignment result.
9. A data processing device, comprising: A determination module, used for determining an initial triple set based on the triples corresponding to the text to be processed; The triple is used to represent the relationship between at least two entities in the text to be processed; A generating module, used for generating a triple subgraph with entities in the initial triple set as nodes and the relationship between two entities as edges; An optimization module is used to input the text to be processed, the triple subgraph and the attribute structure graph generated based on the preset knowledge base into a multimodal optimization model to adjust the entities and the relationships between the entities in the initial triple set to obtain an optimized triple set; wherein the attribute structure graph represents the first attribute type of each entity in the triple set and the second attribute type of the relationship between the entities.
10. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.