Entity relationship prediction method and device, electronic equipment and storage medium
By fusing information from images and text in image-text pairs, and utilizing entity relationship categories in images and dependency relationships in text, the problem of insufficient accuracy in entity relationship prediction in existing technologies is solved, and more accurate entity relationship prediction is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING DAJIA INTERNET INFORMATION TECH CO LTD
- Filing Date
- 2023-02-06
- Publication Date
- 2026-05-15
AI Technical Summary
In existing technologies, entity relationship prediction methods rely solely on information about the entities themselves, resulting in insufficient accuracy in predicting relationships.
By acquiring the text representation and triple representation of the image-text pair, information fusion processing is performed, and the relationship category between the head entity and the tail entity is predicted by utilizing the entity relationship category in the image and the dependency relationship in the text.
It improves the accuracy of entity relationship prediction by fusing information from images and text, thus enhancing the prediction results for entity relationship categories.
Smart Images
Figure CN116628213B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computers, and more particularly to a method, apparatus, electronic device, and storage medium for predicting entity relationships. Background Technology
[0002] Multimodal relation extraction refers to using visual content in images to supplement missing information in text, aiming to identify the relationships between entities in a given image-text pair.
[0003] In related technologies, information about entities in an image and information about entities in text are used to predict relationships between entities in the text. This method allows an entity to simultaneously contain information about that entity in both the image and the text, thus enriching the entity's information, and then predicting relationships between entities based on this enriched information. However, since it only predicts relationships between entities and merely enriches the information about the entities themselves, the accuracy of predicting relationships between entities still needs improvement. Summary of the Invention
[0004] To overcome the problems existing in related technologies, this disclosure provides a method, apparatus, electronic device, and storage medium for predicting entity relationships. The technical solution of this disclosure is as follows:
[0005] According to a first aspect of the present disclosure, a method for predicting entity relationships is provided, comprising:
[0006] Obtain the text representation of the text in an image-text pair, wherein the image and the text in the image-text pair contain at least one identical entity;
[0007] Obtain the triplet representation of triplet text, which includes text triplet text and image triplet text. The text triplet text is used to describe the dependency relationship between phrases in the text, and the image triplet text is used to describe the relationship category between entities in the image.
[0008] Information fusion processing is performed on the text representation and the triplet representation to obtain an updated text representation, which includes information from the text triplet text and information from the image triplet text.
[0009] Based on the updated text representation, predict the relationship category between the head and tail entities in the text.
[0010] Optionally, it also includes:
[0011] Obtain the image representation of the image in the image-text pair;
[0012] The image representation and the triplet representation are subjected to information fusion processing to obtain an updated triplet representation, which includes information from the triplet representation and information from the image.
[0013] Information fusion processing is performed on the text representation and the updated triplet representation to obtain the target text representation, which includes information from the text triplet text, information from the image triplet text, and information from the image.
[0014] Based on the target text representation, predict the relationship category between the head entity and the tail entity.
[0015] Optionally, predicting the relationship category between head entities and tail entities in the text based on the updated text representation includes:
[0016] Extract the vector representations of the head entity and the tail entity from the updated text representation;
[0017] The vector representations of the head entity and the tail entity are concatenated to obtain the concatenated vector representation.
[0018] Predict the confidence level of the concatenated vector representing each relation category;
[0019] The relationship category with the highest confidence level is determined as the relationship category between the head entity and the tail entity.
[0020] Optionally, obtaining the triplet representation of the triplet text includes:
[0021] Obtain the text triplet text and the image triplet text;
[0022] The text triplet text and the image triplet text are concatenated to obtain the triplet text;
[0023] Based on the triplet text, the triplet representation of the triplet text is obtained.
[0024] Optionally, obtaining the text triplet text includes:
[0025] Construct a dependency syntax tree for the text, the dependency syntax tree representing the dependency relationships between phrases in the text, the phrases in the text including the head entity and the tail entity;
[0026] Based on the dependency syntax tree of the text, the text triplet text is obtained.
[0027] Optionally, the image triplet text is obtained, including:
[0028] Identify the individual entities in the image;
[0029] Predict the relationship category between the various entities in the image;
[0030] The image triplet text is generated based on the entities in the image and the relationship categories between the entities.
[0031] Optionally, obtaining the text representation of the text in the image-text pair includes:
[0032] The text is segmented into words to obtain a text sequence composed of multiple word groups;
[0033] Markers are filled at the beginning and end of the text sequence, respectively;
[0034] By padding the head entity with header entity markers before and after the head entity, and padding the tail entity with tail entity markers before and after the tail entity, the processed text sequence is obtained.
[0035] Based on the processed text sequence, a text representation of the text is obtained.
[0036] Optionally, obtaining the image representation of the image in the image-text pair includes:
[0037] Identify each entity in the image and obtain the location information of each entity in the image;
[0038] Obtain the vector representation of each entity;
[0039] The image representation of the image is obtained by concatenating the vector representations of the entities according to their positional order in the image.
[0040] According to a second aspect of the present disclosure, an entity relationship prediction apparatus is provided, comprising:
[0041] The text representation acquisition module is configured to acquire the text representation of text in an image-text pair, wherein the image and the text in the image-text pair contain at least one identical entity;
[0042] The triple representation acquisition module is configured to acquire triple representations of triple text, wherein the triple text includes text triple text and image triple text, wherein the text triple text is used to describe the dependency relationships between phrases in the text, and the image triple text is used to describe the relationship categories between entities in the image;
[0043] The information fusion module is configured to perform information fusion processing on the text representation and the triplet representation to obtain an updated text representation, wherein the updated text representation includes information of the text triplet text and information of the image triplet text;
[0044] The prediction module is configured to predict the relationship category between head entities and tail entities in the text based on the updated text representation.
[0045] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the entity relationship prediction method as described in the first aspect.
[0046] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the entity relationship prediction method as described in the first aspect.
[0047] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the entity relationship prediction method as described in the first aspect.
[0048] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:
[0049] In this disclosure, the image and text in an image-text pair contain at least one common entity, thus information from the image can be used to help predict the relationship categories between entities in the text. Triple text includes text triple text and image triple text. Text triple text describes the dependency relationships between phrases in the text, while image triple text describes the relationship categories between entities in the image. Therefore, the triple representation of triple text contains dependency relationship information between phrases in the text and relationship category information between entities in the image. Information fusion processing between the text representation and the triple representation allows the updated text representation to include the information contained in the triple representation. Therefore, the updated text representation contains information about the text itself, as well as dependency relationship information between phrases in the text and relationship category information between entities in the image. Furthermore, the prediction results are more accurate when predicting the relationship categories between head entities and tail entities in the text based on the updated text representation.
[0050] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description
[0051] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0052] Figure 1 This is a flowchart illustrating a method for predicting entity relationships according to an exemplary embodiment;
[0053] Figure 2 This is a schematic diagram of a dependency syntax tree according to an embodiment of this disclosure;
[0054] Figure 3 This is a structural diagram of the updated text representation obtained in the embodiments of this disclosure;
[0055] Figure 4 This is a flowchart illustrating a method for predicting entity relationships according to an exemplary embodiment;
[0056] Figure 5 This is a structural diagram of the target text representation obtained in this embodiment of the disclosure;
[0057] Figure 6 This is a flowchart illustrating the entity relationship prediction method in this embodiment of the disclosure;
[0058] Figure 7 This is a block diagram illustrating an entity relationship prediction device according to an exemplary embodiment;
[0059] Figure 8 This is a block diagram illustrating an apparatus for predicting entity relationships according to an exemplary embodiment;
[0060] Figure 9 This is a block diagram illustrating an apparatus for predicting entity relationships according to an exemplary embodiment. Detailed Implementation
[0061] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.
[0062] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.
[0063] This disclosure provides a method for predicting entity relationships, which can be used in a terminal or server. The terminal can be at least one of a smartphone, game console, desktop computer, tablet computer, e-book reader, MP3 player, MP4 player, and laptop computer. Those skilled in the art will understand that the number of terminals can be more or less. For example, there can be only one terminal, or there can be dozens or hundreds of terminals, or even more. The server includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center.
[0064] Figure 1 This is a flowchart illustrating a method for predicting entity relationships according to an exemplary embodiment, such as... Figure 1 As shown, the prediction method for entity relationships includes steps S11 to S14.
[0065] In step S11, the text representation of the text in the image-text pair is obtained, wherein the image and the text in the image-text pair contain at least one identical entity.
[0066] An image-text pair includes an image and a text, where the image and the text contain at least one identical entity. There is an association between the image and the text; for example, the image and the text describe the same object or the same event, or the text is a description of the image. For instance, the image content of one image-text pair could be a photo of person A holding an Oscar trophy, and the text content of the image-text pair could be "Regardless of what he's wearing, person A is my favorite Oscar-winning director," where person A is the same entity contained in both the image and the text.
[0067] The text contains at least two entities whose relationship category needs to be predicted. Following the structure of a triple (head entity, relationship category, tail entity) in a knowledge graph, these two entities can be named the head entity and the tail entity, respectively. When the text contains multiple entities, the relationship category between the two entities whose relationship category needs to be predicted can be determined by labeling them.
[0068] By inputting text into a language representation model, a text representation of the text can be obtained. This text representation is essentially the vectorization of the text, using vectors to represent it. Language representation models can include BERT (Bidirectional Encoder Representations from Transformer), Deep RNN, Deep Bidirectional RNN, LSTM (Long Short-Term Memory), or Convolutional Neural Network (CNN), among others.
[0069] Optionally, the text can be first segmented into words to obtain multiple phrases, which in turn form a text sequence. In this text sequence, the order of the phrases within the sequence matches their order within the text itself. It is understood that the multiple phrases in the text include both the beginning and end entities, each of which is also a phrase.
[0070] To improve text recognition, a header marker can be added before the beginning of the text sequence, and a tail marker can be added after the end. To further enhance entity information modeling and better identify head and tail entities during subsequent category prediction of the relationship between them, header entity markers can be added before and after head entities, and tail entity markers can be added before and after tail entities, resulting in a processed text sequence. The processed text sequence includes header markers, tail markers, head entity markers, and tail entity markers.
[0071] Optionally, the head marker can be [CLS], the tail marker can be [SEP], and the head entity marker with padding before the head entity can be [head]. start The header entity tag that fills in the header entity can be [head]. end The tail entity marker that padded the tail entity can be [tail]. start The tail entity marker, which is used to pad the tail entity, can be [tail]. end The processed text sequence input to the language representation model can be: T = ([CLS], t1, t2, ..., [head]). start ],head, [head end ],…,[tail start ],tail,[tail end ],…,t n[SEP]), where T represents the processed text sequence, t1, t2, t3, t4, t5, t6, t7, t8, t9 ... n The phrase "head" indicates the beginning of a word or phrase, and "tail" indicates the end of a word or phrase.
[0072] Thus, by segmenting the text into words, a text sequence can be obtained. Adding various tags to the text sequence helps the language representation model to better recognize the text and model entity information, thereby obtaining a more accurate text representation. Furthermore, tagging the head and tail entities is beneficial for better identifying the vector representations of the head and tail entities from the text representation when predicting the relationship between them in subsequent operations.
[0073] In step S12, the triplet representation of the triplet text is obtained. The triplet text includes text triplet text and image triplet text. The text triplet text is used to describe the dependency relationship between phrases in the text, and the image triplet text is used to describe the relationship category between entities in the image.
[0074] Triple text is obtained by concatenating text triple text and image triple text. Text triple text describes the dependency relationships between phrases in the text of the image-text pair; these dependencies refer to the syntactic collocations between phrases. Image triple text describes the relationship categories between entities in the images of the image-text pair.
[0075] To obtain the triplet representation of triplet text, one can: obtain the text triplet text and the image triplet text; concatenate the text triplet text and the image triplet text to obtain the triplet text; and obtain the triplet representation of the triplet text based on the triplet text.
[0076] Text triples are used to describe dependency relationships between phrases. Therefore, we can first obtain multiple phrases from the text, then identify the dependencies between them, and construct text triples based on these dependencies. A phrase may have dependencies on multiple phrases, so multiple text triples can be constructed for that phrase. Phrases that have no dependencies on any other phrases can be ignored. If only one text triple exists, it is directly identified as the text triple. If multiple text triples exist, they can be concatenated according to the order of the phrases in the text to obtain the text triple.
[0077] It can identify entities in an image and obtain the relationship categories of each entity. Based on the relationship categories, it constructs image triples. If only one image triple exists, it is directly identified as the image triple text. If multiple image triples exist, the multiple image triple texts are concatenated according to the order in which the entities appear in the image to obtain the final image triple text.
[0078] After obtaining the image triplet text and the text triplet text, concatenate the image triplet text and the text triplet text to obtain the text triplet. The order in which the image triplet text and the text triplet text are concatenated is not restricted.
[0079] Triplet text contains multiple triplets, and adjacent triplets can be separated by markers, such as [UNK]. Inputting triplet text into a language representation model yields its triplet representation. Let T1 represent the triplet text input to the language representation model, then the triplet text input to the language representation model can be represented as follows: T1 = (head1, relation1, tail1, [UNK], head2, relation2, tail2, [UNK], ..., head...). n ,relation n ,tail n ), where head1, relation1, and tail1 represent the head entity, relation class, and tail entity of the first triplet in the triplet text, respectively; head2, relation2, and tail2 represent the head entity, relation class, and tail entity of the second triplet in the triplet text, respectively; head n ,relation n ,tail n These represent the head entity, relation category, and tail entity of the nth triple in the triplet text, respectively.
[0080] Thus, since triplet text is obtained by concatenating text triplet text and image triplet text, it contains information from both text triplet text and image triplet text. Because text triplet text is used to describe dependency relationships between phrases in the text, and image triplet text is used to describe the categories of relationships between entities in the image, triplet text contains both dependency relationships between phrases in the text and categories of relationships between entities in the image.
[0081] Optionally, based on the above technical solution, obtaining text triplet text may include: constructing a dependency syntax tree of the text, wherein the dependency syntax tree represents the dependency relationship between phrases in the text, wherein the phrases in the text include the head entity and the tail entity; and obtaining text triplet text based on the dependency syntax tree of the text.
[0082] Dependency trees for text can be constructed using tools such as TweebankNLP (a natural language processing tool) or StanfordNLP (another natural language processing tool). A dependency tree describes the dependency relationships between phrases in a text. It indicates the syntactic collocations between phrases, which are semantically related. Each pair of adjacent nodes and the type of the edge in the dependency tree constitutes a triplet text.
[0083] Figure 2 This is a dependency syntax tree according to an embodiment of this disclosure, and the sentence corresponding to this dependency syntax tree is "Xiaoming is my favorite classmate". From Figure 2 It can be seen that the phrase "like" is the noun subject of the phrase "Xiaoming," the adjective modifier of the noun phrase "classmate," the possessive modifier of the phrase "I," and the linking verb "is." Therefore, based on... Figure 2 The dependency syntax tree can be used to obtain text triples such as (I, like, Xiaoming), (I, like, classmate), etc.
[0084] For methods of constructing dependency syntax trees and constructing triplet texts based on dependency syntax trees, you can refer to related technologies.
[0085] Thus, a text triplet text can be obtained by constructing a dependency syntax tree. The text triplet text contains the dependency relations of phrases within the text. Furthermore, using the text triplet text, a triplet text can be obtained, which also contains the dependency relations of phrases within the text. Since the head entity and tail entity are each one of multiple phrases, the dependency relations of phrases within the text also include the dependency relations on the head entity and tail entity respectively. Therefore, the text triplet text contains the dependency relations of both the head entity and the tail entity.
[0086] Optionally, based on the above technical solution, obtaining image triplet text may include: identifying each entity in the image; predicting the relationship category between each entity in the image; and generating image triplet text based on each entity in the image and the relationship category between each entity.
[0087] Unbiased-SGG (Unbiased Scene Graph Generation) can be used to generate image triple text. Unbiased Scene Graph Generation can identify entities in an image and predict the relationship categories between them. Inputting an image into the Unbiased Scene Graph Generation model yields a graph structure where nodes represent each entity in the image, and edges between entities represent the relationship categories. Based on the graph structure generated by the Unbiased Scene Graph Generation model, image triple text can be generated. In the image triple text, the head and tail entities are two connected entities, and the relationship category in the image triple text is represented by the edge between these two connected entities.
[0088] Alternatively, visual grounding tools can be used to identify the entities in an image, and then image processing models can be used to identify the relationship categories between the entities in the image, thereby obtaining the image triplet text.
[0089] In this way, we can obtain the image triplet text, which contains the relationship categories between entities in the image. Then, we can use the image triplet text to obtain triplet text, which also contains the relationship categories between entities in the image.
[0090] In step S13, information fusion processing is performed on the text representation and the triplet representation to obtain an updated text representation, which includes information from the text triplet text and information from the image triplet text.
[0091] By fusing information from text representations and triple representations, updated text representations and updated triple representations can be obtained. The updated text representation includes information from both text representations and triple representations; the updated triple representation includes information from both triple representations and text representations. Because triple representations include information from both text triples and image triples, the updated text representation includes information from both text triples and image triples. Specifically, the updated text representation includes dependency relationships between phrases in the text and relationship categories between entities in the image.
[0092] Optionally, information fusion processing of the text representation and the triple representation can be performed by inputting the text representation and the triple representation into an attention model to obtain an updated text representation.
[0093] Attention models employ an attention mechanism that calculates the attention between the input triple representation and the vector representation of each word in the text representation. When calculating the attention between the vector representations of two words, information between the two words is fused. Therefore, the updated text representation output by the attention model includes information from the triple representation. Attention models can be transformer models (a deep learning model employing a self-attention mechanism), Axial Transformer models (axial attention mechanism models), or Linformer models (linear complexity self-attention mechanism models), etc.
[0094] Figure 3 This is a structural diagram of the updated text representation obtained in this embodiment of the disclosure. By inputting the text representation and the triplet representation into the attention model, the updated text representation and the updated triplet representation can be obtained.
[0095] Thus, by using an attention model, the updated text representation can include information from the triple representation.
[0096] In step S14, based on the updated text representation, the relationship category of the head entity and the tail entity in the text is predicted.
[0097] A language processing model can be directly used to predict the relation categories of head and tail entities in the text based on the updated text representation. The updated text representation is input into the language processing model, which outputs the confidence level of each head and tail entity belonging to each relation category. The relation category between the head and tail entities is determined based on the confidence level of each relation category. The language processing model can be trained under supervised supervision.
[0098] Optionally, predicting the relationship category between the head entity and the tail entity in the text based on the updated text representation can be achieved by extracting the vector representations of the head entity and the tail entity from the updated text representation, and concatenating the vector representations of the head entity and the tail entity to obtain the concatenated vector representation.
[0099] Because the text sequence input to the language representation model already includes labels for head and tail entities, these labels are not transformed by the model. Therefore, the text representation also contains these labels, and consequently, the updated text representation also includes them. Thus, the vector representations of the head and tail entities can be quickly obtained from the updated text representation.
[0100] The predicted concatenated vector represents the confidence level of each relation category. Based on the confidence level of the head entity and tail entity in each relation category, the relation category with the highest confidence level is used to determine the relation category between the head entity and the tail entity.
[0101] To predict the confidence level of the concatenated vector representation belonging to each relation category, the concatenated vector representation can be input into a fully connected layer, mapped to the dimension of the number of relation categories, and then normalized using softmax (normalized exponential function). This will give the confidence level of the head entity and tail entity belonging to each relation category.
[0102] Thus, because the updated text representation includes dependencies between phrases in the text, as well as relationship categories between entities in the image, predicting the relationship categories of head and tail entities in the text is more accurate based on the updated text representation.
[0103] Using the technical solution of this disclosure, the image and text in an image-text pair contain at least one common entity. Therefore, information from the image can be used to help predict the relationship category between entities in the text. Triple text includes text triple text and image triple text. Text triple text describes the dependency relationships between phrases in the text, while image triple text describes the relationship category between entities in the image. Therefore, the triple representation of triple text contains dependency relationship information between phrases in the text and relationship category information between entities in the image. Information fusion processing of the text representation and the triple representation allows the updated text representation to include the information contained in the triple representation. Therefore, the updated text representation contains information about the text itself, as well as dependency relationship information between phrases in the text and relationship category information between entities in the image. Furthermore, when predicting the relationship category between head entities and tail entities in the text based on the updated text representation, the prediction result is more accurate.
[0104] Compared to related technologies that enrich the information of the entities themselves and then predict the relationship categories between entities based on the enriched information, the technical solution of this disclosure, considering that the prediction is about the relationship categories between entities, proposes a technical concept of enriching the information about the relationship categories between entities. It constructs a graph-text information fusion processing framework centered on a triple representation that includes the relationship category information between entities in the image and the dependency relationships between entities in the text. It considers the relationships between entities in the image and the dependency relationships between phrases in the text, and innovatively uses triples of different modalities as elements to perform information fusion processing on image-text information, achieving fine-grained structured information fusion processing of the relationship categories of entities contained in both the image and the text. Therefore, the updated text representation contains rich information about the relationship categories between entities, and the predicted relationship categories between head and tail entities based on the updated text representation are more accurate.
[0105] Figure 4 This is a flowchart illustrating a method for predicting entity relationships according to an exemplary embodiment, such as... Figure 4 As shown, the prediction method for entity relationships includes steps S21 to S25.
[0106] In step S21, the text representation of the text in the image-text pair is obtained.
[0107] The specific implementation method for this step can be referred to step S11, and will not be repeated here.
[0108] In step S22, the image representation of the image in the image-text pair is obtained.
[0109] It can identify individual entities in an image and obtain their location information within the image; obtain the vector representation of each entity; and concatenate the vector representations of each entity according to their position order in the image to obtain the image representation. Image representation refers to vectorizing an image to obtain a vector-represented image.
[0110] Alternatively, visual grounding tools or image localization models can be used to identify individual entities in an image and the image regions in which each entity is located. The vector representations of each identified entity are then obtained using a ViT model (Vision Transformer, an image deep learning model employing a self-attention mechanism). Alternatively, encoding techniques such as CLIP (Contrastive Language-Image Pre-training) can be used to obtain the vector representations of each identified entity.
[0111] The image representation of the image can be obtained by concatenating the vector representations of each entity in the image from left to right or from top to bottom according to their position in the image.
[0112] In this way, we can obtain the image representation of the image, which contains information about each entity in the image. This image representation can then be used to perform information fusion processing with the triple representation, so that the updated triple representation includes the information of the image.
[0113] In step S23, information fusion processing is performed on the image representation and the triple representation to obtain an updated triple representation, which includes information from the triple representation and information from the image.
[0114] The attention model can be formed by inputting the image representation and the triple representation into the attention model to obtain the updated triple representation. The attention model can be a transformer model, an Axial Transformer model, or a Linformer model, etc.
[0115] Attention models employ an attention mechanism that calculates the attention between the input image representation and the vector representation of each word in the triplet representation. This attention calculation fuses information between the two words, resulting in the updated triplet representation output by the attention model incorporating information from the image representation. Since the image representation contains information about each entity in the image, the updated triplet representation also includes information about those entities.
[0116] In step S24, information fusion processing is performed on the text representation and the updated triplet representation to obtain the target text representation, which includes the information of the text triplet text, the information of the image triplet text, and the information of the image.
[0117] The text representation and the updated triple representation can be input into the attention model to obtain the target text representation. The attention model can be a transformer model, an Axial Transformer model, or a Linformer model, etc.
[0118] Attention models employ an attention mechanism that calculates the correlation between the input text representation and the vector representation of each word in the updated triplet representation. When calculating the attention between the vector representations of two words, information between the two words is fused. Therefore, the target text representation output by the attention model includes information from the updated triplet representation. The updated triplet representation contains information from the image triplet text, the text triplet text, and information about each entity in the image. Similarly, the target text representation also contains information from the image triplet text, the text triplet text, and information about each entity in the image.
[0119] Figure 5 This is a structural diagram of the target text representation obtained in this embodiment of the disclosure. By inputting the image representation and the triple representation into the attention model, updated triple representation text representation and updated image representation are obtained. Inputting the updated triple representation and text representation into the attention model again yields the target text representation.
[0120] In step S25, based on the target text representation, the relationship category between the head entity and the tail entity is predicted.
[0121] The method for predicting the relationship category between head entities and tail entities based on the target text representation can refer to the method for predicting the relationship category between head entities and tail entities based on the updated text representation.
[0122] This can be achieved by extracting the vector representations of the head and tail entities from the target text representation, concatenating them, and obtaining the concatenated vector representation. The confidence level of the concatenated vector representation belonging to each relation category is then predicted. Based on the confidence levels of the head and tail entities belonging to each relation category, the relation category between the head and tail entities is determined. Alternatively, the confidence level of the concatenated vector representation belonging to each relation category can be obtained by inputting the concatenated vector representation into a fully connected layer, mapping it to the dimension of the number of relation categories, and then normalizing it using softmax (a normalized exponential function).
[0123] Using the technical solution of this disclosure, the triplet text includes text triplet text and image triplet text. Text triplet text describes the dependency relationships between phrases in the text, while image triplet text describes the relationship categories between entities in the image. Therefore, the triplet representation of the triplet text contains dependency relationship information between phrases in the text and relationship category information between entities in the image. Information fusion processing between the triplet representation and the image representation allows the updated triplet representation to obtain the information contained in the image representation. Therefore, the updated triplet representation contains entity information in the image, dependency relationship information between phrases in the text, and relationship category information between entities in the image. Information fusion processing between the text representation and the updated triplet representation allows the target text representation to obtain the information contained in the updated triplet representation. Therefore, the target text representation contains entity information in the image, dependency relationship information between phrases in the text, and relationship category information between entities in the image. Furthermore, when predicting the relationship category between head entities and tail entities in the text based on the target text representation, the prediction result is more accurate.
[0124] Optionally, based on the above technical solution, when predicting the relationship category between the head entity and the tail entity, the prediction can be made based on the text representation after multiple updates.
[0125] You can first obtain the image representation V i A triplet represents P i and text representation T i The image representation and the triplet representation are fused to obtain the first updated image representation V. i+1 The triplet represents P i1 The text representation and the triple representation are fused to obtain the first updated text representation T. i+1 The triplet represents P i2 , (P i1 +P i2 )= P i+1 Among them, P i+1 It is the updated triplet representation.
[0126] Reuse V i+1 and P i+1 Perform information fusion processing, and P i+1 and T i+1 Information fusion processing is performed to obtain V i+2 P i+2 and T i+2 By analogy, we can obtain the text V after n updates. i+n P i+n and T i+n We can use the text representation updated n times to predict the relationship category between the head and tail entities.
[0127] Alternatively, one can first obtain an image representation of V. i A triplet represents P i and text representation T i The image representation and the triplet representation are fused to obtain the first updated image representation V. i+1 The triplet P' i+1 Text representation and P' i+1 Information fusion processing is performed to obtain the first updated text representation T. i+1 The triplet represents P i+1 .
[0128] Reuse V i+1 and P i+1 Information fusion processing is performed to obtain P' i+2 , and P' i+2 and T i+1 Information fusion processing is performed to obtain P i+2 and T i+2By analogy, we can obtain the text V after n updates. i+n P i+n and T i+n We can use the text representation updated n times to predict the relationship category between the head and tail entities.
[0129] Thus, the repeatedly updated text representations obtained after multiple information fusion processes more seamlessly integrate entity information in the image and the information contained in the triplet text. Consequently, the predicted relationship categories between head and tail entities based on the repeatedly updated text representations are more accurate.
[0130] Figure 6 This is a flowchart illustrating the entity relationship prediction method in this embodiment. An image representation is obtained based on the image, and a text representation is obtained based on the text. Image triplet text is obtained from the image, and text triplet text is obtained from the text. The image triplet text and the text triplet text are concatenated to obtain a triplet text. The triplet representation of the triplet text is obtained. Based on the image representation, text representation, and triplet representation, information fusion processing centered on the triplet representation is performed to obtain an updated text representation. The relationship category between the head entity and the tail entity is predicted based on the updated text representation.
[0131] The various models used in the embodiments of this disclosure can be obtained through supervised training, unsupervised training, or deep learning. Methods for supervised training, unsupervised training, or deep learning can be found in related technologies.
[0132] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure.
[0133] Figure 7 This is a block diagram illustrating an entity relationship prediction device according to an exemplary embodiment, with reference to... Figure 7 The device includes a text representation acquisition module 71, a triplet representation acquisition module 72, an information fusion module 73, and a prediction module 74.
[0134] The text representation acquisition module 71 is configured to acquire the text representation of text in an image-text pair, wherein the image and the text in the image-text pair contain at least one identical entity;
[0135] The triplet representation acquisition module 72 is configured to acquire triplet representations of triplet text, wherein the triplet text includes text triplet text and image triplet text, wherein the text triplet text is used to describe the dependency relationships between phrases in the text, and the image triplet text is used to describe the relationship categories between entities in the image;
[0136] The information fusion module 73 is configured to perform information fusion processing on the text representation and the triplet representation to obtain an updated text representation, wherein the updated text representation includes information of the text triplet text and information of the image triplet text;
[0137] The prediction module 74 is configured to predict the relationship category between head entities and tail entities in the text based on the updated text representation.
[0138] Optionally, it also includes:
[0139] The image representation acquisition module is configured to acquire the image representation of the image in the image-text pair;
[0140] The first processing module is configured to perform information fusion processing on the image representation and the triplet representation to obtain an updated triplet representation, wherein the updated triplet representation includes information from the triplet representation and information from the image;
[0141] The second processing module is configured to perform information fusion processing on the text representation and the updated triplet representation to obtain a target text representation, wherein the target text representation includes information of the text triplet text, information of the image triplet text, and information of the image.
[0142] The category prediction module is configured to predict the relationship category between the head entity and the tail entity based on the target text representation.
[0143] Optionally, the prediction module 74 includes:
[0144] The extraction unit is configured to extract the vector representations of the head entity and the tail entity from the updated text representation;
[0145] The vector concatenation unit is configured to concatenate the vector representation of the head entity and the vector representation of the tail entity to obtain the concatenated vector representation.
[0146] A confidence prediction unit is configured to predict the confidence level of the concatenated vector representation belonging to each relation category;
[0147] The category determination unit is configured to determine the relationship category with the highest confidence level as the relationship category between the head entity and the tail entity.
[0148] Optionally, the triple representation acquisition module 72 includes:
[0149] The text acquisition unit is configured to acquire the text triplet text and the image triplet text;
[0150] The text concatenation unit is configured to concatenate the text triplet text and the image triplet text to obtain the triplet text;
[0151] The unit is configured to obtain a triple representation of the triple text based on the triple text.
[0152] Optionally, the text acquisition unit includes:
[0153] The construction subunit is configured to construct a dependency syntax tree of the text, the dependency syntax tree representing the dependency relationships between phrases in the text, the phrases in the text including the head entity and the tail entity;
[0154] The resulting subunits are configured as a dependency syntax tree based on the text, resulting in the text triplet text.
[0155] Optionally, the text acquisition unit includes:
[0156] The recognition subunit is configured to recognize various entities in the image;
[0157] The prediction subunit is configured to predict the relationship category between various entities in the image;
[0158] The generation subunit is configured to generate image triplet text based on the entities in the image and the relationship categories between the entities.
[0159] Optionally, the text representation acquisition module 71 includes:
[0160] The segmentation unit is configured to segment the text into words to obtain a text sequence composed of multiple word groups;
[0161] The first padding unit is configured to pad the beginning and end of the text sequence with marks, respectively.
[0162] The second padding unit is configured to pad the head entity before and after the head entity with a head entity marker, and to pad the tail entity before and after the tail entity with a tail entity marker, to obtain the processed text sequence.
[0163] The text representation unit is configured to obtain a text representation of the text based on the processed text sequence.
[0164] Optionally, the image representation acquisition module includes:
[0165] An entity recognition unit is configured to recognize each entity in the image and to obtain the position information of each entity in the image;
[0166] The vector representation acquisition unit is configured to acquire the vector representation of each entity;
[0167] The vector representation splicing unit is configured to splice the vector representations of the entities according to their positional order in the image to obtain the image representation of the image.
[0168] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.
[0169] Figure 8 This is a block diagram illustrating an apparatus 800 for predicting entity relationships according to an exemplary embodiment. For example, apparatus 800 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.
[0170] Reference Figure 8 The device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.
[0171] Processing component 802 typically controls the overall operation of device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the aforementioned entity relationship prediction method. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.
[0172] Memory 804 is configured to store various types of data to support the operation of device 800. Examples of this data include instructions for any application or method operating on device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.
[0173] Power supply component 806 provides power to various components of device 800. Power supply component 806 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to device 800.
[0174] Multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.
[0175] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.
[0176] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.
[0177] Sensor assembly 814 includes one or more sensors for providing status assessments of various aspects of device 800. For example, sensor assembly 814 may detect the on / off state of device 800, the relative positioning of components such as the display and keypad of device 800, changes in the position of device 800 or a component of device 800, the presence or absence of user contact with device 800, the orientation or acceleration / deceleration of device 800, and temperature changes of device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.
[0178] Communication component 816 is configured to facilitate wired or wireless communication between device 800 and other devices. Device 800 can access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
[0179] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the aforementioned method for predicting entity relationships.
[0180] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of the device 800 to complete the aforementioned entity relationship prediction method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.
[0181] Figure 9This is a block diagram illustrating an apparatus 900 for predicting entity relationships according to an exemplary embodiment. For example, apparatus 900 may be provided as a server. (Refer to...) Figure 9 The apparatus 900 includes a processing component 922, which further includes one or more processors, and memory resources represented by memory 932 for storing instructions, such as computer program products, that can be executed by the processing component 922. The computer program products stored in memory 932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 922 is configured to execute instructions to perform the aforementioned entity relationship prediction method.
[0182] Device 900 may also include a power supply component 926 configured to perform power management of device 900, a wired or wireless network interface 950 configured to connect device 900 to a network, and an input / output (I / O) interface 958. Device 900 may operate on an operating system stored in memory 932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.
[0183] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.
[0184] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.
Claims
1. A method for predicting entity relationships, characterized in that, include: Obtain the text representation of the text in an image-text pair, wherein the image and the text in the image-text pair contain at least one identical entity; Obtaining the triplet representation of triplet text, wherein the triplet text includes text triplet text and image triplet text, the text triplet text being used to describe the dependency relationships between phrases in the text, and the image triplet text being used to describe the relationship categories between entities in the image; wherein, obtaining the triplet representation of triplet text includes: obtaining the text triplet text and the image triplet text; concatenating the text triplet text and the image triplet text to obtain the triplet text; and obtaining the triplet representation of the triplet text based on the triplet text; Obtain the image representation of the image in the image-text pair; An attention-based information fusion process is performed on the image representation and the triple representation to obtain an updated triple representation, which includes information from the triple representation and information from the image. The text representation and the updated triplet representation are subjected to information fusion processing based on an attention mechanism to obtain a target text representation, which includes information about the text triplet text, information about the image triplet text, and information about the image. Based on the target text representation, predict the relationship category between the head and tail entities in the text.
2. The method according to claim 1, characterized in that, The step of predicting the relationship category between head entities and tail entities in the text based on the target text representation includes: Extract the vector representations of the head entity and the tail entity from the target text representation; The vector representations of the head entity and the tail entity are concatenated to obtain the concatenated vector representation. Predict the confidence level of the concatenated vector representing each relation category; The relationship category with the highest confidence level is determined as the relationship category between the head entity and the tail entity.
3. The method according to claim 1, characterized in that, The process of obtaining the text triplet text includes: Construct a dependency syntax tree for the text, the dependency syntax tree representing the dependency relationships between phrases in the text, the phrases in the text including the head entity and the tail entity; Based on the dependency syntax tree of the text, the text triplet text is obtained.
4. The method according to claim 1, characterized in that, Obtaining the image triplet text includes: Identify the individual entities in the image; Predict the relationship category between the various entities in the image; The image triplet text is generated based on the entities in the image and the relationship categories between the entities.
5. The method according to claim 1, characterized in that, The acquisition of the text representation of the text in the image-text pair includes: The text is segmented into words to obtain a text sequence composed of multiple word groups; Markers are filled at the beginning and end of the text sequence, respectively; By padding the head entity with header entity markers before and after the head entity, and padding the tail entity with tail entity markers before and after the tail entity, the processed text sequence is obtained. Based on the processed text sequence, a text representation of the text is obtained.
6. The method according to claim 1, characterized in that, The step of obtaining the image representation of the image in the image-text pair includes: Identify each entity in the image and obtain the location information of each entity in the image; Obtain the vector representation of each entity; The image representation of the image is obtained by concatenating the vector representations of the entities according to their positional order in the image.
7. A device for predicting entity relationships, characterized in that, include: The text representation acquisition module is configured to acquire the text representation of text in an image-text pair, wherein the image and the text in the image-text pair contain at least one identical entity; A triplet representation acquisition module is configured to acquire triplet representations of triplet text, wherein the triplet text includes text triplet text and image triplet text. The text triplet text is used to describe the dependency relationships between phrases in the text, and the image triplet text is used to describe the relationship categories between entities in the image. The triplet representation acquisition module includes: a text acquisition unit configured to acquire the text triplet text and the image triplet text; a text concatenation unit configured to concatenate the text triplet text and the image triplet text to obtain the triplet text; and a obtaining unit configured to obtain the triplet representation of the triplet text based on the triplet text. The image representation acquisition module is configured to acquire the image representation of the image in the image-text pair; The first processing module is configured to perform attention-based information fusion processing on the image representation and the triplet representation to obtain an updated triplet representation, wherein the updated triplet representation includes information from the triplet representation and information from the image. The second processing module is configured to perform attention-based information fusion processing on the text representation and the updated triplet representation to obtain a target text representation, wherein the target text representation includes information of the text triplet text, information of the image triplet text, and information of the image. The category prediction module is configured to predict the relationship category between the head and tail entities of the text based on the target text representation.
8. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the entity relationship prediction method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the entity relation prediction method as described in any one of claims 1 to 6.