A semi-supervised graph-text relation extraction system and method based on co-training

Abstract

The present application belongs to the technical field of computer vision and machine learning, and particularly relates to a semi-supervised graph-text relation extraction system based on co-training and a method thereof. The method comprises the following steps: inputting an image, identifying candidate regions of multiple entities through a target detection network; combining the entities in the candidate regions two by two, sorting the scores of the overlapping areas of the entities in the candidate regions between the entity pairs, and screening the candidate entity pairs with scores higher than a threshold value as candidate suggestion entity pairs; obtaining the multi-modal features of the entity pairs based on the candidate suggestion entity pairs through coding; constructing an image view and a semantic view based on the multi-modal features of the entity pairs; training an image classifier and a semantic classifier under the constructed image view and semantic view, updating the network through training, and thereby outputting the visual relationship between each candidate entity pair. The present application uses a semi-supervised learning mode, enables the data of two modalities of image semantics and text semantics to obtain deep information between the modalities through co-training, improves the accuracy of relation detection, and is applied to tasks such as visual question answering and knowledge reasoning.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and machine learning, and particularly relates to a semi-supervised graph-text relation extraction system based on co-training and a method thereof. BACKGROUND

[0002] The visual relation detection task aims to identify entities in an image and extract the relations between entity pairs. In the visual relation detection task, the relation between two entities is usually represented by a triple, i.e., <subject, relation, object>, such as <person, hold, umbrella>. The visual relation detection task generally includes an entity recognition task and a relation recognition task, can promote image scene understanding, and can be applied to tasks such as image-text matching, visual question answering, and knowledge reasoning.

[0003] The visual relation detection task belongs to the image-text multimodal learning task, and existing research methods can be mainly divided into three categories: a visual phrase-based method, an entity and relation decoupling method, and a scene graph generation-based method. The visual phrase-based method detects visual relations by taking visual phrases as a whole, and regards each different combination of object classes and relation predicates as a different class; however, when the number of entity pair combinations becomes very large, this method will be difficult to handle; the entity and relation decoupling method separates relation classification from entity classification, and provides additional clues for relation reasoning by exploring the correlation between entity pairs and relations; the scene graph generation-based method generally jointly learns entity detection and relation detection, and constructs a scene graph by iteratively combining entity and relation features.

[0004] In existing related work, the research on the visual relation detection task has made certain research progress, but due to the large number of entity classes and the complex relations between entity pairs in real life, the training data with relation labels are far from sufficient, which makes the relation detection accuracy low. SUMMARY

[0005] The purpose of the present application is a semi-supervised graph-text relation extraction system based on co-training and a method thereof, to solve the problem of low visual relation detection accuracy caused by insufficient relation labeled data.

[0006] To achieve the above purpose, the technical solution adopted by the present application is:

[0007] A semi-supervised graph-text relation extraction system based on co-training comprises the following modules:

[0008] S21: an image target detection module, configured to obtain image data, and identify candidate regions of multiple entities, visual features of regions where the entities are located, and entity corresponding labels through a target detection network;

[0009] S22: an entity pair suggestion module for screening candidate entity pair suggestions that may have a relationship, combining each of the entities in the candidate entity pair with each other, sorting the scores of the overlapping areas of the candidate regions where each entity in the entity pair is located, and screening those with scores higher than a threshold as candidate entity pair suggestions;

[0010] S23: a multi-modal feature extraction module for obtaining multi-modal features of the candidate entity pair suggestions after feature encoding, including: entity pair relative position features, joint visual features of the entity pair, and entity pair semantic features;

[0011] S24: a view construction module for obtaining feature representations in an image view and a semantic view based on the obtained multi-modal features, constructing the image view by multiplying the joint visual features of the entity pair and the entity pair relative position features, and constructing the semantic view by multiplying the entity pair semantic features and the entity pair relative position features;

[0012] S25: a graph-text relationship discrimination module for discriminating the relationship between the entity pairs, discriminating the relationship between the candidate entity pairs in an image classifier in the image view according to the output relationship scores and discriminating the relationship between the candidate entity pairs in a semantic classifier in the semantic view according to the output relationship scores, and the higher the score, the more final predicted relationships.

[0013] The method for performing semi-supervised graph-text relationship extraction based on collaborative training using the semi-supervised graph-text relationship extraction system based on collaborative training includes the following steps:

[0014] S11: inputting an image, identifying candidate regions of multiple entities and related information of the entities through a target detection network;

[0015] S12: combining each of the entities in the candidate region pairs with each other, sorting the scores of the overlapping areas of the candidate regions where each entity in the entity pair is located, and screening those with scores higher than a threshold as candidate entity pair suggestions;

[0016] S13: obtaining multi-modal features of the entity pairs based on the candidate entity pair suggestions and feature encoding, including: entity pair relative position features, joint visual features of the entity pair, and entity pair semantic features, obtaining multi-modal features of the entity pairs based on the candidate entity pair suggestions and their related information;

[0017] S14: based on the multi-modal features of the entity pairs, constructing an image view by multiplying the joint visual features of the entity pair and the entity pair relative position features, and constructing a semantic view by multiplying the entity pair semantic features and the entity pair relative position features;

[0018] S15: training an image classifier and a semantic classifier in the constructed image view and semantic view, constantly training and updating the network by calculating the loss and backpropagation, and thereby outputting the visual relationship between each candidate entity pair.

[0019] Preferably, in the step S11, the candidate region of the entity and the related information of the entity are specifically as follows:

[0020] The candidate region of the entity: coordinate information of the entity, represented as (x, y, w, h), x and y represent the coordinate of the center point of the region, and w and h represent the width and height of the region, respectively.

[0021] The related information of the entity: visual features of the region where the entity is located and the corresponding label of the entity obtained according to the visual features of the region where the entity is located.

[0022] Preferably, in the step S12, the information of the candidate suggested entity pair is specifically as follows:

[0023] The candidate suggested entity pair is an entity pair that may have a relationship, the entity pair is a subject and an object, and the entity pair that has a relationship has directionality; the related information of the candidate suggested entity pair includes coordinate information of the entity pair and a joint region thereof, visual features of the entity pair and the joint region thereof, and corresponding labels of the entity regions, the coordinate information of the entity pair and the joint region thereof is represented as (x s , y s , w s , h s ), (x o , y o , w o , h o ) and (x u , y u , w u , h u ), wherein s represents the subject, o represents the object, and u represents the joint region of the subject and the object.

[0024] Preferably, in the step S13, the multi-modal features of the entity pair include:

[0025] Entity pair relative position feature The coordinate information of the candidate entity pair and the joint region thereof is obtained by encoding through two fully connected layers;

[0026] Joint visual features of the entity pair The joint visual features of the entity pair are obtained by encoding through two fully connected layers from the visual features of the joint region of the entity pair;

[0027] Entity pair semantic feature The word2vec pre-training language model obtained by using the wikidata knowledge base, so that the word vector representation of the entity label can be obtained, the subject and the object in the entity pair are converted into word vector representations, and the representation obtained by connecting the two word vector representations at the head and tail is encoded through two fully connected layers.

[0028] Preferably, in the step S14, the image view and the semantic view are constructed based on the multi-modal features of the entity pairs, and specifically:

[0029] The joint visual features of the entity pairs in the multi-modal features are multiplied by the relative position features of the entity pairs to construct the image view, to obtain the feature representation of the data in the image view;

[0030] The semantic features of the entity pairs in the multi-modal features are multiplied by the relative position features of the entity pairs to construct the semantic view, to obtain the feature representation of the data in the semantic view.

[0031] Preferably, in the step S15, the image classifier and the semantic classifier in the constructed image view and semantic view include:

[0032] The image classifier is a perceptron composed of a neural network with two fully connected layers;

[0033] The semantic classifier is a perceptron composed of a neural network with two fully connected layers.

[0034] Preferably, the training process includes the following steps:

[0035] Step S101: input a training data set, divide the data set into a labeled data set and an unlabeled data set at a ratio of 1:10 , sample the two data sets into the same batch of data;

[0036] Step S102: obtain multi-modal features of the first batch of data in the labeled data set and the unlabeled data set through the multi-modal feature extraction module, including: joint visual features of entity pairs semantic features of entity pairs and relative position features of entity pairs obtain the feature representation in the image view and the feature representation in the semantic view

[0037] Step S103: input the feature representation in the image view of the labeled data set into the image classifier f1 in the graph-text collaborative classification module, and input the feature representation in the semantic view into the semantic classifier f2 in the graph-text collaborative classification module, to obtain the relationship prediction scores between the entity pairs, respectively, and obtain the normalized relationship prediction scores 1 and relationship prediction scores 2 through a softmax function, calculate the cross-entropy loss of the relationship label and the relationship score 1 and the cross-entropy loss of the relationship label and the relationship score 2, and the sum of the two parts of the loss is the classification loss ​

[0038] Step S104: inputting the unlabeled data set In the image view feature representation input into the image classifier under the image-text collaborative classification module, in the semantic view feature representation input into the semantic classifier under the image-text collaborative classification module, the relationship prediction scores between the entity pairs are obtained respectively, and the normalized relationship prediction scores 3 and 4 are obtained through the softmax function, and the similarity of the prediction scores 3 and 4 is measured by the JS divergence degree as the collaborative loss

[0039] Step S105: inputting the labeled data set In the image view feature representation input into the semantic classifier under the image-text collaborative classification module, in the semantic view feature representation input into the image classifier under the image-text collaborative classification module, the relationship prediction scores between the entity pairs are obtained respectively, and the normalized relationship prediction scores 5 and 6 are obtained through the softmax function, the cross-entropy loss of the relationship label and the relationship score 5 and the cross-entropy loss of the relationship label and the relationship score 6 are calculated, and the sum of the two parts of the loss is taken as the modal cross-loss on the labeled data set

[0040] Step S106: inputting the unlabeled data set In the image view feature representation input into the semantic classifier under the image-text collaborative classification module, in the semantic view feature representation input into the image classifier under the image-text collaborative classification module, the relationship prediction scores between the entity pairs are obtained respectively, and the normalized relationship prediction scores 7 and 8 are obtained through the softmax function, the cross-entropy loss of the relationship prediction score 7 and the relationship score 3 and the cross-entropy loss of the relationship prediction score 8 and the relationship score 4 are calculated, and the sum of the two parts of the loss is taken as the modal cross-loss on the unlabeled data set

[0041] Step S107: calculating the classification loss Collaborative loss Modal cross-loss on the labeled data set and modal cross-loss on the unlabeled data set The sum of the above is taken as the total loss The gradient is calculated, and the parameters of the model are updated through back propagation, and the processing of the next batch of data is entered;

[0042] Step S108, repeat steps S102-S107 until all batches of data are processed, update the learning rate, at this time the data is resampled according to step S101, and enters the next round of iteration until the maximum iteration number is reached.​​

[0043] the total loss is defined as follows:

[0044]

[0045] wherein, is the classification loss, is the synergy loss, is the modality-cross loss on the labeled dataset, is the modality-cross loss on the unlabeled dataset, loss is defined as follows:

[0046]

[0047]

[0048]

[0049]

[0050] wherein, denotes the multimodal feature representation of an entity pair in Y, j denotes the relation label of the corresponding entity pair, denotes the cross-entropy, denotes the multimodal feature representation of an entity pair in Y, denotes the relation prediction score 1, denotes the relation prediction score 2, denotes the relation prediction score 3, denotes the relation prediction score 4, denotes the relation prediction score 5, denotes the relation prediction score 6, denotes the relation prediction score 7, denotes the relation prediction score 8, is defined as follows:

[0051]

[0052]

[0053]

[0054]

[0055]

[0056]

[0057]

[0058]

[0059] wherein f1 represents an image classifier, f2 represents a semantic classifier, represents a feature representation of the labeled dataset under the image view, represents a feature representation of the labeled dataset under the semantic view, represents a feature representation of the unlabeled dataset under the image view, represents a feature representation of the unlabeled dataset under the semantic view.

[0060] Compared with the prior art, the present application has the beneficial effects that:

[0061] The present application is a semi-supervised image relation extraction method and system based on collaborative training, which effectively solves the problem of insufficient training samples and high cost of acquisition, and uses the collaborative training of data in two modalities to obtain deep information between modalities, uses limited labeled data and a large amount of unlabeled data to establish the relationship between the two modalities of data, improves the accuracy of relation recognition, and can be applied to visual question answering, knowledge reasoning and other tasks. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 An illustrative diagram is defined for the image-text relation extraction of the present application.

[0063] Figure 2 A flowchart of the semi-supervised image-text relation extraction method based on collaborative training in the embodiment of the present application.

[0064] Figure 3 An illustrative diagram of the semi-supervised image-text relation extraction framework based on collaborative training in the embodiment of the present application.

[0065] Figure 4 An illustrative diagram of the view construction module in the training of the semi-supervised image-text relation extraction method based on collaborative training in the embodiment of the present application.

[0066] Figure 5 An illustrative diagram of the image-text collaborative classification module in the training of the semi-supervised image-text relation extraction method based on collaborative training in the embodiment of the present application.

[0067] Figure 6 An internal structure diagram of the semi-supervised image-text relation extraction system based on collaborative training in the embodiment of the present application. DETAILED DESCRIPTION

[0068] As Figure 1As shown, the application is a semi-supervised graph-text relation extraction method and system based on collaborative training, which can extract entity pairs and their relationships in the image.

[0069] In order to make the purpose, characteristics and advantages of the present application more obvious and easy to understand, the embodiments of the present application will be clearly and completely described below in combination with the specific implementation modes involved in the accompanying drawings.

[0070] The specific embodiment of the present application provides a semi-supervised graph-text relation extraction method based on collaborative training, and the flowchart of the semi-supervised graph-text relation extraction method based on collaborative training is as follows Figure 2 , the method comprises the following steps:

[0071] Step S11: inputting an image, identifying a plurality of entity candidate regions and related information of the entities through a target detection network;

[0072] Step S12: combining the candidate regions and the entities two by two, sorting the scores of the overlapping areas of the entities in the entity pairs, and screening the candidate suggested entity pairs with scores higher than the threshold value;

[0073] Step S13: obtaining the multi-modal features of the entity pairs based on the candidate suggested entity pairs through feature encoding, including: entity pair relative position features, joint visual features of the entity pairs, and entity pair semantic features;

[0074] Step S14: based on the multi-modal features of the entity pairs, constructing an image view by multiplying the joint visual features of the entity pairs and the entity pair relative position features, and constructing a semantic view by multiplying the entity pair semantic features and the entity pair relative position features;

[0075] Step S15: training the image classifier and the semantic classifier under the constructed image view and semantic view, constantly training and updating the network by calculating the loss and back propagation, so as to output the visual relationship between each candidate entity pair.

[0076] In step S11, the candidate region of the entity and the related information of the entity are specifically:

[0077] The entity candidate region is the coordinate information of the entity, represented as (x, y, w, h), x and y represent the center point coordinates of the region, and w and h are the width and height of the region respectively; the related information of the entity includes the visual features of the region where the entity is located and the corresponding label of the entity obtained according to the visual features of the region where the entity is located.

[0078] In step S12, the information of the candidate suggested entity pair is specifically:

[0079] The candidate suggestion entity pair is an entity pair that can have a relationship, the entity pair is a subject and an object, and the entity pair having a relationship has directionality; the candidate suggestion entity pair related information includes coordinate information of the entity pair and a joint region of the entity pair, visual features of the entity pair and the joint region of the entity pair, and entity region corresponding labels, the coordinate information of the entity pair and the joint region of the entity pair is represented as (x s , y s , w s , h s ), (x o , y o , w o , h o ) and (x u , y u , w u , h u ), wherein s represents the subject, o represents the object, and u represents the joint region of the subject and the object.

[0080] In step S13, the multi-modal features of the entity pair are specifically:

[0081] The relative position features of the entity pair, the joint visual features of the entity pair, and the semantic features of the entity pair; the coordinate information of the candidate entity pair and the joint region of the candidate entity pair is encoded through two fully connected layers to obtain the relative position features of the entity pair The joint visual features of the entity pair are encoded through two fully connected layers to obtain the joint visual features of the entity pair The pre-trained language model word2vec obtained by using the wikidata knowledge base, so that the word vector representation of the entity label can be obtained, the subject and the object in the entity pair are converted into word vector representations, and the representation obtained by connecting the two word vector representations at the head and tail is encoded through two fully connected layers to obtain the semantic features of the entity pair

[0082] In step S14, the image view and the semantic view are specifically:

[0083] The joint visual features of the entity pair and the relative position features of the entity pair in the multi-modal features are multiplied to construct the image view, and the feature representation of the data in the image view is obtained; the semantic features of the entity pair and the relative position features of the entity pair in the multi-modal features are multiplied to construct the semantic view, and the feature representation of the data in the semantic view is obtained.

[0084] In step S15, the image classifier and the semantic classifier are specifically:

[0085] The image classifier and the semantic classifier are both perceptual machines composed of two fully connected layers of neural networks.

[0086] The present invention provides a semi-supervised graph-text relationship extraction network based on collaborative training.

[0087] Figure 3 This is a schematic diagram of a semi-supervised graph-text relationship extraction framework based on collaborative training, including: object detection module, entity pair proposal module, multimodal feature extraction module, and view construction module (see reference). Figure 4 ) and the image-text collaborative classification module (see reference) Figure 5 The target detection module obtains target entity information from the input image and transmits it to the entity pair suggestion module to obtain candidate suggested entity pair information. The multimodal feature extraction module obtains multimodal features of the entity pairs based on the entity pair information. The multimodal feature extraction module constructs an image view and a semantic view based on the multimodal features. Finally, the image-text collaborative classification module performs relationship discrimination between the inputs in the image view and the semantic view and the corresponding image classifier and semantic classifier.

[0088] Combination Figure 3 The present invention provides a training process for a semi-supervised graph-text relationship extraction method based on collaborative training, specifically as follows:

[0089] S101, Input the training dataset, and split the dataset into labeled datasets at a ratio of 1:10. and unlabeled datasets The two datasets were then sampled and divided into the same batch.

[0090] S102, the first batch of labeled datasets The data from the unlabeled dataset and the data from the unlabeled dataset were respectively processed by the multimodal feature extraction module to obtain multimodal features, including: joint visual features of entity pairs. Entity pairs semantic features Relative position features of entities The feature representation of the image view is obtained through the view construction module. Feature representation under semantic view

[0091] S103, label the dataset Feature representation in image view The feature representation in the semantic view is input into the image classifier f1 under the image-text collaborative classification module. The semantic classifier f2 under the image-text collaborative classification module is input to obtain the relationship prediction scores between entity pairs. After passing through the softmax function, normalized relationship prediction scores 1 and 2 are obtained. The cross-entropy loss between the relationship label and relationship score 1 and the cross-entropy loss between the relationship label and relationship score 2 are calculated. The sum of these two losses is used as the classification loss. SeeFigure 5 The arrow points to ①.

[0092] S104, the unmarked data set In the image view under the feature representation input the image classifier under the image-text collaborative classification module, in the semantic view under the feature representation input the semantic classifier under the image-text collaborative classification module, respectively get the relationship prediction score between entity pairs, get the normalized relationship prediction score 3 and relationship prediction score 4 through the softmax function, use JS divergence to measure the similarity of prediction score 3 and relationship prediction score 4 as collaborative loss See Figure 5 The arrow points to ② (left).

[0093] S105, the labeled data set In the image view under the feature representation input the semantic classifier under the image-text collaborative classification module, in the semantic view under the feature representation input the image classifier under the image-text collaborative classification module, respectively get the relationship prediction score between entity pairs, get the normalized relationship prediction score 5 and relationship prediction score 6 through the softmax function, calculate the cross entropy loss of relationship label and relationship score 5 and the cross entropy loss of relationship label and relationship score 6, calculate the sum of the two parts as the modal cross loss of the labeled data set See Figure 5 The arrow points to ② (right).

[0094] S106, the unmarked data set In the image view under the feature representation input the semantic classifier under the image-text collaborative classification module, in the semantic view under the feature representation input the image classifier under the image-text collaborative classification module, respectively get the relationship prediction score between entity pairs, get the normalized relationship prediction score 7 and relationship prediction score 8 through the softmax function, calculate the cross entropy loss of relationship prediction score 7 and relationship score 3 and the cross entropy loss of relationship prediction score 8 and relationship score 4, calculate the sum of the two parts as the modal cross loss of the unmarked data set See Figure 5 The arrow points to ③.

[0095] S107, calculate the classification loss Collaborative loss Modal cross loss on the labeled data set And the modal cross loss on the unmarked data set The sum of the two as the total loss Calculate the gradient, and update the parameters of the model through back propagation, enter the processing of the next batch of data.

[0096] ​​S108, repeating steps S102-S107 until all batch data is processed, updating the learning rate, at which time the data is resampled according to step S101, and enters the next round of iteration until the maximum number of iterations is reached.

[0097] In combination Figure 3 For the corresponding training process, the specific embodiments are as follows:

[0098] Initialize the model, the learning rate is a, use Adam optimizer, the maximum number of iterations is T.

[0099] According to step S101, input the training data set, divide the data set into labeled data set and unlabeled data set at a ratio of 1:10 And Use the same sampling batch b, wherein, Indicates the labeled data set, Indicates the unlabeled data set, m>>n, m represents the number of relationships in the unlabeled data set, and n represents the number of relationships in the labeled data set,

[0100] According to step S102, input the images in And Into the multi-modal feature extraction module respectively, identify the entities in each image, and combine them into entity pairs (entity pairs are directional, that is, <entity 1, relationship, entity 2>≠<entity 2, relationship, entity 1>), according to the size of the intersection of the entity pair position frame, filter out the candidate entity pairs, and according to the frame of the candidate entity pair and the visual features of the region where the frame is located, obtain the multi-modal features: the joint visual features of the entity pair Entity semantic features And the relative position features of the entity pair Align the three features through two fully connected layers to obtain the final feature representation And

[0101] Input And Into the view construction module, use And To get the feature representation under the image view Use And To get the feature representation under the semantic view

[0102] Wherein, The visual features of the joint bounding box of the individual bounding boxes of two entities are obtained using VGG16 Faster R-CNN. Word vector representations of the two entity categories are then obtained using word2vec. This is obtained by concatenating the word vectors of two entities. The coordinates of the two entity bounding boxes and their joint bounding box are obtained by encoding the bounding boxes of the two entities and their joint bounding box, where the coordinates of the two entity bounding boxes and their joint bounding box are represented as (x... s y s w s h s ), (x o y o w o h o ), (x u y u w u h u ), (x s y s (w) represents the center coordinates of the region where the subject s is located. s h s (x) represents the width and height of the region containing the main body s, (x) o y o (w) represents the center coordinates of the region where object o is located. o h o (x) represents the width and height of the region containing object o, (x) u y u (w) represents the center coordinates of the joint region of subject s and object o. u h u The area of ​​the joint region of subject s and object o is represented by s, and the area of ​​the border of each entity is represented by S. s S o The area of ​​an entity relative to a joint region is represented by S. u The positional features of the two entities and They are respectively Location feature representation of the joint region for This is obtained by connecting the three positional features end to end, i.e.

[0103] According to step S103, image classifier f1 and semantic classifier f2 are initialized. Both f1 and f2 are perceptrons composed of two fully connected neural networks. Feature representation in image view Enter F1 to Feature representation under semantic view The f2 is input respectively, and the relation prediction score 1 and the relation prediction score 2 are obtained after normalization by a softmax function: The classification loss is calculated:

[0104]

[0105] In the formula, Indicates The multimodal feature representation of the entity pair in the image view, y j Indicates the relation label of the corresponding entity pair, Indicates the cross-entropy, Indicates the relation prediction score 1, Indicates the relation prediction score 2, y j Indicates x j The corresponding label.

[0106] According to step S104, the f1 is input respectively, and the relation prediction score 3 and the relation prediction score 4 are obtained after normalization by a softmax function: The feature representation in the image view The f2 is input respectively, and the relation prediction score 3 and the relation prediction score 4 are obtained after normalization by a softmax function: The feature representation in the semantic view The f2 is input respectively, and the relation prediction score 3 and the relation prediction score 4 are obtained after normalization by a softmax function: The collaborative loss is calculated:

[0107]

[0108] In the formula, Indicates The multimodal feature representation of the entity pair in the image view, Indicates the cross-entropy, Indicates the relation prediction score 3, Indicates the relation prediction score 4.

[0109] According to step S105, the f1 is input respectively, and the relation prediction score 5 and the relation prediction score 6 are obtained after normalization by a softmax function: The feature representation in the image view The f2 is input respectively, and the relation prediction score 5 and the relation prediction score 6 are obtained after normalization by a softmax function: The feature representation in the semantic view The f1 is input respectively, and the relation prediction score 5 and the relation prediction score 6 are obtained after normalization by a softmax function: The modal cross loss on is calculated:

[0110]

[0111] In the formula, Indicates The multimodal feature representation of the entity pair in the image view, y j Indicates the relation label of the corresponding entity pair, Indicates the cross-entropy, representing the relation prediction score 6, representing the relation prediction score 5, y j representing x j corresponding label.

[0112] According to step S106, the feature representation under the image view input f2, the feature representation under the semantic view respectively input f1, and after normalization by the softmax function, the relation prediction score 7 and the relation prediction score 8 are obtained: calculate the modal cross loss on

[0113]

[0114] In the formula, representing the multi-modal feature representation of the entity pair in representing the cross entropy, representing the relation prediction score 8, representing the relation prediction score 7, representing the relation prediction score 3, representing the relation prediction score 4

[0115] According to step S107, the total loss is calculated, and the gradient is calculated to update the parameters by back propagation.

[0116] According to step S108, steps S102-S107 are repeated until all batch data is processed, the learning rate a is updated, at this time the data is resampled according to step S101, and enters the next round of iteration until the maximum iteration number T is reached.

[0117] The embodiment of the present application provides a semi-supervised image-text relation extraction system S20 based on collaborative training, referring to Figure 6 , specifically:

[0118] S21: an image target detection module, configured to obtain at least one image data to be judged, and identify a plurality of candidate regions of entities through a target detection network, extract visual features of the candidate regions to the regions through a convolution network, and finally obtain corresponding labels of the entities of the regions through entity classification according to the visual features of the regions;

[0119] S22: an entity pair suggestion module, configured to filter candidate suggestion entity pairs that may have relations, combine the corresponding entities of the candidate regions two by two, sort scores of overlapping areas of the candidate regions of each entity between the entity pairs, and filter the candidate suggestion entity pairs with scores higher than a threshold value.

[0120] S23: a multi-modal feature extraction module, configured to obtain multi-modal features of the candidate suggestion entity pair after feature encoding, including: entity pair relative position features, joint visual features of the entity pair, and entity pair semantic features;

[0121] S24: a view construction module, configured to obtain feature representations in an image view and feature representations in a semantic view from the obtained multi-modal features, and construct the image view by multiplying the joint visual features of the entity pair and the entity pair relative position features, and construct the semantic view by multiplying the entity pair semantic features and the entity pair relative position features;

[0122] S25: a graph-text relationship discrimination module, configured to discriminate the relationship between the entity pairs, and discriminate the relationship between the candidate entity pairs according to output relationship scores in an image classifier in the image view and in a semantic classifier in the semantic view, and take the score as the final predicted relationship.

Claims

1. A semi-supervised graph-text relationship extraction system based on collaborative training, characterized in that, Includes the following modules: S21: Image target detection module, used to acquire image data and identify candidate regions of multiple entities, visual features of the regions where the entities are located, and the corresponding labels of the entities through the target detection network; S22: Entity pair suggestion module, used to filter candidate suggestion entity pairs that may have a relationship. It combines entity pairs in the candidate region, sorts the scores of the overlapping area of ​​each entity in the candidate region between the entity pairs, and selects those with scores higher than the threshold as candidate suggestion entity pairs. S23: Multimodal feature extraction module, used to obtain multimodal features of candidate entity pairs after feature encoding, including: relative position features of entity pairs, joint visual features of entity pairs, and semantic features of entity pairs; S24: View construction module, used to obtain feature representations under image view and feature representations under semantic view from the acquired multimodal features. It constructs image view by multiplying the joint visual features of entity pairs with the relative position features of entity pairs, and constructs semantic view by multiplying the semantic features of entity pairs with the relative position features of entity pairs. S25: Image-text relationship discrimination module, used to discriminate the relationship between entity pairs. In the image classifier under the image view, the relationship between candidate entity pairs is discriminated based on the output relationship score, and in the semantic classifier under the semantic view, the relationship between candidate entity pairs is discriminated based on the output relationship score. The higher score is taken as the final predicted relationship.

2. A method for performing semi-supervised graph-text relationship extraction based on collaborative training using the semi-supervised graph-text relationship extraction system based on collaborative training as described in claim 1, characterized in that, Includes the following steps: S11: Input image, and identify candidate regions of multiple entities and related information of the entities through object detection network; S12: Combine the entities corresponding to the candidate regions in pairs, sort the scores of the overlapping areas of the candidate regions of each entity in the entity pair, and select the entity pairs with scores higher than the threshold as candidate suggested entity pairs. S13: Based on candidate proposed entity pairs, multimodal features of entity pairs are obtained through feature encoding, including: relative position features of entity pairs, joint visual features of entity pairs, and semantic features of entity pairs. Multimodal features of entity pairs are obtained based on candidate proposed entity pairs and their related information. S14: Based on the multimodal features of entity pairs, construct an image view by multiplying the joint visual features of entity pairs with the relative position features of entity pairs, and construct a semantic view by multiplying the semantic features of entity pairs with the relative position features of entity pairs. S15: Train the image classifier and semantic classifier under the constructed image view and semantic view. Continuously train and update the network by calculating the loss and backpropagation, thereby outputting the visual relationship between each candidate entity pair.

3. The method according to claim 2, characterized in that, In step S11, the candidate region of the entity and the relevant information of the entity are specifically as follows: Entity candidate region: The coordinate information of the entity, represented as (x, y, w, h), where x and y represent the coordinates of the center point of the region, and w and h are the width and height of the region, respectively; Entity-related information: including the visual features of the area where the entity is located, and the corresponding label of the entity obtained based on the visual features of the area where the entity is located.

4. The method according to claim 2, characterized in that, In step S12, the information of the candidate suggested entity pair is specifically as follows: Candidate entity pairs are entity pairs that may have a relationship. Each entity pair consists of a subject and an object, and there is a directionality between the related entity pairs. The relevant information for candidate entity pairs includes the coordinates of the entity pair and its joint region, the visual features of the entity pair and its joint region, and the corresponding labels for the entity regions. The coordinates of the entity pair and its joint region are represented as (x...). s y s w s h s ), (x o y o w o h o ) and (x u y u w u h u ), where s represents the subject, o represents the object, and u represents the joint region of the subject and the object.

5. The method according to claim 2, characterized in that, In step S13, the multimodal features of the entity pair include: Entity pair relative position features The coordinate information of candidate entity pairs and their joint regions is obtained by encoding through two fully connected layers; Joint visual features of entity pairs The visual features of the joint region of entity pairs are obtained by encoding them through two fully connected layers; Entity pairs semantic features By using the pre-trained language model word2vec obtained from the wikidata knowledge base, word vector representations of entity labels can be obtained. The subject and object in the entity pair are converted into word vector representations, and the representation obtained by concatenating the two word vector representations is encoded through two fully connected layers.

6. The method according to claim 2, characterized in that, In step S14, constructing the image view and semantic view based on the multimodal features of entity pairs specifically involves: The image view is constructed by multiplying the joint visual features of entity pairs in the multimodal features with the relative position features of entity pairs, thus obtaining the feature representation of the data in the image view; A semantic view is constructed by multiplying the semantic features of entity pairs with the relative position features of entity pairs in the multimodal features, thus obtaining the feature representation of the data under the semantic view.

7. The method according to claim 2, characterized in that, In step S15, the image classifier and semantic classifier under the constructed image view and semantic view include: The image classifier is a perceptron consisting of a neural network with two fully connected layers; The semantic classifier is a perceptron consisting of a neural network with two fully connected layers.

8. The method according to claim 2, characterized in that, The training process includes the following steps: Step S101: Input the training dataset and split it into labeled datasets at a ratio of 1:

10. and unlabeled datasets The two datasets were then sampled and divided into the same batch. Step S102: The first batch of labeled datasets... The data from the unlabeled dataset and the data from the unlabeled dataset were respectively processed by the multimodal feature extraction module to obtain multimodal features, including: joint visual features of entity pairs. Entity pairs semantic features Relative position features of entities The feature representation under the image view is obtained through the view construction module. Feature representation under semantic view Step S103: Label the dataset Feature representation in image view The feature representation in the semantic view is input into the image classifier f1 under the image-text collaborative classification module. The semantic classifier f2 under the image-text collaborative classification module is input to obtain the relationship prediction scores between entity pairs. After passing through the softmax function, normalized relationship prediction scores 1 and 2 are obtained. The cross-entropy loss between the relationship label and relationship score 1 and the cross-entropy loss between the relationship label and relationship score 2 are calculated. The sum of these two losses is used as the classification loss. Step S104: Transfer the unlabeled dataset The feature representations in the image view are input into the image classifier of the image-text collaborative classification module, and the feature representations in the semantic view are input into the semantic classifier of the image-text collaborative classification module. Relationship prediction scores between entity pairs are obtained respectively. After passing through the softmax function, normalized relationship prediction scores 3 and 4 are obtained. The similarity between relationship prediction scores 3 and 4 is measured using JS divergence as the collaborative loss. Step S105: Label the dataset The feature representations under the image view are input into the semantic classifier of the image-text collaborative classification module, and the feature representations under the semantic view are input into the image classifier of the image-text collaborative classification module. Relationship prediction scores between entity pairs are obtained respectively. After passing through the softmax function, normalized relationship prediction scores 5 and 6 are obtained. The cross-entropy loss between the relationship label and relationship score 5, and the cross-entropy loss between the relationship label and relationship score 6 are calculated. The sum of these two losses is used as the labeled dataset. Modal crossover loss Step S106: Transfer the unlabeled dataset The feature representations under the image view are input into the semantic classifier of the image-text co-classification module, and the feature representations under the semantic view are input into the image classifier of the image-text co-classification module. Relationship prediction scores between entity pairs are obtained, and after passing through the softmax function, normalized relationship prediction scores 7 and 8 are obtained. The cross-entropy loss between relationship prediction scores 7 and 3, and between relationship prediction scores 8 and 4 are calculated. The sum of these two losses is used as the unlabeled dataset. Modal crossover loss Step S107: Calculate the classification loss Collaborative loss Modal crossover loss on labeled datasets Modal crossover loss on unlabeled datasets The sum of these is the total loss. Calculate the gradient and update the model parameters through backpropagation before proceeding to the next batch of data processing; Step S108: Repeat steps S102-S107 until all batches of data have been processed, update the learning rate, and then resample the data according to step S101 and enter the next round of iteration until the maximum number of iterations is reached.

9. The method according to claim 8, characterized in that, Total loss The definition is as follows: in, For classifying losses, To mitigate losses, For modal crossover loss on the labeled dataset, For modal crossover loss on unlabeled datasets, the loss is... The definition is as follows: in, express Multimodal feature representation of entity pairs, y j Represents the relational tags of corresponding entity pairs. Represents cross-entropy. express Multimodal feature representation of entity pairs in the middle This indicates a relationship prediction score of 1. The relationship prediction score is 2. The relationship prediction score is 3. The relationship prediction score is 4. The relationship prediction score is 5. The relationship prediction score is 6. The relationship prediction score is 7. The relationship prediction score is 8. The definition is as follows: Where f1 represents the image classifier and f2 represents the semantic classifier. This represents the feature representation of the labeled dataset in the image view. This represents the feature representation of the labeled dataset in the semantic view. This represents the feature representation of the unlabeled dataset in the image view. This represents the feature representation of the unlabeled dataset in the semantic view.

Images