A scene graph-based image text retrieval method and system

Abstract

The application discloses a kind of scene graph-based image text retrieval method and system.It includes: input image and text;Feature extraction and feature fusion are carried out to image, and visual scene graph is constructed, is learned using graph attention network and is updated;Word feature and semantic triple feature extraction are carried out to text, and text scene graph is constructed, is learned using graph attention network and is updated, and visual scene graph is enhanced using text scene graph;Overall similarity score is calculated using text scene graph and visual scene graph, global-local visual-text scene graph similarity measurement model is constructed, model is trained using loss function until function converges, and the model of convergence is obtained;User inputs query object into model, and obtains matching result.The application can make full use of global information of image, fine-grained information in image and text and interaction and multi-modal knowledge relationship between them, improve retrieval accuracy, and expand retrieval application scenarios.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and multi-modal retrieval, and in particular to an image-text retrieval method and system based on a scene graph. BACKGROUND

[0002] With the rapid development of digitization, the generation and storage of image and text data have shown an explosive growth trend. At the same time, traditional information retrieval methods have been unable to meet people's demand for efficient and accurate retrieval, especially in the field of multi-modal information retrieval, the challenge is more prominent.

[0003] Under such a background, image-text retrieval as an important information retrieval task has attracted much attention. It aims to effectively retrieve relevant image or text information from a large image and text database to provide personalized and accurate retrieval services for users. Existing technology applications mainly map images and texts into a unified feature space for matching, which is often only suitable for relatively simple images and texts, and is difficult to capture more fine-grained information in images and texts, or only uses regional information of images without considering the interaction between entities / objects in images and texts, resulting in low retrieval performance. Recently, some research works have constructed scene graphs to realize image-text matching. These technologies mainly construct scene graphs of images and texts, and then compare the similarity of the two scene graphs to perform image-text matching. However, this kind of technology often only uses single modal information when processing image and text information, without considering the fusion of other modal information for feature enhancement, which affects the accuracy and diversity of retrieval.

[0004] One of the current existing technologies is an image-text retrieval method in the paper "Dual-path convolutional image-text embeddings with instance loss" which encodes images and texts into global vectors in the same space. This technology builds an end-to-end dual-path convolutional network to learn the representation of images and texts, and introduces an instance loss to solve the matching problem between images and texts. The disadvantage of this method is that this technical solution is only suitable for simple cases where there is only one object in the image or only one short sentence in the text, because it ignores the importance of more fine-grained information in images and texts, such as the relationship between objects and interaction, which limits its applicability in certain application scenarios.

[0005] The second prior art is a fine-grained image-text matching method in the paper "Stacked cross attention for image-text matching", which captures the fine-grained interaction between images and texts by stacking cross-attention to infer the potential semantic alignment between objects or other salient regions and corresponding words, and infers image-text similarity to achieve image-text retrieval. The disadvantage of this method is that this method only uses the region information of the image, does not consider the interaction between entities / objects in the image and the text, and ignores the multi-modal knowledge relationship between the image and the text. SUMMARY

[0006] The purpose of the present application is to overcome the shortcomings of the prior art, and a scene graph-based image-text retrieval method and system are proposed. The main problem solved by the present application is how to comprehensively and fully utilize the global information of the image, the fine-grained information in the image and the text, and the interaction and multi-modal knowledge relationship between the two, to improve the retrieval accuracy and expand the application scenarios of retrieval.

[0007] In order to solve the above problems, the present application proposes a scene graph-based image-text retrieval method, which comprises:

[0008] Input the image and text from the retrieval data set, pre-process the image to obtain an image of uniform size, and divide the retrieval data set into a training set and a test set;

[0009] Feature extraction is performed on the image of uniform size, and feature fusion is performed using a fully connected network to obtain object multi-modal fusion features and object relationship multi-modal fusion features. A visual scene graph is constructed with the object multi-modal fusion features as nodes and the object relationship multi-modal fusion features as edges. A graph attention network is used to learn and update the visual scene graph to obtain an updated visual scene graph;

[0010] Word features and semantic triple features are extracted from the text. A text scene graph is constructed with the word features as nodes and the semantic triple features as edges. A graph attention network is used to learn and update the text scene graph to obtain an updated text scene graph. The updated text scene graph is used to enhance the updated visual scene graph to obtain a text-enhanced visual scene graph;

[0011] The updated text scene graph and the text-enhanced visual scene graph are used to calculate the overall similarity score, and a global-local visual-text scene graph similarity measurement model is constructed. The global-local visual-text scene graph similarity measurement model is trained using a loss function until the function converges, and a converged global-local visual-text scene graph similarity measurement model is obtained;

[0012] The user inputs a query object into the converged global-local visual-textual scene graph similarity measurement model to obtain a matching result.

[0013] Preferably, the image and the text are input from the retrieval data set, the image is preprocessed to obtain an image of uniform size, and the retrieval data set is divided into a training set and a test set, specifically:

[0014] The retrieval data set is composed of images and texts and is stored in the form of image-text pairs, each image has one or more corresponding texts for describing the content of the image;

[0015] The image is preprocessed by scaling adjustment to a preset image size;

[0016] The data set is divided into a training set and a test set.

[0017] Preferably, the feature extraction is performed on the image of uniform size, and the feature fusion is performed by using a fully connected network to obtain object multi-modal fusion features and object relationship multi-modal fusion features, the visual scene graph is constructed by taking the object multi-modal fusion features as nodes and the object relationship multi-modal fusion features as edges, the graph attention network is learned and updated to obtain an updated visual scene graph, and specifically:

[0018] Preferably, the word feature and the semantic triple feature are extracted from the text, the text scene graph is constructed by taking the word feature as a node and the semantic triple feature as an edge, the graph attention network is learned and updated to obtain an updated text scene graph, and the text-enhanced visual scene graph is obtained by enhancing the updated visual scene graph with the updated text scene graph, and specifically:

[0019] Preferably, the overall similarity score is calculated by using the updated text scene graph and the text-enhanced visual scene graph, the global-local visual-textual scene graph similarity measurement model is constructed, the global-local visual-textual scene graph similarity measurement model is trained by using a loss function until the function converges, and a converged global-local visual-textual scene graph similarity measurement model is obtained, and specifically:

[0020] Preferably, the user inputs a query object into the converged global-local visual-textual scene graph similarity measurement model to obtain a matching result, and specifically:

[0021] Correspondingly, the application also provides an image text retrieval system based on a scene graph, which comprises:

[0022] A preprocessing unit is configured to input an image and a text from a retrieval data set, pre-process the image to obtain an image of uniform size, and divide the retrieval data set into a training set and a test set.

[0023] The global-local visual-text scene graph similarity measurement model construction unit is configured to perform feature extraction on the unified size image, perform feature fusion by using a full connection network, obtain object multi-modal fusion features and object relationship multi-modal fusion features, construct a visual scene graph by taking the object multi-modal fusion features as nodes and the object relationship multi-modal fusion features as edges, learn and update the visual scene graph by using a graph attention network, and obtain an updated visual scene graph; perform word feature and semantic triple feature extraction on the text, construct a text scene graph by taking the word features as nodes and the semantic triple features as edges, learn and update the text scene graph by using a graph attention network, and obtain an updated text scene graph; enhance the updated visual scene graph by using the updated text scene graph, calculate an overall similarity score by using the updated text scene graph and the text enhanced visual scene graph, construct a global-local visual-text scene graph similarity measurement model, train the global-local visual-text scene graph similarity measurement model by using a loss function until the function converges, and obtain a converged global-local visual-text scene graph similarity measurement model;

[0024] The query retrieval unit is configured to input a query object into the converged global-local visual-text scene graph similarity measurement model by a user, and obtain a matching result.

[0025] The present application has the following beneficial effects:

[0026] The present application can promote a deeper understanding of image and text queries and help enhance image-text retrieval by constructing scene graphs of images and texts to capture the interactions between objects and the objects in the images and the texts. The present application can improve the accuracy of retrieval by fusing label semantic information and text semantic information in the construction of a visual scene graph, fusing visual information in the construction of a text scene graph, realizing the interactions between different modalities, and mining hidden multi-modal knowledge relationships. The present application can more comprehensively evaluate the matching degree between images and texts by calculating the similarity between visual scene graphs and text scene graphs from two angles, and thus improve the performance of image-text retrieval tasks. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 is a flowchart of an image-text retrieval method based on a scene graph according to an embodiment of the present application;

[0028] Figure 2 is a generation flowchart of a visual scene graph according to an embodiment of the present application;

[0029] Figure 3 is a generation flowchart of a text scene graph according to an embodiment of the present application;

[0030] Figure 4is a structural diagram of an image-text retrieval system based on a scene graph according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0032] Figure 1 is a flowchart of an image-text retrieval method based on a scene graph according to an embodiment of the present application, as shown in the figure, the method comprises the following steps: Figure 1

[0033] S1, input images and texts from a retrieval data set, pre-process the images to obtain images of a uniform size, and divide the retrieval data set into a training set and a test set;

[0034] S2, perform feature extraction on the images of the uniform size, perform feature fusion by using a fully connected network, obtain object multi-modal fusion features and object relationship multi-modal fusion features, construct a visual scene graph by taking the object multi-modal fusion features as nodes and the object relationship multi-modal fusion features as edges, learn and update the visual scene graph by using a graph attention network, and obtain an updated visual scene graph;

[0035] S3, perform word feature and semantic triple feature extraction on the texts, construct a text scene graph by taking the word features as nodes and the semantic triple features as edges, learn and update the text scene graph by using a graph attention network, obtain an updated text scene graph, and enhance the updated visual scene graph by using the updated text scene graph, and obtain a text-enhanced visual scene graph;

[0036] S4, calculate an overall similarity score by using the updated text scene graph and the text-enhanced visual scene graph, construct a global-local visual-text scene graph similarity measurement model, train the global-local visual-text scene graph similarity measurement model by using a loss function until the function converges, and obtain a converged global-local visual-text scene graph similarity measurement model;

[0037] S5, a user inputs a query object into the converged global-local visual-text scene graph similarity measurement model, and obtains a matching result.

[0038] Step S1 is specifically as follows:

[0039] ​S1-1, the retrieved dataset consists of images and texts, stored in the form of image-text pairs, each image has one or more texts corresponding to it, which are used to describe the content of the image;

[0040] S1-2, pre-process the images by scaling adjustment to a preset image size.

[0041] Step S2, generate a visual scene graph, as shown in Figure 2 , specifically as follows:

[0042] S2-1, detect the regions of objects in the uniform-sized images using Faster-R-CNN, and obtain the position coordinates of all objects; for each detected object, use CLIP to encode the object region into a visual feature, then map the visual feature and the position coordinates through two linear layers into a high-dimensional space, and add the two features to obtain an initialized object visual feature; for any two objects, generate a correlation region covering both, obtain the relationship position coordinates of the region, use CLIP to encode the correlation region into a relationship visual feature, then map the relationship visual feature and the relationship position coordinates through two linear layers into a high-dimensional space, and add the two features to obtain an initialized object relationship visual feature;

[0043] S2-2, extract the categories, attributes and relationships between objects of the detected objects in the uniform-sized images using Faster-R-CNN, to obtain initial object semantic labels and initial object relationship semantic labels, and use fast-Text to extract features of the two types of labels respectively and map them through linear layers into a high-dimensional space to obtain object semantic features and object relationship semantic features;

[0044] S2-3, use a fully connected network to fuse the features through vector splicing, connect the object visual features and the object semantic features, and the object relationship visual features and the object relationship semantic features, to obtain object multi-modal fusion features and object relationship multi-modal fusion features;

[0045] S2-4, update the nodes and edges of the visual scene graph respectively to obtain an updated visual scene graph, that is, for the nodes of the visual scene graph, update the nodes of the visual scene graph through a fully connected layer according to the information of the previous layer itself; for the edges of the visual scene graph, update the edges of the visual scene graph through a fully connected layer in combination with the information of the previous layer itself and the information of the neighbor nodes of the previous layer.

[0046] Step S3, generate a text scene graph, as shown in Figure 3 , specifically as follows:

[0047] S3-1, extract the semantic triplets of the text using the SPICE method;

[0048] S3-2, mapping each word of the text and the semantic triplets to a high-dimensional space and representing them as dense real-valued vectors by GloVe word embedding model, obtaining word embeddings of the words and word embeddings of the semantic triplets;

[0049] S3-3, inputting the word embeddings and the semantic triplets embeddings into two BiLSTM models respectively, obtaining forward and backward features of the words and the semantic triplets and taking average values, obtaining word features and semantic triplet features;

[0050] S3-4, constructing a preliminary text scene graph with the word features as nodes and the semantic triplet features as edges;

[0051] S3-5, mapping the features of the updated visual scene graph to semantic features by a learnable linear layer, and enhancing the preliminary text scene graph by vector connection, obtaining an enhanced text scene graph;

[0052] S3-6, learning and updating the features of the enhanced text scene graph by a graph attention network, obtaining an updated text scene graph;

[0053] S3-7, mapping the semantic features back to the features of the updated visual scene graph by a learnable linear layer, and enhancing the features of the updated visual scene graph by vector connection with the features of the updated text scene graph, obtaining a text-enhanced visual scene graph.

[0054] Step S4 is as follows:

[0055] S4-1, performing local similarity measurement on the updated text scene graph and the text-enhanced visual scene graph, i.e. matching each node feature and each edge feature of the updated text scene graph and the text-enhanced visual scene graph respectively; for a node, calculating the matching score between it and the related object in the image and taking the average value as the node score; for an edge, calculating the score between it and the related matching edge in the image and taking the average value as the edge score, and adding the node score and the edge score to obtain the local similarity score;

[0056] S4-2, converting the updated text scene graph and the text-enhanced visual scene graph into text scene graph embedding and visual scene graph embedding by a learnable linear layer, and performing global similarity measurement on the text scene graph embedding and the visual scene graph embedding, i.e. calculating the cosine distance of the vectors of the text scene graph embedding and the visual scene graph embedding, obtaining the global similarity score;

[0057] S4-3, adding the local similarity score and the global similarity score to obtain a total similarity score, and constructing a global-local visual-text scene graph similarity measurement model;

[0058] S4-4, training the global-local visual-text scene graph similarity measurement model using an improved triplet loss function in which a negative sample is replaced by a most difficult negative sample, so that the global similarity score of similar image-text pairs is higher than that of dissimilar image-text pairs, and the loss function includes two parts: one part is for real image-text pairs, and the other part is for the most difficult negative triplet, and the expression is:

[0059]

[0060] wherein m is a boundary parameter, k and l represent corresponding image-text pairs, and k and represent a mismatch pair, and l also represent a mismatch pair;

[0061] S4-5, stopping training when the loss function converges to obtain a converged global-local visual-text scene graph similarity measurement model.

[0062] Step S5, specifically as follows:

[0063] S5-1, a user inputs a query object into the converged global-local visual-text scene graph similarity measurement model, constructs a corresponding scene graph and matches it with a relative scene graph in the model, and calculates a total similarity score, that is, if the user input is an image, a visual scene graph corresponding to the input image and a text scene graph of the text in the test set are constructed to obtain a plurality of visual-text scene graph pairs, and a total similarity score is calculated for each visual-text scene graph pair; if the user input is text, a text scene graph of the input text and a visual scene graph of the image in the test set are constructed to obtain a plurality of text-scene graph pairs, and a total similarity score is calculated for each text-scene graph pair;

[0064] S5-2, sorting the total similarity scores, and the result with a higher total similarity score is a more relevant answer retrieved and output as a matching result.

[0065] Correspondingly, the application also provides an image-text retrieval system based on a scene graph, as shown in the accompanying drawings, comprising: Figure 4

[0066] A preprocessing unit 1 is used for inputting images and texts from a retrieval data set, pre-processing the images to obtain images of a uniform size, and dividing the retrieval data set into a training set and a test set;

[0067] ​Specifically, the retrieval dataset is composed of images and texts, and is stored in the form of image-text pairs, each image having one or more texts corresponding thereto for describing the content of the image;

[0068] The image is preprocessed by scaling adjustment to a preset image size.

[0069] The global-local visual-text scene graph similarity measurement model construction unit 2 is configured to perform feature extraction on the images of the uniform size, perform feature fusion by using a fully connected network, obtain object multi-modal fusion features and object relationship multi-modal fusion features, construct a visual scene graph by taking the object multi-modal fusion features as nodes and the object relationship multi-modal fusion features as edges, learn and update the visual scene graph by using a graph attention network, obtain an updated visual scene graph, extract word features and semantic triple features from the texts, construct a text scene graph by taking the word features as nodes and the semantic triple features as edges, learn and update the text scene graph by using a graph attention network, obtain an updated text scene graph, enhance the updated visual scene graph by using the updated text scene graph, obtain a text-enhanced visual scene graph, calculate an overall similarity score by using the updated text scene graph and the text-enhanced visual scene graph, construct a global-local visual-text scene graph similarity measurement model, train the global-local visual-text scene graph similarity measurement model by using a loss function until the function converges, and obtain a converged global-local visual-text scene graph similarity measurement model;

[0070] Specifically, the Faster-R-CNN is used to detect the regions of the objects of the images of the uniform size, and the position coordinates of all the objects are obtained; for each detected object, the CLIP is used to encode the object region into a visual feature, and then the visual feature and the position coordinates are mapped into a high-dimensional space through two linear layers, and the two features are added to obtain an initialized object visual feature; for any two objects, a correlation region covering both objects is generated, and the relationship position coordinates of the region are obtained, the CLIP is used to encode the correlation region into a relationship visual feature, and then the relationship visual feature and the relationship position coordinates are mapped into a high-dimensional space through two linear layers, and the two features are added to obtain an initialized object relationship visual feature;

[0071] The Faster-R-CNN is used to extract the categories, attributes and relationships between the objects of the detected objects in the images of the uniform size, and the initial object semantic labels and the initial object relationship semantic labels are obtained, the fast-Text is used to perform feature extraction on the two types of labels respectively and map them into a high-dimensional space through a linear layer, and the object semantic features and the object relationship semantic features are obtained;

[0072] The object multi-modal fusion features and the object relationship multi-modal fusion features are obtained by connecting the object visual features and the object semantic features and the object relationship visual features and the object relationship semantic features through vector splicing using a full connection network;

[0073] The nodes and edges of the visual scene graph are updated respectively to obtain an updated visual scene graph, that is, for the nodes of the visual scene graph, the nodes of the visual scene graph are updated through a full connection layer according to the information of the previous layer itself; for the edges of the visual scene graph, the edges of the visual scene graph are updated through a full connection layer in combination with the information of the previous layer itself and the information of the neighbor nodes of the previous layer;

[0074] The semantic triplets of the text are extracted by using a SPICE method;

[0075] Each word of the text and the semantic triplets is mapped to a high-dimensional space and represented as a dense real-valued vector by using a GloVe word embedding model to obtain word embedding of the words and word embedding of the semantic triplets;

[0076] The word embedding of the words and the word embedding of the semantic triplets are respectively input into two BiLSTM models to obtain forward and backward features of the words and the semantic triplets and take an average value to obtain word features and semantic triplet features;

[0077] A preliminary text scene graph is constructed by taking the word features as nodes and the semantic triplet features as edges;

[0078] The features of the updated visual scene graph are mapped to semantic features by a learnable linear layer, and the preliminary text scene graph is enhanced by a vector connection method to obtain an enhanced text scene graph;

[0079] The features of the enhanced text scene graph are learned and updated by using a graph attention network to obtain an updated text scene graph;

[0080] The semantic features are mapped back to the features of the updated visual scene graph by a learnable linear layer, and the features of the updated visual scene graph are enhanced by the features of the updated text scene graph through a vector connection method to obtain a text-enhanced visual scene graph;

[0081] performing local similarity measurement on the updated text scene graph and the text-enhanced visual scene graph, i.e., matching each node feature and each edge feature of the updated text scene graph and the text-enhanced visual scene graph respectively; for a node, calculating a matching score between it and a relevant object in an image and taking an average value as a node score; for an edge, calculating a score between it and a relevant matching edge in the image and taking an average value as an edge score, and adding the node score and the edge score to obtain a local similarity score;

[0082] converting the updated text scene graph and the text-enhanced visual scene graph into text scene graph embeddings and visual scene graph embeddings by using a learnable linear layer, and performing global similarity measurement on the text scene graph embeddings and the visual scene graph embeddings, i.e., calculating a cosine distance between vectors of the text scene graph embeddings and the visual scene graph embeddings to obtain a global similarity score;

[0083] adding the local similarity score and the global similarity score to obtain a total similarity score, and constructing a global-local visual-text scene graph similarity measurement model;

[0084] training the global-local visual-text scene graph similarity measurement model by using an improved triplet loss function in which a negative sample is replaced by a most difficult negative sample, so that a global similarity score of a similar image-text pair is higher than that of a dissimilar image-text pair, the loss function including two parts: one part is for a real image-text pair, and the other part is for a most difficult negative triplet, and an expression of the loss function is:

[0085]

[0086] wherein m is a boundary parameter, k and l represent corresponding image-text pairs, and k and represent a mismatched pair, and l also represent a mismatched pair;

[0087] stopping training when the loss function converges to obtain a converged global-local visual-text scene graph similarity measurement model.

[0088] a query retrieval unit 3 configured to input a query object by a user into the converged global-local visual-text scene graph similarity measurement model to obtain a matching result;

[0089] Specifically, the user inputs the query object into the converged global-local visual-textual scene graph similarity measurement model, constructs the corresponding scene graph and matches with the relative scene graph in the model, and calculates the overall similarity score, that is, if the user input is an image, the visual scene graph corresponding to the input image and the text scene graph of the text in the test set are constructed, a plurality of visual-textual scene graph pairs are obtained, and the overall similarity score is calculated for each visual-textual scene graph pair; if the user input is text, the text scene graph of the input text and the visual scene graph of the image in the test set are constructed, a plurality of text-scene graph pairs are obtained, and the overall similarity score is calculated for each text-scene graph pair.

[0090] The overall similarity scores are sorted, the result with a higher overall similarity score is a more relevant answer retrieved, and the result is output as a matching result.

[0091] Therefore, the present application can promote a deeper understanding of image and text queries by constructing scene graphs of images and texts to capture the interaction between objects and between the objects in the images and texts, and can help to enhance image-text retrieval. The label semantic information and the text semantic information are fused in the construction of the visual scene graph, the visual information is fused in the construction of the text scene graph, the interaction between different modalities is realized, and the hidden multi-modal knowledge relationship is mined, which can improve the accuracy of retrieval. The similarity between the visual scene graph and the text scene graph is calculated from two angles of local and global, which can more comprehensively evaluate the matching degree between the image and the text, thereby improving the performance of the image-text retrieval task.

[0092] The above provides a detailed introduction to the image-text retrieval method and system based on scene graphs provided by the embodiments of the present application. The principles and implementation manners of the present application are described by applying specific examples in this paper. The above embodiment description is only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed, and the above description should not be understood as a limitation of the present application.

Claims

1. A method of image text retrieval based on a scene graph, characterized by, The method comprises: inputting images and texts from a retrieval data set, pre-processing the images to obtain images of uniform size, and dividing the retrieval data set into a training set and a test set; extracting features from the images of uniform size, and fusing the features by using a fully connected network to obtain object multi-modal fusion features and object relationship multi-modal fusion features, constructing a visual scene graph with the object multi-modal fusion features as nodes and the object relationship multi-modal fusion features as edges, learning and updating the visual scene graph by using a graph attention network to obtain an updated visual scene graph; extracting word features and semantic triple features from the texts, constructing a text scene graph with the word features as nodes and the semantic triple features as edges, learning and updating the text scene graph by using a graph attention network to obtain an updated text scene graph, and enhancing the updated visual scene graph by using the updated text scene graph to obtain a text-enhanced visual scene graph; calculating a total similarity score by using the updated text scene graph and the text-enhanced visual scene graph, constructing a global-local visual-text scene graph similarity measurement model, training the global-local visual-text scene graph similarity measurement model by using a loss function until the function converges to obtain a converged global-local visual-text scene graph similarity measurement model; a user inputs a query object into the converged global-local visual-text scene graph similarity measurement model to obtain a matching result, specifically: the user inputs the query object into the converged global-local visual-text scene graph similarity measurement model, constructs a corresponding scene graph, and matches the corresponding scene graph with a relative scene graph in the model to calculate a total similarity score, including: if the user input is an image, constructing a visual scene graph corresponding to the image and a text scene graph of a text in the test set to obtain a plurality of visual-text scene graph pairs, and calculating a total similarity score for each visual-text scene graph pair; or if the user input is a text, constructing a text scene graph of the text and a visual scene graph of an image in the test set to obtain a plurality of text-scene graph pairs, and calculating a total similarity score for each text-scene graph pair; sorting the total similarity scores, and outputting a matching result as a more relevant answer retrieved from a result with a higher total similarity score.

2. The image text retrieval method based on scene graph according to claim 1, wherein, The inputting images and texts from a retrieval data set, pre-processing the images to obtain images of uniform size, and dividing the retrieval data set into a training set and a test set specifically comprises: the retrieval data set is composed of images and texts, and is stored in the form of image-text pairs, each image has one or more corresponding texts for describing the content of the image; the images are pre-processed by scaling adjustment to a preset image size.

3. The scene graph based image text retrieval method of claim 1, wherein, The extracting features from the images of uniform size, and fusing the features by using a fully connected network to obtain object multi-modal fusion features and object relationship multi-modal fusion features, constructing a visual scene graph with the object multi-modal fusion features as nodes and the object relationship multi-modal fusion features as edges, learning and updating the visual scene graph by using a graph attention network to obtain an updated visual scene graph specifically comprises: detecting regions of objects in the image of the uniform size using Faster-R-CNN to obtain position coordinates of all objects; for each object detected, encoding the object region into visual features using CLIP, then mapping the visual features and the position coordinates into a high-dimensional space through two linear layers, and adding the two features to obtain an initialized object visual feature; for any two objects, generating a related region covering both, obtaining the relationship position coordinates of the region, encoding the related region into relationship visual features using CLIP, then mapping the relationship visual features and the relationship position coordinates into a high-dimensional space through two linear layers, and adding the two features to obtain an initialized object relationship visual feature; extracting the categories, attributes and relationships between objects in the detected objects in the image of the uniform size using Faster-R-CNN to obtain initial object semantic labels and initial object relationship semantic labels, and extracting features of the two types of labels using fast-Text respectively and mapping them into a high-dimensional space through linear layers to obtain object semantic features and object relationship semantic features; using a fully connected network to perform feature fusion through vector splicing to connect the object visual features and the object semantic features and the object relationship visual features and the object relationship semantic features to obtain object multi-modal fusion features and object relationship multi-modal fusion features; updating the nodes and edges of the visual scene graph respectively to obtain an updated visual scene graph, including: for the nodes of the visual scene graph, updating the nodes of the visual scene graph through a fully connected layer according to the information of the previous layer itself; for the edges of the visual scene graph, updating the edges of the visual scene graph through a fully connected layer in combination with the information of the previous layer itself and the information of the neighbor nodes of the previous layer.

4. The scene graph based image text retrieval method of claim 1, wherein, extracting word features and semantic triple features from the text, constructing a text scene graph with the word features as nodes and the semantic triple features as edges, learning and updating the text scene graph using a graph attention network, obtaining an updated text scene graph, and enhancing the updated visual scene graph using the updated text scene graph to obtain a text-enhanced visual scene graph, specifically: extracting semantic triples of the text using the SPICE method; mapping each word of the text and the semantic triples to a high-dimensional space and representing them as dense real-valued vectors using the GloVe word embedding model to obtain word embeddings of the words and word embeddings of the semantic triples; inputting the word embeddings of the words and the word embeddings of the semantic triples into two BiLSTM models respectively to obtain forward and backward features of the words and the semantic triples and taking the average to obtain word features and semantic triple features; constructing a preliminary text scene graph with the word features as nodes and the semantic triple features as edges; mapping the features of the updated visual scene graph to semantic features through a learnable linear layer, and enhancing the preliminary text scene graph using the semantic features through vector connection to obtain an enhanced text scene graph; learning, updating the features of the enhanced text scene graph to obtain an updated text scene graph; mapping the semantic features back to the features of the updated visual scene graph through a learnable linear layer, enhancing the features of the updated visual scene graph with the features of the updated text scene graph through vector connection to obtain a text-enhanced visual scene graph.

5. The scene graph based image text retrieval method of claim 1, wherein, The global-local visual-text scene graph similarity measurement model is constructed by calculating the overall similarity score based on the updated text scene graph and the text-enhanced visual scene graph, and the global-local visual-text scene graph similarity measurement model is trained by using a loss function until the function converges, and the converged global-local visual-text scene graph similarity measurement model is obtained, specifically as follows: The local similarity measurement is performed on the updated text scene graph and the text-enhanced visual scene graph, specifically by matching each node feature and each edge feature of the updated text scene graph and the text-enhanced visual scene graph respectively; for a node, the matching score between the node and the related object in the image is calculated, and the average value is taken as the node score; for an edge, the score between the edge and the related matching edge in the image is calculated, and the average value is taken as the edge score, and the local similarity score is obtained by adding the node score and the edge score; The updated text scene graph and the text-enhanced visual scene graph are converted into text scene graph embedding and visual scene graph embedding by using a learnable linear layer, and the global similarity measurement is performed by using the text scene graph embedding and the visual scene graph embedding, specifically by calculating the cosine distance of the vectors of the text scene graph embedding and the visual scene graph embedding to obtain the global similarity score; The local similarity score and the global similarity score are added to obtain the overall similarity score, and the global-local visual-text scene graph similarity measurement model is constructed; The global-local visual-text scene graph similarity measurement model is trained by using an improved triplet loss function in which the negative sample is replaced by the most difficult negative sample, so that the global similarity score of similar image-text pairs is higher than that of dissimilar image-text pairs, and the loss function includes two parts: one part is used for real image-text pairs, and the other part is used for the most difficult negative triplet, and the expression is as follows: where m is a boundary parameter, k and l represent the corresponding image-text pair, and k and represent a mismatch pair, and l also represent a mismatch pair; The training is stopped when the loss function converges, and the converged global-local visual-text scene graph similarity measurement model is obtained.

6. A scene graph-based image text retrieval system, characterized by, The system comprises: a preprocessing unit configured to input images and texts from a retrieval data set, pre-process the images to obtain images of a uniform size, and divide the retrieval data set into a training set and a test set; The global-local visual-textual scene graph similarity measurement model construction unit is configured to perform feature extraction on the unified size image, perform feature fusion by using a full connection network, obtain object multi-modal fusion features and object relationship multi-modal fusion features, construct a visual scene graph by taking the object multi-modal fusion features as nodes and the object relationship multi-modal fusion features as edges, learn and update the visual scene graph by using a graph attention network, and obtain an updated visual scene graph; extract word features and semantic triple features from the text, construct a textual scene graph by taking the word features as nodes and the semantic triple features as edges, learn and update the textual scene graph by using a graph attention network, and obtain an updated textual scene graph; enhance the updated visual scene graph by using the updated textual scene graph; calculate an overall similarity score by using the updated textual scene graph and the textual enhanced visual scene graph, construct a global-local visual-textual scene graph similarity measurement model, train the global-local visual-textual scene graph similarity measurement model by using a loss function until the function converges, and obtain a converged global-local visual-textual scene graph similarity measurement model; The query retrieval unit is configured to input a query object into the converged global-local visual-textual scene graph similarity measurement model by a user, and obtain a matching result. Specifically, the query retrieval unit specifically includes: The user inputs a query object into the converged global-local visual-textual scene graph similarity measurement model, constructs a corresponding scene graph, and matches the corresponding scene graph with a relative scene graph in the model, and calculates an overall similarity score, including: if the user input is an image, a visual scene graph corresponding to the image and a textual scene graph of text in the test set are constructed, a plurality of visual-textual scene graph pairs are obtained, and an overall similarity score is calculated for each visual-textual scene graph pair; if the user input is text, a textual scene graph of the text and a visual scene graph of an image in the test set are constructed, a plurality of textual-scene graph pairs are obtained, and an overall similarity score is calculated for each textual-scene graph pair; The overall similarity scores are sorted, and a result with a higher overall similarity score is a more relevant answer retrieved and output as a matching result.

7. A scene graph based image text retrieval system as claimed in claim 6, wherein, The preprocessing unit is configured to input images and text from a retrieval data set, pre-process the images to obtain unified size images, and divide the retrieval data set into a training set and a test set, specifically including: The retrieval data set is composed of images and text, and is stored in the form of image-text pairs. Each image has one or more texts corresponding to the image, which are used to describe the content of the image. The images are pre-processed by scaling adjustment to a preset image size.

8. A scene graph based image text retrieval system as claimed in claim 6, wherein, The global-local visual-textual scene graph similarity measurement model construction unit is configured to perform feature extraction on the image of the uniform size, perform feature fusion using a fully connected network, obtain object multi-modal fusion features and object relationship multi-modal fusion features, construct a visual scene graph using the object multi-modal fusion features as nodes and the object relationship multi-modal fusion features as edges, learn and update the visual scene graph using a graph attention network, and obtain an updated visual scene graph; perform word feature and semantic triple feature extraction on the text, construct a textual scene graph using the word features as nodes and the semantic triple features as edges, learn and update the textual scene graph using a graph attention network, and obtain an updated textual scene graph; enhance the updated visual scene graph using the updated textual scene graph, and obtain a textual-enhanced visual scene graph; calculate an overall similarity score using the updated textual scene graph and the textual-enhanced visual scene graph, construct a global-local visual-textual scene graph similarity measurement model, train the global-local visual-textual scene graph similarity measurement model using a loss function until the function converges, and obtain a converged global-local visual-textual scene graph similarity measurement model, specifically as follows: detect the regions of objects in the image of the uniform size using Faster-R-CNN, and obtain the position coordinates of all objects; for each detected object, encode the object region into a visual feature using CLIP, then map the visual feature and the position coordinates into a high-dimensional space through two linear layers, and add the two features to obtain an initialized object visual feature; for any two objects, generate a correlation region covering both objects, obtain the relationship position coordinates of the region, encode the correlation region into a relationship visual feature using CLIP, then map the relationship visual feature and the relationship position coordinates into a high-dimensional space through two linear layers, and add the two features to obtain an initialized object relationship visual feature; extract the categories, attributes, and relationships between objects of the detected objects in the image of the uniform size using Faster-R-CNN, obtain initial object semantic labels and initial object relationship semantic labels, and extract features of the two types of labels using fast-Text and map them into a high-dimensional space through linear layers to obtain object semantic features and object relationship semantic features; perform feature fusion by vector splicing using a fully connected network, connect the object visual features and the object semantic features, and connect the object relationship visual features and the object relationship semantic features, to obtain object multi-modal fusion features and object relationship multi-modal fusion features; update the nodes and edges of the visual scene graph respectively to obtain an updated visual scene graph, including: for the nodes of the visual scene graph, update the nodes of the visual scene graph through a fully connected layer according to the information of the previous layer; for the edges of the visual scene graph, update the edges of the visual scene graph through a fully connected layer in combination with the information of the previous layer and the information of the neighbor nodes of the previous layer; extract semantic triples of the text using the SPICE method; mapping each word of the text and the semantic triplets to a high-dimensional space and representing as a dense real-valued vector by a GloVe word embedding model, obtaining word embeddings of the words and word embeddings of the semantic triplets; inputting the word embeddings of the words and the word embeddings of the semantic triplets into two BiLSTM models respectively, obtaining forward and backward features of the words and the semantic triplets and taking average values, obtaining word features and semantic triplet features; constructing a preliminary text scene graph with the word features as nodes and the semantic triplet features as edges; mapping the features of the updated visual scene graph to semantic features through a learnable linear layer, and enhancing the preliminary text scene graph through vector connection, obtaining an enhanced text scene graph; learning and updating the features of the enhanced text scene graph by a graph attention network, obtaining an updated text scene graph; mapping the semantic features back to the features of the updated visual scene graph through a learnable linear layer, and enhancing the features of the updated visual scene graph through vector connection with the features of the updated text scene graph, obtaining a text-enhanced visual scene graph; performing local similarity measurement on the updated text scene graph and the text-enhanced visual scene graph, specifically, matching each node feature and each edge feature of the updated text scene graph and the text-enhanced visual scene graph respectively; for nodes, calculating the matching score between nodes and related objects in the image and taking the average value as the node score; for edges, calculating the score between edges and related matching edges in the image and taking the average value as the edge score, and adding the node score and the edge score to obtain the local similarity score; converting the updated text scene graph and the text-enhanced visual scene graph into text scene graph embedding and visual scene graph embedding through a learnable linear layer, and performing global similarity measurement by using the text scene graph embedding and the visual scene graph embedding, specifically, calculating the cosine distance of the vectors of the text scene graph embedding and the visual scene graph embedding, obtaining the global similarity score; adding the local similarity score and the global similarity score to obtain the overall similarity score, and constructing the global-local visual-text scene graph similarity measurement model; training the global-local visual-text scene graph similarity measurement model using an improved triplet loss function in which the negative sample is replaced by the most difficult negative sample, so that the global similarity score of similar image-text pairs is higher than that of dissimilar image-text pairs, and the loss function includes two parts: one part is for real image-text pairs, and the other part is for the most difficult negative triplets, and its expression is: where m is a boundary parameter, k and l represent the corresponding image-text pair, and k and represent a mismatch pair, and l also represent a mismatch pair; stopping training when the loss function converges, obtaining the converged global-local visual-text scene graph similarity measurement model.

Images