Method and device for retrieving target

Abstract

The embodiments of the present disclosure disclose a method and device for retrieving a target. A specific implementation of the method includes: obtaining at least one image and a descriptive text of a specified object; using a pre-trained cross-media feature extraction network to extract image features of the image and text features of the descriptive text; matching the image features with the text features to determine the image containing the specified object, thereby using the cross-media feature extraction feature to project the image features and the text features into the common feature space of the image and text for feature matching, thereby achieving cross-media target retrieval.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and more particularly to a method and device for retrieving a target. Background Art

[0002] With the rapid development of Internet technology, the presentation methods of media data are becoming more and more diverse, and different types of media data describe the same thing from different perspectives.

[0003] People expect to achieve cross-media retrieval between different types of media data, that is, through one type of media data, query and retrieve media data of another media type with the same semantics. Summary of the invention

[0004] The embodiments of the present disclosure provide a method and apparatus for retrieving a target.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for retrieving a target, the method comprising: obtaining at least one image and a descriptive text of a specified object; extracting image features of the image and text features of the descriptive text using a pre-trained cross-media feature extraction network, wherein the cross-media feature extraction network projects the text features and the image features into a common feature space of the image and text; matching the image features with the text features to determine an image containing the specified object.

[0006] In some embodiments, a cross-media feature extraction network is generated according to the following method: obtaining a training sample set, wherein the training sample set includes a sample image-text pair, and the sample image-text pair includes: a sample image and a sample text describing an object contained in the sample image; obtaining an initial network, wherein the initial network includes a cross-media feature extraction network to be trained, a discriminant network for discriminating the data category of the feature source, and a feature conversion network, and the cross-media feature extraction network to be trained includes an image-graph attention network and a text-graph attention network; inputting a sample image in the sample image-text pair into the image-graph attention network to obtain a first feature; inputting a sample text in the sample image-text pair into the text-graph attention network to obtain a second feature; inputting the first feature and the second feature into the discriminant network to obtain a category discrimination result, and calculating a discrimination loss value according to the category discrimination result, wherein the feature The data categories of the source include text and image, and the discrimination loss value represents the category judgment error of the first feature and the second feature; the first feature and the second feature are input into the feature transformation network to obtain the feature transformation result, and the value of the recognition loss function and the value of the pairwise loss function are calculated according to the feature transformation result, wherein the recognition loss function represents the ability of the first feature and the second feature to distinguish different objects contained in the common feature space of the image and text, and the pairwise loss function represents the semantic difference between the first feature and the second feature of the same object; based on the value of the recognition loss function and the value of the pairwise loss function, a preset feature loss value is obtained; based on the discrimination loss value and the feature loss value, the cross-media feature extraction network and the feature transformation network to be trained are used as the generation network, and adversarial training is performed with the discrimination network to obtain the trained cross-media feature extraction network, the discrimination network, and the feature transformation network.

[0007] In some embodiments, the image-graph attention network includes a residual network, an image-graph attention convolutional network, and a joint embedding layer; and inputting the sample image into the image-graph attention network to obtain a first feature includes: extracting initial image features of the sample image using the residual network, inputting the initial image features into the image-graph attention convolutional network, outputting structured image features of the sample image, fusing the initial image features with the structured image features to generate image features, and inputting the image features into the joint embedding layer to obtain the first feature.

[0008] In some embodiments, the initial image features are input into an image graph attention convolutional network, and structured image features of the sample image are output, including: performing target object detection on the sample image, determining the position of the target object and the rectangular bounding box of the target object in the sample image, and extracting relevant features of the target object based on the position of the rectangular bounding box of the target object, wherein the relevant features of the target object include at least one of the following: appearance features of the target object, position features of the target object, and type features of the target object; constructing an image feature directed graph, wherein the vertices of the image feature directed graph represent the target objects, and the directed edges of the image feature directed graph represent the association relationship between the target objects.

[0009] In some embodiments, a text graph attention network includes a bidirectional long short-term memory network, a text graph attention convolutional network, and a joint embedding layer; and inputting a sample text into the text graph attention network to obtain a second feature includes: performing word segmentation on the sample text to determine the word vector of the sample text; using the bidirectional long short-term memory network to extract the initial text features of the word vector of the sample text, inputting the initial text features into the text graph attention convolutional network, outputting the structured text features of the sample text, fusing the initial text features with the structured text features to generate text features, and inputting the text features into the joint embedding layer to obtain the second feature.

[0010] In some embodiments, the initial text features are input into a text graph attention convolutional network, and the structured text features of the sample text are output, including: constructing a text feature directed graph, the vertices of the text feature directed graph represent the target objects indicated by the word vectors, and the directed edges of the text feature directed graph represent the association relationship between the target objects indicated by the word vectors, wherein the relevant features of the target objects indicated by the word vectors include at least one of the following: attribute features of the target object and type features of the target object; according to the text feature directed graph, the structured text features of the sample text are generated.

[0011] In a second aspect, an embodiment of the present disclosure provides a device for retrieving a target, the device comprising: an acquisition unit, configured to acquire at least one image and a descriptive text of a specified object; an extraction unit, configured to extract image features of the image and text features of the descriptive text using a pre-trained cross-media feature extraction network, wherein the cross-media feature extraction network projects the text features and the image features into a common feature space of the image and text; a matching unit, configured to match the image features with the text features to determine an image containing the specified object.

[0012] In some embodiments, a cross-media feature extraction network is generated according to the following method: obtaining a training sample set, wherein the training sample set includes a sample image-text pair, and the sample image-text pair includes: a sample image and a sample text describing an object contained in the sample image; obtaining an initial network, wherein the initial network includes a cross-media feature extraction network to be trained, a discriminant network for discriminating the data category of the feature source, and a feature conversion network, and the cross-media feature extraction network to be trained includes an image-graph attention network and a text-graph attention network; inputting a sample image in the sample image-text pair into the image-graph attention network to obtain a first feature; inputting a sample text in the sample image-text pair into the text-graph attention network to obtain a second feature; inputting the first feature and the second feature into the discriminant network to obtain a category discrimination result, and calculating a discrimination loss value according to the category discrimination result, wherein the feature The data categories of the source include text and image, and the discrimination loss value represents the category judgment error of the first feature and the second feature; the first feature and the second feature are input into the feature transformation network to obtain the feature transformation result, and the value of the recognition loss function and the value of the pairwise loss function are calculated according to the feature transformation result, wherein the recognition loss function represents the ability of the first feature and the second feature to distinguish different objects contained in the common feature space of the image and text, and the pairwise loss function represents the semantic difference between the first feature and the second feature of the same object; based on the value of the recognition loss function and the value of the pairwise loss function, a preset feature loss value is obtained; based on the discrimination loss value and the feature loss value, the cross-media feature extraction network and the feature transformation network to be trained are used as the generation network, and adversarial training is performed with the discrimination network to obtain the trained cross-media feature extraction network, the discrimination network, and the feature transformation network.

[0013] In some embodiments, the image-graph attention network includes a residual network, an image-graph attention convolutional network, and a joint embedding layer; and the first feature is obtained according to the following method: using the residual network to extract the initial image features of the sample image, inputting the initial image features into the image-graph attention convolutional network, outputting the structured image features of the sample image, fusing the initial image features with the structured image features to generate image features, and inputting the image features into the joint embedding layer to obtain the first feature.

[0014] In some embodiments, the structured image features of the sample image are obtained according to the following method: performing target object detection on the sample image, determining the position of the target object and the rectangular bounding box of the target object in the sample image, and extracting relevant features of the target object based on the position of the rectangular bounding box of the target object, wherein the relevant features of the target object include at least one of the following: appearance features of the target object, position features of the target object, and type features of the target object; constructing an image feature directed graph, wherein the vertices of the image feature directed graph represent the target objects, and the directed edges of the image feature directed graph represent the association relationship features between the target objects.

[0015] In some embodiments, the text graph attention network includes a bidirectional long short-term memory network, a text graph attention convolutional network and a joint embedding layer; the second feature is obtained according to the following method: the sample text is segmented to determine the word vector of the sample text; the initial text features of the word vector of the sample text are extracted using the bidirectional long short-term memory network, the initial text features are input into the text graph attention convolutional network, the structured text features of the sample text are output, the initial text features are fused with the structured text features to generate text features, and the text features are input into the joint embedding layer to obtain the second feature.

[0016] In some embodiments, the structured text features of the sample text are obtained according to the following method: constructing a text feature directed graph, the vertices of the text feature directed graph represent the target objects indicated by the word vectors, and the directed edges of the text feature directed graph represent the association relationship features between the target objects indicated by the word vectors, wherein the relevant features of the target objects indicated by the word vectors include at least one of the following: attribute features of the target object and type features of the target object; generating the structured text features of the sample text according to the text feature directed graph.

[0017] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: one or more processors; a storage device on which one or more programs are stored; when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0018] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0019] The method and device for retrieving a target provided by the embodiments of the present disclosure obtain at least one image and a descriptive text of a specified object, and then use a pre-trained cross-media feature extraction network to extract image features of the image and text features of the descriptive text, wherein the cross-media feature extraction network projects text features and image features into a common feature space of image and text, and finally, matches the image features with the text features to determine the image containing the specified object, thereby using cross-media feature extraction features to project image features and text features into the common feature space of image and text for feature matching, thereby achieving cross-media target retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0021] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;

[0022] Figure 2 is a flowchart of an embodiment of a method for retrieving a target according to the present disclosure;

[0023] Figure 3 is a flowchart of an implementation method of the above-mentioned cross-media feature extraction network generation method;

[0024] Figure 4 It is a schematic diagram of an architecture of an implementation process of the method for retrieving a target disclosed in the present invention;

[0025] Figure 5 is a schematic structural diagram of an embodiment of a device for retrieving a target according to the present disclosure;

[0026] Figure 6 It is a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure. DETAILED DESCRIPTION

[0027] The present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0028] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0029] Figure 1 An exemplary architecture 100 is shown to which the method of retrieving a target or the apparatus of retrieving a target of the present disclosure can be applied.

[0030] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0031] The terminal devices 101, 102, 103 interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications, such as image editing applications, text editing applications, browser applications, etc., may be installed on the terminal devices 101, 102, 103.

[0032] Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens and supporting Internet access, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc. When terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0033] The server 105 may be a server that provides various services, such as a background server that matches images and texts acquired by the terminal devices 101, 102, and 103. The background server may perform recognition, matching, and other processing on the received images and texts.

[0034] It should be noted that the method for retrieving a target provided in the embodiments of the present disclosure is generally executed by the server 105 , and accordingly, the device for retrieving a target is generally disposed in the server 105 .

[0035] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0036] Continue to refer Figure 2 , shows a process 200 of an embodiment of a method for retrieving a target according to the present disclosure. The method for retrieving a target comprises the following steps:

[0037] Step 201, obtaining at least one image and a description text of a specified object.

[0038] In this embodiment, the execution subject of the above-mentioned method of retrieving the target (for example Figure 1 The server shown in the figure can obtain at least one image and a description text of a specified object from a pre-stored image library. Here, the above image can be any image to be retrieved in the pre-stored image library.

[0039] Here, the designated object refers to the entity to be retrieved, and the designated object may be an object with a variable position, such as a pedestrian, a vehicle, etc., or an object with a fixed position, such as a building, a landscape, etc. The number of the above-mentioned designated objects may be one or more. The image may contain the designated object or other objects. The text may be a sentence or word that describes the characteristics of the designated object. For example, if the designated object is pedestrian A, then in step 201, an image to be retrieved containing the pedestrian and a descriptive text of the appearance, movements and other characteristics of pedestrian A may be obtained from the image library. Then, in subsequent steps, the image to be retrieved is matched with the descriptive text of pedestrian A, and an image containing pedestrian A that matches the descriptive text of pedestrian A is determined from the image to be retrieved containing the pedestrian.

[0040] Step 202: extract image features of the image and text features of the description text using a pre-trained cross-media feature extraction network.

[0041] In this embodiment, the execution subject can use a pre-trained cross-media feature extraction network to extract the image features of at least one image and the text features of the description text of the specified object in step 201. The cross-media feature extraction network can project the text features and the image features into a common feature space of the image and text. Specifically, the cross-media feature extraction network can extract features of data of different media types, and can convert the features of data of different media types into the same common feature space, so that the features of data of different media types can be matched in the same common feature space.

[0042] Here, the cross-media feature extraction network can be an artificial neural network. Based on at least one image obtained in step 201 and the descriptive text of the specified object, the above-mentioned execution entity can input one or more images and the descriptive text into a pre-trained artificial neural network to extract image features corresponding to the image and text features corresponding to the text. A partial or entire image can be selected from the above-mentioned one or more images to extract image features of the selected partial or entire image. A partial or entire descriptive text can be selected from the above-mentioned descriptive text of the specified object to extract text features of the selected partial or entire text. The above-mentioned features can be represented by feature vectors.

[0043] Step 203: Match the image features with the text features to determine the image containing the specified object.

[0044] In this embodiment, the execution entity may match the image features extracted in step 202 with the text features, and determine an image containing the specified object from at least one image as an image matching the text description.

[0045] Here, the execution subject can determine whether the image feature matches the text feature by calculating the similarity between the image feature of each image and the text feature describing the text. The similarity can be related to the distance between the image feature and the text feature. For example, the similarity between the image feature and the text feature can be calculated using Euclidean distance, Minkowski distance, Manhattan distance, Chebyshev distance, Mahalanobis distance, etc.

[0046] In practice, in order to more comprehensively match image features similar to text features, the similarity between each image feature and the text feature can be calculated first, and the similarity between each image feature and the text feature can be sorted from high to low, and the image feature sorted in the front preset position is selected, and the image indicated by the image feature sorted in the front preset position is used as the image matching the text description. The preset position can be set according to actual needs, and can be one or more. Here, the higher the similarity between the text feature and the image feature, the greater the possibility that the object indicated by the text feature and the image feature is the same object.

[0047] The method for retrieving a target provided by the above-mentioned embodiment of the present disclosure obtains at least one image and a descriptive text of a specified object, and then uses a pre-trained cross-media feature extraction network to extract image features of the image and text features of the descriptive text, wherein the cross-media feature extraction network projects text features and image features into a common feature space of image and text, and finally, matches the image features with the text features to determine the image containing the specified object, thereby utilizing cross-media feature extraction features to project image features and text features into a common feature space of image and text for feature matching, thereby achieving cross-media target retrieval.

[0048] Continue to refer Figure 3 , Figure 3 300 is a flowchart of an implementation of the above-mentioned method for generating a cross-media feature extraction network. The process 300 of the method for generating a cross-media feature extraction network may include the following steps:

[0049] Step 301: Obtain a training sample set.

[0050] In this embodiment, the execution subject may obtain a training sample set from a preset database. The training sample set may include a sample image-text pair. The sample image-text pair may include a sample image and a sample text describing an object included in the sample image.

[0051] Step 302: Acquire an initial network.

[0052] In this embodiment, the execution subject may first obtain an initial network. Specifically, the initial network may include a cross-media feature extraction network to be trained, a discriminant network for discriminating the data category of the feature source, and a feature conversion network. The cross-media feature extraction network to be trained may include an image graph attention network and a text graph attention network. The initial network may be an untrained neural network after initialization of parameters, or a pre-trained neural network.

[0053] Step 303, select samples from the training sample set and perform training steps.

[0054] In this embodiment, the execution subject may select samples from the sample set obtained in step 301, and perform the training steps from step 3031 to step 3036. The method and number of sample selection are not limited in the present disclosure. For example, the execution subject may select at least one sample.

[0055] More specifically, the training step includes the following steps:

[0056] Step 3031, input the sample image in the sample image-text pair into the image-graph attention network to obtain the first feature.

[0057] In this embodiment, the execution subject may input the sample image in the sample image-text pair into the image graph attention network of the cross-media feature extraction network to be trained, and output the first feature, namely, the image feature of the sample image.

[0058] In some optional implementations of this embodiment, the image-graph attention network includes a residual network, an image-graph attention convolutional network, and a joint embedding layer. Specifically, the first feature can be obtained by the following steps:

[0059] In the first step, the residual network is used to extract the initial image features of the sample image.

[0060] In this optional implementation, the execution subject may use a residual network to extract initial image features, i.e., low-level visual features, of the sample image. Here, the initial image features may be global image features, such as color features, texture features, shape features, structural features, and the like.

[0061] Here, the above-mentioned execution subject can also use the residual network to extract the initial image features of the sample image, and then use the average pooling layer to generate a global appearance visual feature vector from the initial image features. The average pooling layer is used to reduce the dimension of the initial image features, and the generated global appearance visual feature vector can retain the significant features in the initial image features.

[0062] In the second step, the initial image features are input into the image graph attention convolutional network to output the structured image features of the sample image.

[0063] In this optional implementation, the above-mentioned execution subject can input the extracted low-level visual features into a pre-trained image graph attention convolutional network to obtain structured image features of the sample image. Here, the structured image features can be used to represent the structured semantic visual features of the sample image.

[0064] Optionally, the initial image features can also be input into the image graph attention convolutional network through the following steps to output the structured image features of the sample image: perform target object detection on the sample image, determine the position of the target object and the rectangular bounding box of the target object in the sample image, and extract relevant features of the target object according to the position of the rectangular bounding box of the target object; construct an image feature directed graph, the vertices of the image feature directed graph represent the target objects, and the directed edges of the image feature directed graph represent the association relationship between the target objects; generate the structured image features of the sample image according to the image feature directed graph.

[0065] In this optional implementation, the execution subject may obtain the structured image features of the sample image by constructing a directed graph of image features.

[0066] Specifically, the above-mentioned execution subject can first use a preset target detection algorithm to identify the target object in the sample image and the position of the rectangular bounding box of the target object, and then use the target detection algorithm to extract the appearance features of the target object from the rectangular bounding box of the target object. According to the position of the rectangular bounding box of the target object, the relevant features of the target object are extracted. For example, the relevant features of the target object may include at least one of the following: the appearance features of the target object, the position features of the target object, and the type features of the target object. Specifically, for example, the target detection algorithm R-CNN (Region-Convolutional Neural Networks) can be used to detect the target object in the sample image. Here, the target object can be any object pre-specified according to actual application requirements, such as shoes, glasses and other objects. The position features of the rectangular bounding box of the target object can be represented by the position coordinates of the rectangular bounding box of the target object in the sample image, such as the position coordinates of a multi-tuple representing the relationship between the horizontal coordinate of a vertex of the rectangular bounding box, the vertical coordinate of a vertex of the rectangular bounding box, the width of the rectangular bounding box, and the height of the rectangular bounding box. The category of the target object can be identified, for example, based on its shape, color, and other features, and the execution subject can use a preset entity relationship classifier to determine the association relationship between the target objects and a preset attribute classifier to determine the attributes of the target object. The appearance features of the target object can include global appearance features and local appearance features.

[0067] Then we can construct a directed graph of image features. The vertices of the directed graph represent the target objects, which can be represented by O i (O 1 , O 2 , O 3 , O 4 , O 5 , O 6 ) indicates that O i It can represent the subject or object in the triple "subject-predicate-object". The directed edges of the image feature directed graph represent the relationship between the target objects, which can be expressed by e ij (e 15 、e 16 、e 21 、e 31 、e 41 ) means that, for example, the directed edge e ij Represents object O i With object O j The relationship between ij Represents the predicate in the triple "subject-predicate-object", O i It can be the subject in the triple "subject-predicate-object", j It can be the object in the triple "subject-predicate-object". The vertex features of the image feature directed graph can be represented by the relevant features characterizing the target object obtained above, and the relevant features of the target object can include at least one of the following: appearance features of the target object, location features of the target object, and type features of the target object. The directed edge features of the image feature directed graph can be composed of at least one of the following: appearance features of the target object, location features of the target object, and type features of the target object.

[0068] Finally, the graph attention convolution layer can be used to update the image feature directed graph constructed above and extract the structured image features of the sample image. Specifically, the graph attention convolution layer can be used to update the image feature directed graph to obtain the updated vertices. The following update formula can be set to update the vertex features in the image feature directed graph:

[0069]

[0070] Among them, g s , g o represents the fully connected layer, w ij represents the weight of node j to node i, w ik represents the weight of node k to node i, o j Indicates the node representing the subject, o j represents the node representing the predicate, Represents node characteristics, Represents the directed edge features between nodes.

[0071] w ij It can be calculated by the following formula:

[0072]

[0073] Among them, w a Represents directed edge features The weight of b a Represents the bias term.

[0074] The updated vertex features of the image feature directed graph can be obtained through the above update formula. In order to fuse the updated vertex features, a virtual vertex connecting the vertices of the image feature directed graph is set. The above virtual vertex can generate the structured image features of the sample image through the following formula:

[0075]

[0076] in, represents the virtual vertex feature, w i represents the weight of node i, g v represents a fully connected layer.

[0077] w j It can be calculated by the following formula:

[0078]

[0079] Among them, w c Represents virtual vertex features The weight of b c Represents the bias term.

[0080] Since the virtual vertices are connected to each vertex of the image feature directed graph, and the image feature directed graph contains the relevant features of the target object and the association relationship features between the target objects, the virtual vertices can fuse all the features in the updated image feature directed graph to generate structured image features that represent the structured semantic information of the sample image, which can contain more effective structured semantic information, represent image features more comprehensively and accurately, and more effectively distinguish and identify the target objects contained in the image.

[0081] The third step is to fuse the initial image features with the structured image features to generate image features.

[0082] In this optional implementation, the execution subject may fuse the initial image features obtained in the first step with the structured image features obtained in the second step to obtain image features of the sample image.

[0083] In the fourth step, the image features are input into the joint embedding layer to obtain the first features.

[0084] In this optional implementation, the above-mentioned execution entity can input the image features obtained in the third step into the joint embedding layer to obtain the first features, and the joint embedding layer can be composed of three fully connected layers.

[0085] Step 3032: input the sample text in the sample image-text pair into the text graph attention network to obtain the second feature.

[0086] In this embodiment, the execution entity may input the sample text in the sample image-text pair into the text graph attention network of the cross-media feature extraction network to be trained, and output the second feature, namely, the text feature of the sample text.

[0087] In some optional implementations of this embodiment, the text graph attention network includes a bidirectional long short-term memory network, a text graph attention convolutional network, and a joint embedding layer. Specifically, the second feature can be obtained by the following steps:

[0088] Step 1: Segment the sample text and determine the word vector of the sample text.

[0089] In this optional implementation, the execution subject may perform word segmentation on the sample text using a commonly used word segmentation tool or manual annotation, and each word in the sample text is projected into a word vector.

[0090] In the second step, the bidirectional long short-term memory network is used to extract the initial text features of the word vector of the sample text.

[0091] In this optional implementation, the execution entity may utilize a bidirectional long short-term memory network to extract initial text features having context information from the sample text.

[0092] In the third step, the initial text features are input into the text graph attention convolutional network to output the structured text features of the sample text.

[0093] In this optional implementation, the execution subject may input the extracted initial text features into a pre-trained text graph attention convolutional network to obtain structured text features of the sample text. Here, the structured text features may be used to characterize the structured semantic text features of the sample text.

[0094] Optionally, the initial text features can be input into the text graph attention convolutional network through the following steps to output the structured text features of the sample text: construct a text feature directed graph, the vertices of the text feature directed graph represent the target objects indicated by the word vectors, and the directed edges of the text feature directed graph represent the association relationship between the target objects indicated by each word vector; according to the text feature directed graph, generate the structured text features of the sample text.

[0095] In this optional implementation, the execution subject may obtain the structured text features of the sample text through the constructed text feature directed graph.

[0096] Specifically, the execution subject can first construct a text feature directed graph, the vertices of which represent the target object, which can be represented by O i (O 1 , O 2 , O 3 , O 4 , O 5 , O 6 ) indicates that O i It can represent the subject or object in the triple "subject-predicate-object". The directed edges of the text feature directed graph represent the relationship between the target objects, which can be expressed by e ij (e 15 、e 16 、e 21 、e 31 、e 41 ) means that, for example, the directed edge e ij Represents object O i With object O j The relationship between ij Represents the predicate in the triple "subject-predicate-object", O i It can be the subject in the triple "subject-predicate-object", j It can be the object in the triple "subject-predicate-object". The vertex features of the text feature directed graph can be composed of relevant features of the target object, and the relevant features of the target object include at least one of the following: the attribute features of the target object and the type features of the target object. The directed edge features of the text feature directed graph can be composed of the type features of the target object.

[0097] Then, the graph attention convolution layer can be used to update the text feature directed graph constructed above, and the vertex features in the text feature directed graph can be updated using the update formula (1). The execution subject can set a virtual vertex connecting the vertices of the text feature directed graph. The virtual vertex can generate the structured text features of the sample text through formula (3), which can contain more effective structured semantic information, more comprehensively and accurately represent the text features, and more effectively distinguish and identify the target objects contained in the text.

[0098] The fourth step is to fuse the initial text features with the structured text features to generate text features.

[0099] In this optional implementation, the execution entity may fuse the initial text features obtained in the second step with the structured text features obtained in the third step to obtain text features of the sample text.

[0100] In the fifth step, the text features are input into the joint embedding layer to obtain the second features.

[0101] In this optional implementation, the above-mentioned execution entity can input the text features obtained in the fourth step into the joint embedding layer to obtain the second feature, and the joint embedding layer can be composed of three fully connected layers.

[0102] Step 3033: Input the first feature and the second feature into the discriminant network to obtain a category discrimination result, and calculate the discrimination loss value according to the category discrimination result.

[0103] In this embodiment, the execution subject may input the first feature and the second feature into the discriminant network to obtain a category discrimination result, and calculate the discrimination loss value according to the category discrimination result, wherein the data category of the feature source includes text category and image category, and the discrimination loss value represents the category discrimination error of the first feature and the second feature. The discriminant network may be composed of three fully connected layers, aiming to better judge the data type of the source of the given feature, that is, the modal category of the feature, such as text category, image category, and the discrimination loss value L may be calculated by the following loss function: adv (θ D ):

[0104]

[0105] Among them, v i represents the first feature, t i represents the second characteristic, D(v i θ D )、D(t i θ D ) represents the data category probability of the feature source of input sample i, θ D Represents the network parameters of the discriminant network.

[0106] Step 3034: Input the first feature and the second feature into the feature transformation network to obtain a feature transformation result, and calculate the value of the recognition loss function and the value of the pairwise loss function according to the feature transformation result.

[0107] In this embodiment, the execution subject may input the first feature and the second feature into the feature conversion network to obtain the feature conversion result, and calculate the value of the recognition loss function and the value of the pair loss function according to the feature conversion result, wherein the recognition loss function represents the ability of the first feature and the second feature to distinguish different objects contained in the common feature space of the image and text, and the pair loss function represents the semantic difference between the first feature and the second feature of the same object. The execution subject may calculate the recognition loss value L through the following loss function ide (θ V ,θ T ):

[0108]

[0109] Among them, y i Indicates the number of the target object corresponding to the i-th sample (sample text or sample image), x i represents the first feature or the second feature, θ V represents the network parameters of the image graph attention convolutional network, θ T represents the network parameters of the text graph attention convolutional network, W j represents the j-th column of the weight matrix W, b represents the bias term, and N represents the number of samples.

[0110] The above execution entity can calculate the pairwise loss value L through the following loss function pair (θ V ,θ T ):

[0111]

[0112] Among them, y i A two-dimensional vector indicating whether the sample image and sample text input pair indicate the same target object, z i The fusion features of text features and image features, W p,j Represents the weight matrix W p The jth column of b p represents the bias term, and M represents the number of sample image and sample text input pairs.

[0113] Step 3035, based on the value of the recognition loss function and the value of the pairwise loss function, a preset feature loss value is obtained.

[0114] In this embodiment, the execution entity may add the recognition loss value and the pairwise loss value obtained in step 3034 to obtain a feature loss value of the feature conversion network.

[0115] Step 3036, based on the discriminant loss value and the feature loss value, the cross-media feature extraction network and feature conversion network to be trained are used as the generating network, and adversarial training is performed with the discriminant network to obtain the trained cross-media feature extraction network, discriminant network, and feature conversion network.

[0116] In this embodiment, the above-mentioned execution subject can use the cross-media feature extraction network and the feature conversion network to be trained as the generation network based on the discriminant loss value obtained in step 3033 and the feature loss value obtained in step 3035, and perform adversarial training with the discriminant network. Specifically, the network parameter θ of the image graph attention convolution network can be set as follows: V , the network parameters θ of the text graph attention convolutional network T, the network parameters θ of the discriminant network D To optimize guided training:

[0117]

[0118]

[0119] Among them, L fea (θ V ,θ T ) represents the feature loss value, L adv (θ D ) represents the discriminant loss value.

[0120] When the difference between the feature loss value and the discriminant loss value reaches the maximum, the network parameter θ of the optimized image graph attention convolutional network is V , the network parameters θ of the text graph attention convolutional network T As the network parameters of the trained cross-media feature extraction network.

[0121] Through the above training steps, the cross-media feature extraction network can extract text features and image features with structured semantics, so that it has modal invariance, semantic discrimination and cross-modal semantic similarity.

[0122] Continue to refer Figure 4 , Figure 4 It is a schematic diagram of the architecture of an implementation process of the method for retrieving a target disclosed in the present invention.

[0123] like Figure 4 As shown, the system architecture may include an image graph attention network, a text graph attention network, and an adversarial learning module.

[0124] The image graph attention network is used to extract the image features of the image. The image graph attention network can be composed of five residual network modules, a visual scene graph module and a joint embedding layer, wherein the visual scene graph module can be composed of an image feature directed graph and a graph attention convolution layer, the graph attention convolution layer is used to update the image feature directed graph, and the joint embedding layer can be composed of three fully connected layers. Specifically, the above-mentioned execution entity can first use five residual network modules to extract the initial image features of the image, and then input the initial image features into the visual scene graph module to extract the structured image features of the image, and finally use the joint embedding layer to project the structured image features into the image-text common feature space.

[0125] The text graph attention network is used to extract text features of the text. The text graph attention network can be composed of a bidirectional LSTM (Long Short-Term Memory), a text scene graph module, and a joint embedding layer, wherein the text scene graph module can be composed of a text feature directed graph and a graph attention convolution layer, the graph attention convolution layer is used to update the text feature directed graph, and the joint embedding layer can be composed of three fully connected layers. Specifically, the above-mentioned execution entity can first use the bidirectional LSTM to extract the initial text features of the text, and then input the initial text features into the text scene graph module to extract the structured text features of the text, and finally use the joint embedding layer to project the structured text features into the image-text common feature space.

[0126] The adversarial learning module is used to determine the image-text common feature space of image features and text features, and the adversarial learning module can be composed of a feature converter and a modality discriminator. Specifically, the above-mentioned execution subject can first input the image features extracted by the image-graph attention network and the text features extracted by the text-graph attention network into the adversarial learning module, and the feature converter is used to project the features of different modal types (text features or image features) to the image-text common feature space to generate the converted features, and the modality discriminator is used to distinguish the modality type (text type or image type) of the converted features generated by the feature converter, and then use the image-graph attention network, the text-graph attention network features and the converter as the generation network, and the modality discriminator as the discriminant network to perform joint adversarial learning, and finally use the trained image-graph attention network and text-graph attention network features as the cross-media feature extraction network.

[0127] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a device for retrieving a target, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0128] like Figure 5 As shown, the device 500 for retrieving a target provided in this embodiment includes an acquisition unit 501, an extraction unit 502, and a matching unit 503. The acquisition unit 501 is configured to acquire at least one image and a description text of a specified object; the extraction unit 502 is configured to extract image features of the image and text features of the description text using a pre-trained cross-media feature extraction network; and the matching unit 503 is configured to match the image features with the text features to determine an image containing the specified object.

[0129] In this embodiment, in the device for retrieving a target 500, the specific processing of the acquisition unit 501, the extraction unit 502 and the matching unit 503 and the technical effects thereof can be referred to in Figure 2 The relevant descriptions of step 201, step 202 and step 203 in the corresponding embodiment are not repeated here.

[0130] In some optional implementations of the present embodiment, the cross-media feature extraction network is generated according to the following method: obtaining a training sample set, wherein the training sample set includes sample image-text pairs, and the sample image-text pairs include: a sample image and a sample text describing an object contained in the sample image; obtaining an initial network, wherein the initial network includes a cross-media feature extraction network to be trained, a discriminant network for discriminating the data category of the feature source, and a feature conversion network, and the cross-media feature extraction network to be trained includes an image-graph attention network and a text-graph attention network; inputting the sample image in the sample image-text pair into the image-graph attention network to obtain a first feature; inputting the sample text in the sample image-text pair into the text-graph attention network to obtain a second feature; inputting the first feature and the second feature into the discriminant network to obtain a category discrimination result, and calculating a discrimination loss value according to the category discrimination result. Among them, the data categories of the feature source include text category and image category, and the discrimination loss value represents the category judgment error of the first feature and the second feature; the first feature and the second feature are input into the feature transformation network to obtain the feature transformation result, and the value of the recognition loss function and the value of the pair loss function are calculated according to the feature transformation result, wherein the recognition loss function represents the ability of the first feature and the second feature to distinguish different objects contained in the common feature space of the image and text, and the pair loss function represents the semantic difference between the first feature and the second feature of the same object; based on the value of the recognition loss function and the value of the pair loss function, a preset feature loss value is obtained; based on the discrimination loss value and the feature loss value, the cross-media feature extraction network and the feature transformation network to be trained are used as the generation network, and adversarial training is performed with the discrimination network to obtain the trained cross-media feature extraction network, the discrimination network, and the feature transformation network.

[0131] In some optional implementations of this embodiment, the image-graph attention network includes a residual network, an image-graph attention convolutional network and a joint embedding layer; and the first feature is obtained in the following manner: using the residual network to extract the initial image features of the sample image, inputting the initial image features into the image-graph attention convolutional network, outputting the structured image features of the sample image, fusing the initial image features with the structured image features to generate image features, and inputting the image features into the joint embedding layer to obtain the first feature.

[0132] In some optional implementations of the present embodiment, the structured image features of the sample image are obtained in the following manner: performing target object detection on the sample image, determining the position of the target object and the rectangular bounding box of the target object in the sample image, and extracting relevant features of the target object based on the position of the rectangular bounding box of the target object, wherein the relevant features of the target object include at least one of the following: appearance features of the target object, position features of the target object, attribute features of the target object, and type features of the target object; constructing an image feature directed graph, wherein the vertices of the image feature directed graph represent the target objects, and the directed edges of the image feature directed graph represent the association relationship between the target objects; and generating the structured image features of the sample image based on the image feature directed graph.

[0133] In some optional implementations of the present embodiment, the text graph attention network includes a bidirectional long short-term memory network, a text graph attention convolutional network and a joint embedding layer; and the second feature is obtained in the following manner: the sample text is segmented to determine the word vector of the sample text; the initial text features of the word vector of the sample text are extracted using the bidirectional long short-term memory network, the initial text features are input into the text graph attention convolutional network, the structured text features of the sample text are output, the initial text features are fused with the structured text features to generate text features, and the text features are input into the joint embedding layer to obtain the second feature.

[0134] In some optional implementations of the present embodiment, the structured text features of the sample text are obtained in the following manner: a text feature directed graph is constructed, wherein the vertices of the text feature directed graph represent the target objects indicated by the word vectors, and the directed edges of the text feature directed graph represent the association relationships between the target objects indicated by the word vectors, wherein the relevant features of the target objects indicated by the word vectors include at least one of the following: attribute features of the target object and type features of the target object; and according to the text feature directed graph, the structured text features of the sample text are generated.

[0135] The device provided by the above-mentioned embodiment of the present disclosure obtains at least one image and a descriptive text of a specified object through an acquisition unit 501, and an extraction unit 502 uses a pre-trained cross-media feature extraction network to extract image features of the image and text features of the descriptive text. A matching unit 503 matches the image features with the text features to determine an image containing the specified object, thereby using cross-media feature extraction features to project the image features and text features into a common feature space of the image and text for feature matching, thereby achieving cross-media target retrieval.

[0136] Reference below Figure 6 , which shows an electronic device (eg, Figure 1 A schematic diagram of the structure of the server in FIG600. Figure 6 The server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0137] like Figure 6 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0138] Typically, the following devices may be connected to the I / O interface 605: input devices 605 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 6 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0139] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0140] It should be noted that the computer-readable medium of the embodiment of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiment of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the embodiment of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0141] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains at least one image and a descriptive text of a specified object; uses a pre-trained cross-media feature extraction network to extract image features of the image and text features of the descriptive text; matches the image features with the text features to determine the image containing the specified object.

[0142] Computer program code for performing the operations of embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as a separate software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0143] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0144] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The units described may also be set in a processor, for example, may be described as: a processor, including an acquisition unit, an extraction unit, and a matching unit. The names of these units do not constitute limitations on the units themselves in certain circumstances, for example, the acquisition unit may also be described as "a unit for acquiring at least one image and a description text of a specified object".

[0145] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) to form a technical solution.

Claims

1. A method for retrieving a target, comprising: Get at least one image and a description text of a specified object; Extracting image features of the image and text features of the description text using a pre-trained cross-media feature extraction network, wherein the cross-media feature extraction network projects the text features and the image features into a common feature space of image and text; Matching the image features with the text features to determine an image containing the specified object; The cross-media feature extraction network is generated in the following manner: Acquire a training sample set, wherein the training sample set includes sample image-text pairs; Obtaining an initial network, wherein the initial network includes a cross-media feature extraction network to be trained, a discriminant network, and a feature conversion network, and the cross-media feature extraction network to be trained includes an image graph attention network and a text graph attention network; Inputting the sample image in the sample image-text pair into the image-graph attention network to obtain a first feature; Inputting the sample text in the sample image-text pair into a text graph attention network to obtain a second feature; Inputting the first feature and the second feature into the discriminant network to obtain a category discrimination result, and calculating a discrimination loss value according to the category discrimination result; Inputting the first feature and the second feature into the feature conversion network to obtain a feature conversion result, and calculating a value of a recognition loss function and a value of a pairwise loss function according to the feature conversion result; Based on the value of the recognition loss function and the value of the pairwise loss function, a preset feature loss value is obtained; Based on the discriminant loss value and the feature loss value, the cross-media feature extraction network to be trained and the feature conversion network are used as generation networks, and adversarial training is performed with the discriminant network to obtain trained cross-media feature extraction network, discriminant network, and feature conversion network.

2. The method according to claim 1, wherein: The image-graph attention network includes a residual network, an image-graph attention convolutional network, and a joint embedding layer; And the step of inputting the sample image into the image graph attention network to obtain the first feature includes: The residual network is used to extract the initial image features of the sample image, and the initial image features are input into the image graph attention convolutional network to output the structured image features of the sample image. The initial image features are fused with the structured image features to generate image features, and the image features are input into the joint embedding layer to obtain the first features.

3. The method according to claim 2, wherein: The initial image features are input into the image graph attention convolutional network, and the structured image features of the sample image are output, including: Performing target object detection on the sample image, determining the position of the target object and the rectangular bounding box of the target object in the sample image, and extracting relevant features of the target object according to the position of the rectangular bounding box of the target object, wherein the relevant features of the target object include at least one of the following: appearance features of the target object, position features of the target object, and type features of the target object; Constructing an image feature directed graph, wherein the vertices of the image feature directed graph represent the target objects, and the directed edges of the image feature directed graph represent the association relationship between the target objects; According to the image feature directed graph, structured image features of the sample image are generated.

4. The method according to claim 1, wherein: The text graph attention network includes a bidirectional long short-term memory network, a text graph attention convolutional network and a joint embedding layer; and the inputting the sample text into the text graph attention network to obtain the second feature includes: Perform word segmentation on the sample text to determine the word vector of the sample text; A bidirectional long short-term memory network is used to extract the initial text features of the word vector of the sample text, the initial text features are input into the text graph attention convolutional network, the structured text features of the sample text are output, the initial text features are fused with the structured text features to generate text features, and the text features are input into the joint embedding layer to obtain the second features.

5. The method according to claim 4, wherein: The initial text features are input into the text graph attention convolutional network, and the structured text features of the sample text are output, including: Constructing a text feature directed graph, wherein the vertices of the text feature directed graph represent the target objects indicated by the word vectors, and the directed edges of the text feature directed graph represent the association relationship between the target objects indicated by the word vectors, wherein the relevant features of the target objects indicated by the word vectors include at least one of the following: an attribute feature of the target object and a type feature of the target object; According to the text feature directed graph, structured text features of the sample text are generated.

6. A device for retrieving a target, comprising: An acquisition unit, configured to acquire at least one image and a description text of a designated object; an extraction unit configured to extract image features of the image and text features of the description text using a pre-trained cross-media feature extraction network, wherein the cross-media feature extraction network projects the text features and the image features into a common feature space of image and text; A matching unit, configured to match the image feature with the text feature to determine an image containing the specified object; The cross-media feature extraction network is generated in the following manner: Acquire a training sample set, wherein the training sample set includes sample image-text pairs; Obtaining an initial network, wherein the initial network includes a cross-media feature extraction network to be trained, a discriminant network, and a feature conversion network, and the cross-media feature extraction network to be trained includes an image graph attention network and a text graph attention network; Inputting the sample image in the sample image-text pair into the image-graph attention network to obtain a first feature; Inputting the sample text in the sample image-text pair into a text graph attention network to obtain a second feature; Inputting the first feature and the second feature into the discriminant network to obtain a category discrimination result, and calculating a discrimination loss value according to the category discrimination result; Inputting the first feature and the second feature into the feature conversion network to obtain a feature conversion result, and calculating a value of a recognition loss function and a value of a pairwise loss function according to the feature conversion result; Based on the value of the recognition loss function and the value of the pairwise loss function, a preset feature loss value is obtained; Based on the discriminant loss value and the feature loss value, the cross-media feature extraction network to be trained and the feature conversion network are used as generation networks, and adversarial training is performed with the discriminant network to obtain trained cross-media feature extraction network, discriminant network, and feature conversion network.

7. The device according to claim 6, wherein: The image-graph attention network includes a residual network, an image-graph attention convolutional network, and a joint embedding layer; and the first feature is obtained as follows: The residual network is used to extract the initial image features of the sample image, and the initial image features are input into the image graph attention convolutional network to output the structured image features of the sample image. The initial image features are fused with the structured image features to generate image features, and the image features are input into the joint embedding layer to obtain the first features.

8. The device according to claim 7, wherein: The structured image features of the sample image are obtained in the following manner: Performing target object detection on the sample image, determining the position of the target object and the rectangular bounding box of the target object in the sample image, and extracting relevant features of the target object according to the position of the rectangular bounding box of the target object, wherein the relevant features of the target object include at least one of the following: appearance features of the target object, position features of the target object, and type features of the target object; Constructing an image feature directed graph, wherein the vertices of the image feature directed graph represent the target objects, and the directed edges of the image feature directed graph represent the association relationship between the target objects; According to the image feature directed graph, structured image features of the sample image are generated.

9. The device according to claim 6, wherein: The text-graph attention network includes a bidirectional long short-term memory network, a text-graph attention convolutional network, and a joint embedding layer; and the second feature is obtained as follows: Perform word segmentation on the sample text to determine the word vector of the sample text; A bidirectional long short-term memory network is used to extract the initial text features of the word vector of the sample text, the initial text features are input into the text graph attention convolutional network, the structured text features of the sample text are output, the initial text features are fused with the structured text features to generate text features, and the text features are input into the joint embedding layer to obtain the second features.

10. The device according to claim 9, wherein: The structured text features of the sample text are obtained in the following manner: Constructing a text feature directed graph, wherein the vertices of the text feature directed graph represent the target objects indicated by the word vectors, and the directed edges of the text feature directed graph represent the association relationship between the target objects indicated by the word vectors, wherein the relevant features of the target objects indicated by the word vectors include at least one of the following: an attribute feature of the target object and a type feature of the target object; According to the text feature directed graph, structured text features of the sample text are generated.

11. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, When one or more programs are executed by one or more processors, the one or more processors implement the method as claimed in any one of claims 1 to 5.

12. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images