Visual relationship detection method, electronic device, storage medium and program product

Abstract

The present invention provides a visual relationship detection method, an electronic device, a storage medium and a program product. The visual relationship detection method comprises: determining an image to be detected, inputting the image to be detected into a visual relationship detection model, and obtaining a visual relationship detection result output by the visual relationship detection model; wherein the visual relationship detection model is obtained by training based on an image training set including visible classes and invisible classes, the visible classes are images marked with visual relationship labels, and the invisible classes are images not marked with visual relationship labels. The present invention can pre-learn the visual relationship corresponding to the visible class and the invisible class, thereby realizing accurate detection of images containing the invisible class, and finally realizing zero-sample visual relationship detection.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a visual relationship detection method, electronic equipment, storage medium and program product. Background Art

[0002] Visual relations are used to represent objects in an image and their relationships, which can be represented by triples (subject-predicate-object). Usually, visual relations are used to generate scene graphs. As a structured image semantic representation, scene graphs have been widely used in visual understanding tasks such as image description and visual question answering. Since scene graphs represent objects in an image and their visual relationships as graph structures, structured semantics can be extracted from unstructured image data.

[0003] Currently, visual relationship detection mostly relies on large-scale manually labeled datasets. However, due to the complexity of visual relationships between objects in reality, it is very expensive to label all existing visual relationships, making it impossible to label comprehensive visual relationship labels.

[0004] Therefore, the current visual relationship detection methods are all visual relationship detection methods for visible visual relationships in the training set, that is, they can only detect visual relationships corresponding to visible classes in the image training set. If the visual relationship corresponding to the image to be detected has never been annotated in the image training set, that is, there are no invisible class sample images in the image training set, then the current visual relationship detection methods cannot accurately detect the visual relationship of the image to be detected. Summary of the invention

[0005] The present invention provides a visual relationship detection method, electronic device, storage medium and program product, which are used to solve the defect that the existing visual detection method cannot detect invisible visual relationships in a training set, and realize high-accuracy visual relationship detection.

[0006] The present invention provides a visual relationship detection method, comprising:

[0007] Determine an image to be detected;

[0008] Inputting the image to be detected into a visual relationship detection model to obtain a visual relationship detection result output by the visual relationship detection model;

[0009] The visual relationship detection model is trained based on visible class sample images and their labels, as well as invisible class labels.

[0010] According to a visual relationship detection method provided by the present invention, the step of inputting the image to be detected into a visual relationship detection model to obtain a visual relationship detection result output by the visual relationship detection model comprises:

[0011] Inputting the image to be detected into the object feature representation layer of the visual relationship detection model to obtain multiple object feature representations output by the object feature representation layer;

[0012] Inputting visual relationship representations corresponding to any two object feature representations among the multiple object feature representations into a feature reconstruction layer of the visual relationship detection model to obtain reconstructed features output by the feature reconstruction layer, wherein the feature reconstruction layer is obtained by training based on a first loss function determined based on the visual relationship feature center of the image training set and the reconstructed feature training data corresponding to the image training set;

[0013] The reconstructed features are input into the visual relationship prediction layer of the visual relationship detection model to obtain a visual relationship detection result output by the visual relationship prediction layer, wherein the visual relationship prediction layer is trained based on a second loss function determined based on the visual relationship category probability and the visual relationship label of the visible class sample image, and the visual relationship category probability is obtained by performing distance judgment based on the reconstructed feature training data and the visual relationship label of the visible class sample image.

[0014] According to a visual relationship detection method provided by the present invention, the step of inputting the image to be detected into the object feature representation layer of the visual relationship detection model to obtain multiple object feature representations output by the object feature representation layer includes:

[0015] Inputting the image to be detected into the object detection layer of the object feature representation layer to obtain a plurality of visual features, a plurality of category labels and a plurality of coordinate information output by the object detection layer;

[0016] The multiple visual features, the semantic features of the multiple category labels and the position features of the multiple coordinate information are input into the feature fusion layer of the object feature representation layer to obtain multiple object feature representations output by the feature fusion layer.

[0017] According to a visual relationship detection method provided by the present invention, the inputting of the multiple visual features, the semantic features of the multiple category labels and the position features of the multiple coordinate information into the feature fusion layer of the object feature representation layer to obtain multiple object feature representations output by the feature fusion layer includes:

[0018] Inputting the multiple visual features into the attention layer of the object feature representation layer, performing feature transformation, and obtaining multiple object attention features output by the attention layer, wherein the attention layer is obtained by training based on the semantic feature training data corresponding to the image training set and a third loss function determined by preset grammatical knowledge, and the preset grammatical knowledge is the hyponym relationship between object categories;

[0019] The multiple object attention features, the multiple semantic features and the multiple position features are input into the feature fusion layer to obtain the object feature representation output by the feature fusion layer.

[0020] According to a visual relationship detection method provided by the present invention, the step of inputting visual relationship representations corresponding to any two object feature representations among the plurality of object feature representations into a feature reconstruction layer of the visual relationship detection model to obtain reconstructed features output by the feature reconstruction layer includes:

[0021] Inputting the visual relationship representation into the associative vector mapping layer of the feature reconstruction layer to obtain the associative vector output by the associative vector mapping layer;

[0022] The association vector is input to the memory feature extraction layer of the feature reconstruction layer to obtain the memory feature output by the memory feature extraction layer, wherein the memory feature extraction layer is used to determine the memory feature based on the association vector and the visual relationship feature center of the image training set, the visual relationship feature center of the visible class sample image is obtained by performing cluster center calculation based on the visual relationship label of the visible class sample image, and the visual relationship feature center of the invisible class is generated based on the semantic association between the visible class and the invisible class;

[0023] Inputting the visual relationship representation into the category selection layer of the feature reconstruction layer to obtain a category vector output by the category selection layer, wherein the category vector is used to represent the dependency scores of different visual relationship categories on the memory feature;

[0024] The visual relationship representation, the memory feature and the category vector are input into the feature calculation layer of the feature reconstruction layer to obtain the reconstructed features output by the feature calculation layer.

[0025] According to a visual relationship detection method provided by the present invention, the visual relationship representation is obtained by splicing any two object feature representations from the multiple object feature representations.

[0026] According to a visual relationship detection method provided by the present invention, the step of inputting the reconstructed features into a visual relationship prediction layer of the visual relationship detection model to obtain a visual relationship detection result output by the visual relationship prediction layer comprises:

[0027] Inputting the reconstructed features into the semantic space mapping layer of the visual relationship prediction layer to obtain the semantic mapping features output by the semantic space mapping layer;

[0028] Inputting the semantic mapping features into the semantic space layer of the visual relationship prediction layer to obtain the visual relationship category probability output by the semantic space layer, wherein the semantic space layer is used to perform distance calculation based on the semantic mapping features and the visual relationship labels in the same semantic space to obtain the visual relationship category probability;

[0029] The visual relationship category probability is input into the output layer of the visual relationship prediction layer to obtain the visual relationship detection result output by the output layer.

[0030] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the visual relationship detection methods described above are implemented.

[0031] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the visual relationship detection methods described above are implemented.

[0032] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of visual relationship detection as described above are implemented.

[0033] The visual relationship detection method, electronic device, storage medium and program product provided by the present invention determine the image to be detected, input the image to be detected into the visual relationship detection model, and obtain the visual relationship detection result output by the visual relationship detection model. The visual relationship detection model is trained based on the visible class sample image and its label and the invisible class label, so that the visual relationship detection model can pre-learn the visual relationship corresponding to the visible class and the invisible class. Therefore, even if the visual relationship of the image to be detected belongs to the invisible visual relationship in the image training set, that is, the image training set does not contain sample images with visual relationship labels marked with the visual relationship, accurate visual relationship detection results can be obtained, thereby achieving accurate detection of images containing invisible classes, and finally achieving zero-sample visual relationship detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 One of the flow charts of the visual relationship detection method provided by the present invention;

[0036] Figure 2 The second flowchart of the visual relationship detection method provided by the present invention;

[0037] Figure 3 This is a schematic structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] Visual relations are used to represent objects in an image and their relationships, which can be represented by triples (subject-predicate-object). Usually, visual relations are used to generate scene graphs. As a structured image semantic representation, scene graphs have been widely used in visual understanding tasks such as image description and visual question answering. Since scene graphs represent objects in an image and their visual relationships as graph structures, structured semantics can be extracted from unstructured image data.

[0040] Most scene graph generation methods rely on large-scale manually annotated datasets. However, given the complexity of visual relationships between objects in reality, it is undoubtedly costly or even impossible to annotate all existing visual relationships. Therefore, it is considered to introduce zero-shot tasks into scene graph generation tasks. However, these methods will follow the traditional definition of scene graphs and take visual relationship triplets (subject, predicate, object) as the basic structure of visual relationships in scene graphs. This means that the visual relationship triplets that need to be inferred in the zero-shot tasks they define have no visible samples, but the elements in the triplets (subject, predicate, object) are visible in the training set. Such a definition will make it impossible for the model to reason when introducing new subjects, predicates, or objects. Therefore, it is very important to achieve true zero-shot visual relationship reasoning.

[0041] At present, memory mechanism has achieved great success in few-shot learning. However, memory mechanism has not been applied to zero-shot learning tasks. In order to enable memory mechanism to effectively model the unseen visual relationship feature prototypes in zero-shot visual relationship reasoning tasks, and to use the unseen visual relationship feature prototypes as the basis for model reasoning to establish a robust zero-shot visual relationship reasoning model, the following two aspects are worth considering: (1) Object feature representation learning. The feature representation of objects is the basis of visual relationship reasoning. However, due to the imbalance of data distribution, the features of objects of the same type are relatively scattered, which is not conducive to the learning of visual relationship feature prototypes. Therefore, it is necessary to design a method to establish a compact feature space; (2) Feature prototype modeling. Visual relationship feature prototype modeling is crucial in zero-shot visual relationship reasoning. However, since the unseen class cannot directly obtain features, it is necessary to design a method to model the connection between the visible class and the unseen class, and establish the feature prototype of the unseen class through this connection.

[0042] Visual relationship detection mostly relies on large-scale manually annotated datasets. However, due to the complexity of visual relationships between objects in reality, it is very expensive to annotate all existing visual relationships, making it impossible to annotate comprehensive visual relationship labels.

[0043] The current visual relationship detection methods are all visual relationship detection methods for the visible visual relationships in the training set, that is, they can only detect the visual relationships corresponding to the visible classes in the image training set. If the visual relationship corresponding to the image to be detected has never been annotated in the image training set, the current visual relationship detection methods cannot accurately detect the visual relationship of the image to be detected.

[0044] To this end, the present invention provides the following embodiments.

[0045] Figure 1 One of the flow charts of the visual relationship detection method provided by the present invention is as follows: Figure 1 As shown, the visual relationship detection method provided by the present invention includes the following steps 110-120:

[0046] Step 110: determine the image to be detected.

[0047] In this embodiment, the image to be detected is an image that needs to be subjected to visual relationship detection.

[0048] The image to be detected can be acquired by an image acquisition device or a video acquisition device, and can also be an originally stored image, which is not specifically limited here. The image acquisition device can be a computer, a smart phone, a tablet computer, a camera, a video camera, etc., and the video acquisition device can be a computer, a smart phone, a tablet computer, a camera, a video camera, etc., which is not specifically limited here.

[0049] In one embodiment, the image to be detected may also be an image for which a scene graph needs to be generated, that is, the visual relationship of the image to be detected is obtained through the visual relationship detection method of the present invention, so as to generate a corresponding scene graph based on the visual relationship.

[0050] It should be noted that the image to be detected can be determined to be an image that already meets the input requirements of the visual relationship detection model. If the image to be detected does not meet the input requirements of the visual relationship detection model, the image to be detected can be preprocessed (such as scaling, denoising, etc.) to obtain an image to be detected that meets the input requirements of the visual relationship detection model. Of course, if the image to be detected meets the input requirements of the visual relationship detection model, it can also be preprocessed so that the subsequent visual relationship detection model can better perform visual relationship detection to improve the accuracy of visual relationship detection.

[0051] Step 120, inputting the image to be detected into a visual relationship detection model to obtain a visual relationship detection result output by the visual relationship detection model;

[0052] The visual relationship detection model is trained based on visible class sample images and their labels, as well as invisible class labels.

[0053] In this embodiment, the visual relationship detection result is the visual relationship of the image to be detected, and the visual relationship is used to represent the objects in the image to be detected and the relationship between them. Specifically, the visual relationship is represented by "subject-predicate-object".

[0054] The visual relationship detection model is obtained by training an image training set, and the images in the image training set can be any images. In one embodiment, the image training set can be a visual genome dataset.

[0055] It should be noted that, since the image training set includes visible class sample images (also referred to as visible classes) and their visual relationship labels, as well as invisible class visual relationship labels, the visual relationship detection model can pre-learn the visual relationships corresponding to the visible and invisible classes. Therefore, even if the visual relationship of the image to be detected belongs to an invisible visual relationship in the image training set, that is, the image training set does not contain images corresponding to the visual relationship labels annotated with the visual relationship, accurate visual relationship detection results can be obtained, thereby achieving accurate detection of images containing invisible classes.

[0056] In one embodiment, the visible class may be images with known visual relationships, and the invisible class may be images with complex and unknown visual relationships.

[0057] In addition, before executing the above step 120, a visual relationship detection model may be pre-trained. Specifically, the visual relationship detection model may be trained in the following manner:

[0058] Input images in an image training set into the object feature representation layer of a model to be trained to obtain multiple object feature representations output by the object feature representation layer, wherein the image training set includes visible class sample images and their labels and invisible class labels; input visual relationship representations corresponding to any two object feature representations in the multiple object feature representations into the feature reconstruction layer of the model to be trained to obtain reconstructed features output by the feature reconstruction layer; input the reconstructed features into the visual relationship prediction layer of the model to be trained to obtain visual relationship prediction results output by the visual relationship prediction layer; determine a second loss function based on the visual relationship prediction results and the visual relationship labels of the visible class; and train the model to be trained based on the second loss function to obtain the visual relationship detection model.

[0059] The initial model of the model to be trained can be a single neural network model or a combination of multiple neural network models.

[0060] It should be noted that the model to be trained can be trained on the application end of the visual relationship detection method of the present invention, or it can be trained on another training end and then deployed on the application end.

[0061] Furthermore, the step of inputting the images in the image training set into the object feature representation layer of the model to be trained to obtain a plurality of object feature representations output by the object feature representation layer includes:

[0062] The images in the image training set are input into the object detection layer of the object feature representation layer to obtain multiple visual features, multiple category labels and multiple coordinate information output by the object detection layer; the multiple visual features, the semantic features of the multiple category labels and the position features of the multiple coordinate information are input into the feature fusion layer of the object feature representation layer to obtain multiple object feature representations output by the feature fusion layer.

[0063] Furthermore, the step of inputting the plurality of visual features, the semantic features of the plurality of category labels, and the position features of the plurality of coordinate information into a feature fusion layer of the object feature representation layer to obtain a plurality of object feature representations output by the feature fusion layer includes:

[0064] The multiple visual features are input into the attention layer of the object feature representation layer to perform feature transformation to obtain multiple object attention features output by the attention layer, wherein the attention layer is trained based on a third loss function determined based on the semantic features and preset grammatical knowledge, and the preset grammatical knowledge is the semantics of the upper and lower meanings of the object; the multiple object attention features, the multiple semantic features and the multiple position features are input into the feature fusion layer to obtain the object feature representation output by the feature fusion layer.

[0065] Furthermore, the step of inputting the visual relationship representation corresponding to any two object feature representations among the plurality of object feature representations into the feature reconstruction layer of the model to be trained to obtain the reconstructed features output by the feature reconstruction layer includes:

[0066] The visual relationship representation is input into the associative vector mapping layer of the feature reconstruction layer to obtain the associative vector output by the associative vector mapping layer; the associative vector is input into the memory feature extraction layer of the feature reconstruction layer to obtain the memory feature output by the memory feature extraction layer, the memory feature extraction layer is used to determine the memory feature based on the associative vector and the visual relationship feature center of the image training set, the visual relationship feature center of the visible class sample image is obtained by clustering center calculation based on the visual relationship label of the visible class sample image, and the visual relationship feature center of the invisible class is generated based on the semantic association between the visible class and the invisible class; the visual relationship representation is input into the category selection layer of the feature reconstruction layer to obtain the category vector output by the category selection layer, the category vector is used to represent the dependency score of different visual relationship categories on the memory feature; the visual relationship representation, the memory feature and the category vector are input into the feature calculation layer of the feature reconstruction layer to obtain the reconstructed feature output by the feature calculation layer.

[0067] It can be understood that the specific execution process of the training method of the above-mentioned visual relationship detection model is basically the same as the execution process of the application process of the visual relationship detection model. Based on this, you can refer to the following embodiments of the application process, and the training process will not be described here one by one.

[0068] According to the visual relationship detection method of the embodiment of the present invention, by determining the image to be detected, the image to be detected is input into the visual relationship detection model, and the visual relationship detection result output by the visual relationship detection model is obtained. Among them, the visual relationship detection model is obtained based on training including visible class sample images and their labels and invisible class labels, so that the visual relationship detection model can pre-learn the visual relationship corresponding to the visible class and the invisible class. Therefore, even if the visual relationship of the image to be detected belongs to an invisible visual relationship in the image training set, that is, the image training set does not contain sample images with visual relationship labels marked with the visual relationship, accurate visual relationship detection results can be obtained, thereby achieving accurate detection of images containing invisible classes, and finally achieving zero-sample visual relationship detection.

[0069] Furthermore, based on the above embodiment, another embodiment of the visual relationship detection method of the present invention is proposed. Figure 2 The second flow chart of the visual relationship detection method provided by the present invention is as follows: Figure 2 As shown, in this embodiment, the above step 120 includes the following steps 121-123:

[0070] Step 121: input the image to be detected into the object feature representation layer of the visual relationship detection model to obtain multiple object feature representations output by the object feature representation layer.

[0071] The object feature representation layer is used to determine the object feature representation corresponding to each object in the image to be detected based on the image to be detected.

[0072] Step 122, input the visual relationship representation corresponding to any two object feature representations among the multiple object feature representations to the feature reconstruction layer of the visual relationship detection model to obtain the reconstructed features output by the feature reconstruction layer, wherein the feature reconstruction layer is trained based on a first loss function determined based on the visual relationship feature center of the image training set and the reconstructed feature training data corresponding to the image training set.

[0073] Wherein, any visual relationship representation is used to represent the visual relationship between any two objects in the image to be detected. Any visual relationship can be obtained by aggregating the feature representations of two objects.

[0074] The feature reconstruction layer is used to map the visual relationship representation to the reconstructed features. During the training process, the feature reconstruction layer learns the mapping relationship between the visual relationship representation and the reconstructed features, so that the visual relationship representation can be converted into the reconstructed features.

[0075] The visual relationship feature center of the image training set is obtained by performing clustering calculation on the image training set based on the visual relationship. The clustering calculation method is not limited here.

[0076] The reconstructed feature training data is the reconstructed features obtained during the training process, which are basically the same as the reconstructed features during the application process.

[0077] In one embodiment, any two object feature representations among the multiple object feature representations are input into the feature fusion layer of the visual relationship detection model to perform feature fusion to obtain the visual relationship output by the feature fusion layer.

[0078] Among them, the feature fusion layer can be a concat network structure, which is not specifically limited here.

[0079] In another embodiment, the visual relationship representation is obtained by splicing any two object feature representations from the multiple object feature representations.

[0080] For example, a visual relationship representation can be expressed as follows:

[0081] q ij =[t i ;t j ],

[0082] Among them, q ij is the visual relationship representation, t i is the object feature representation of the i-th object in the image to be detected, t j is the object feature representation of the j-th object in the image to be detected, and [;] represents the feature concatenation operation.

[0083] In one embodiment, the first loss function is as follows:

[0084]

[0085] in, To reconstruct the feature training data, c k is the cluster center corresponding to the kth element in the current visual relationship representation in the visual relationship feature center of the image training set, K is the number of all visual relationships in the visual relationship representation, k is the kth element in the current visual relationship representation, k′ is the element other than k, ∈ 2 is a manually set threshold, for example, ∈ 2 =12.

[0086] Step 123, input the reconstructed feature to the visual relationship prediction layer of the visual relationship detection model to obtain the visual relationship detection result output by the visual relationship prediction layer, wherein the visual relationship prediction layer is trained based on a second loss function determined based on the visual relationship category probability and the visual relationship label of the visible class, and the visual relationship category probability is obtained by distance judgment based on the reconstructed feature training data and the visual relationship label of the visible class.

[0087] Among them, the visual relationship prediction layer is used to calculate the distance between the reconstructed feature and each visual relationship label, obtain the category probability of the visual relationship, and obtain the visual relationship detection result based on the category probability.

[0088] In one embodiment, the probability that the image to be detected corresponds to each visual relationship can be determined based on the distance between the reconstructed feature and each visual relationship label, and the visual relationship with the greatest probability is used as the visual relationship detection result. In another embodiment, the visual relationship label closest to the reconstructed feature can be determined as the visual relationship detection result.

[0089] In one embodiment, the second loss function is as follows:

[0090]

[0091] in, Represents two objects o in the image i , o j The true visual relationship label between Represents two objects o i , o j The visual relationship between probability.

[0092] According to the visual relationship detection method of the embodiment of the present invention, the object feature representation of the image to be detected is determined by the object feature characterization layer of the visual relationship detection model, and then the reconstructed features are determined by the feature reconstruction layer of the visual relationship detection model, and then the visual relationship detection is performed by the visual relationship prediction layer of the visual relationship detection model, thereby achieving accurate detection of images containing invisible classes and ultimately achieving zero-sample visual relationship detection.

[0093] Further, based on any of the above embodiments, another embodiment of the visual relationship detection method of the present invention is proposed. In this embodiment, the above step 121 includes the following steps 1211-1212:

[0094] Step 1211, inputting the image to be detected into the object detection layer of the object feature representation layer, and obtaining a plurality of visual features, a plurality of category labels and a plurality of coordinate information output by the object detection layer;

[0095] Among them, the object detection layer is used to determine the visual features, category labels and coordinate information corresponding to the image to be detected based on the image to be detected.

[0096] It should be noted that one object in the image to be detected corresponds to one visual feature, one category label and one coordinate information.

[0097] In one embodiment, the object detection layer may be a target detection layer, which is a network model constructed by a target detection algorithm. The target detection layer may be a Faster R-CNN model, an R-CNN model, an SSD model, etc., which are not specifically limited here.

[0098] In a specific embodiment, the visual features can be extracted by the ROI pooling layer of the Faster R-CNN model. Of course, the output of any layer in the target detection layer can also be extracted. For example, by extracting through the ROI pooling layer of the Faster R-CNN model, a visual feature with a dimension of 3*14*14 is obtained.

[0099] Among them, the category label can be obtained after target detection by the target detection layer, and the coordinate information can also be obtained after target detection by the target detection layer.

[0100] Specifically, the image to be detected is input into the object detection layer, multiple candidate boxes corresponding to multiple objects in the image to be detected are determined, and then the visual features, category labels and coordinate information of each candidate box are obtained for output by the object detection layer.

[0101] Step 1212: input the multiple visual features, the semantic features of the multiple category labels, and the position features of the multiple coordinate information into the feature fusion layer of the object feature representation layer to obtain multiple object feature representations output by the feature fusion layer.

[0102] The feature fusion layer is used to fuse the visual features, the semantic features of the category labels, and the position features of the coordinate information to obtain the object feature representation. The feature fusion layer can be a concat network structure, which is not specifically limited here.

[0103] It should be noted that if the dimensions of the visual features, the semantic features of the category labels, and the positional features of the coordinate information are different, a dimensional conversion can be performed to make the dimensions of each feature the same, thereby achieving a feature fusion operation.

[0104] In another embodiment, the object feature representation is obtained by concatenating visual features, semantic features of category labels, and position features of coordinate information.

[0105] For example, object feature representation can be expressed as follows:

[0106]

[0107] Among them, t is the object feature representation, f is the visual feature, is the semantic feature of the category label, l is the position feature of the coordinate information, and [;;] represents the feature concatenation operation.

[0108] Among them, the semantic features of the category labels can be extracted through a pre-trained word vector model, for example, a word2vec model or a glove model, etc., which is not specifically limited here.

[0109] For example, the category label is input into the word2vec model to obtain the semantic feature output by the word2vec model, and the semantic feature can be a 128-dimensional text feature.

[0110] The position feature of the coordinate information is obtained by encoding the coordinate information, for example, the coordinate information is linearly transformed through the object coordinates to obtain a 128-dimensional position feature.

[0111] According to the visual relationship detection method of the embodiment of the present invention, the visual features, category labels and coordinate information of each object in the image to be detected are determined through the object detection layer of the visual relationship detection model, and then the object feature representation is determined through the feature fusion layer of the visual relationship detection model, and then the reconstructed features are determined through the feature reconstruction layer of the visual relationship detection model, and then visual relationship detection is performed through the visual relationship prediction layer of the visual relationship detection model, thereby achieving accurate detection of images containing invisible classes and ultimately achieving zero-sample visual relationship detection.

[0112] Further, based on any of the above embodiments, another embodiment of the visual relationship detection method of the present invention is proposed. In this embodiment, the above step 1212 includes the following steps 12121-12122:

[0113] Step 12121, input the multiple visual features into the attention layer of the object feature representation layer, perform feature transformation, and obtain multiple object attention features output by the attention layer, wherein the attention layer is trained based on the semantic feature training data corresponding to the image training set and a third loss function determined by preset grammatical knowledge, and the preset grammatical knowledge is the hyponym-hyponym relationship between object categories.

[0114] Step 12122: input the multiple object attention features, the multiple semantic features, and the multiple position features into the feature fusion layer to obtain the object feature representation output by the feature fusion layer.

[0115] The attention layer is used to map visual features to object attention features. During the training process, the attention layer learns the attention function, thereby mapping visual features to object attention features based on the attention function. The attention function is used to transform the visual features.

[0116] Among them, the semantic feature training data is the semantic features in the training process, which are basically the same as the semantic features in the application process and will not be repeated here.

[0117] It can be understood that the attention layer is an attention network based on grammatical knowledge.

[0118] In one embodiment, the attention function is as follows:

[0119] att(f)=σ(τ([f max ;f avg ])),

[0120] Where att(f) is the attention function of the visual feature f; σ represents the activation function, such as the sigmoid function; τ represents the convolution operation, f max and f avg They represent the output of f after maximum pooling and average pooling respectively, and [;] represents the feature concatenation operation.

[0121] For ease of understanding, the object attention features are shown below:

[0122] f att =f×att(f),

[0123] Among them, f att is the object attention feature, f is the visual feature, and att(f) is the attention function.

[0124] In one embodiment, the synonym and hyponym list structure defined in an online dictionary (eg, WordNet) may be used as lexical knowledge.

[0125] In one embodiment, the third loss function is as follows:

[0126]

[0127] Among them, i is the i-th object in the image; A collection of objects with the same meaning; is the semantic feature of the category label of the i-th object; Its semantic feature;∈ 1 is a manually set threshold, for example ∈ 1 is 8. This semantic feature It can also be extracted through a pre-trained word vector model, for example, a word2vec model or a glove model, etc., which is not specifically limited here.

[0128] According to the visual relationship detection method of an embodiment of the present invention, the object attention features corresponding to the visual features are determined through the attention layer of the object feature representation layer, and a compact feature distribution can be constructed based on the attention network based on lexical knowledge to constrain objects with the same superordinate semantics to share the same attention weight, thereby eliminating differences within the object class, so that the feature reconstruction layer of the subsequent visual relationship detection model can learn a more robust feature prototype.

[0129] Further, based on any of the above embodiments, another embodiment of the visual relationship detection method of the present invention is proposed. In this embodiment, the above step 122 includes the following steps 1221-1224:

[0130] Step 1221: input the visual relationship representation to the associative vector mapping layer of the feature reconstruction layer to obtain the associative vector output by the associative vector mapping layer.

[0131] The associative vector mapping layer is used to map the visual relationship representation to the associative vector. During the training process, the associative vector mapping layer learns the mapping relationship between the visual relationship representation and the associative vector, thereby mapping the visual relationship representation to the associative vector based on the associative vector mapping layer.

[0132] The associative vector mapping layer can be any neural network, for example, a lightweight neural network T map Representing the visual relationship q ij Mapped to the associative vector θ ij =T map (q ij ).

[0133] In one embodiment, any two object feature representations among the multiple object feature representations are input into the feature fusion layer of the visual relationship detection model to perform feature fusion to obtain the visual relationship output by the feature fusion layer.

[0134] Among them, the feature fusion layer can be a concat network structure, which is not specifically limited here.

[0135] In another embodiment, the visual relationship representation is obtained by splicing any two object feature representations from the multiple object feature representations.

[0136] For example, a visual relationship representation can be expressed as follows:

[0137] q ij =[t i ;t j ],

[0138] Among them, q ij is the visual relationship representation, t iis the object feature representation of the i-th object in the image to be detected, t j is the object feature representation of the j-th object in the image to be detected, and [;] represents the feature concatenation operation.

[0139] Step 1222, input the associative vector to the memory feature extraction layer of the feature reconstruction layer to obtain the memory feature output by the memory feature extraction layer, the memory feature extraction layer is used to determine the memory feature based on the associative vector and the visual relationship feature center of the image training set, the visual relationship feature center of the visible class sample image is obtained by calculating the clustering center based on the visual relationship label of the visible class sample image, and the visual relationship feature center of the invisible class is generated based on the semantic association between the visible class and the invisible class.

[0140] Among them, the memory feature extraction layer is used to extract the memory features of the visual relationship between two objects.

[0141] In one embodiment, the formula for calculating memory features in the memory feature extraction layer is as follows:

[0142]

[0143] Among them, m ij is the memory feature, K is the number of all visual relations in the visual relationship representation, Represents the association vector θ ij The kth element in c k is the cluster center corresponding to the kth element in the current visual relationship representation in the visual relationship feature center of the image training set.

[0144] In a specific embodiment, the method for calculating the cluster center of the visible class sample image is: first, neighborhood sampling is performed, that is, during training, intra-class and inter-class samples are sampled to form a small batch of data, and then, these samples are grouped according to their visual relationship labels, and the cluster center of each group is updated by the merged features of this small batch of data.

[0145] It can be understood that the model then iteratively updates the visual relationship representation and cluster centers to minimize the distance between each merged feature and the cluster center of its group and maximize the distance to other cluster centers.

[0146] In a specific embodiment, the visual relationship feature center of the invisible class is established through semantic relevance. The calculation formula of the visual relationship feature center of the invisible class is as follows:

[0147]

[0148] in, is the visual relationship feature center of the u-th invisible class; represents the visual relationship feature center set of the visible class whose distance to the uth invisible class in the semantic space is less than the threshold ρ, |ε| is the number of the set, and the threshold ρ can be set according to actual needs, for example, ρ = 8; is the visual relationship feature center (cluster center) of the nth visible class.

[0149] It should be noted that, although there are no samples of the invisible class in the image training set, the embodiment of the present invention uses the semantic association between the visible class and the invisible class to establish the visual relationship feature center (feature prototype) of the invisible class.

[0150] Step 1223, input the visual relationship representation to the category selection layer of the feature reconstruction layer to obtain the category vector output by the category selection layer, wherein the category vector is used to represent the dependency scores of different visual relationship categories on the memory feature.

[0151] The category selection layer is used to determine the category vector based on the visual relationship representation. During the training process, the category selection layer learns the mapping relationship between the visual relationship representation and the category vector, thereby mapping the visual relationship representation to the category vector based on the category selection layer.

[0152] In one embodiment, the category selection layer may include a fully connected layer and an activation function, and the activation function may be tanh.

[0153] For example, the category vector e ij Through a single-layer fully connected network T sel Computing visual relationship representation q ij The category vector e ij , and its calculation formula is:

[0154] e ij =tanh(T sel (q ij )).

[0155] Step 1224: input the visual relationship representation, the memory feature, and the category vector into the feature calculation layer of the feature reconstruction layer to obtain the reconstructed features output by the feature calculation layer.

[0156] Among them, the feature calculation layer is used to aggregate visual relationship representation, memory features and category vectors into reconstructed features.

[0157] In one embodiment, the calculation formula of the feature calculation layer is:

[0158]

[0159] in, Represents the visual relationship q ij The reconstructed features after reconstruction, e ij is the category vector, m ij For memory characteristics.

[0160] Of course, the calculation formula of the feature calculation layer may also be in other forms, which are not specifically limited here.

[0161] According to the visual relationship detection method of the embodiment of the present invention, the associative vector corresponding to the visual relationship representation is determined through the associative vector mapping layer of the feature reconstruction layer, and the memory feature corresponding to the visual relationship representation is determined through the memory feature extraction layer of the feature reconstruction layer, and the category vector corresponding to the visual relationship representation is determined through the category selection layer of the feature reconstruction layer, and then the visual relationship representation is feature reconstructed based on the memory feature and the category vector to obtain the reconstructed feature corresponding to the visual relationship representation, so as to realize the zero-sample learning task through the memory mechanism, thereby realizing accurate detection of images containing invisible classes, and finally realizing zero-sample visual relationship detection.

[0162] Further, based on any of the above embodiments, another embodiment of the visual relationship detection method of the present invention is proposed. In this embodiment, the above step 123 includes the following steps 1231-1233:

[0163] Step 1231: input the reconstructed features into the semantic space mapping layer of the visual relationship prediction layer to obtain the semantic mapping features output by the semantic space mapping layer.

[0164] Among them, the semantic space mapping layer is used to map the reconstructed features into the semantic space to obtain semantic mapping features.

[0165] The semantic space mapping layer can be any neural network architecture, such as CNN, RNN, etc., which is not specifically limited here.

[0166] In one embodiment, the semantic space mapping layer may be an MLP. For example, to identify the i-th object o i and the jth object o j The visual relationship between them is reconstructed using an MLP with learnable parameters W. Map into semantic space to obtain semantic mapping features

[0167] Step 1232, input the semantic mapping feature into the semantic space layer of the visual relationship prediction layer to obtain the visual relationship category probability output by the semantic space layer, and the semantic space layer is used to perform distance calculation based on the semantic mapping feature and the visual relationship label in the same semantic space to obtain the visual relationship category probability.

[0168] Specifically, the probability that the visual relationship between two objects belongs to a certain class of visual relationship is measured by the pre-distance between the mapped semantic mapping features and the representation of the class in the semantic space.

[0169] In one embodiment, the calculation formula of the visual relationship category probability is as follows:

[0170]

[0171] Among them, P(r k |o i ,o j ) are two objects o i , o j The visual relationship between them belongs to the kth visual relationship r k The probability of; sigmoid is the activation function, of course, other activation functions can also be selected; is the semantic mapping feature; y k is the representation of this type of visual relationship in the semantic space.

[0172] Step 1233: input the visual relationship category probability to the output layer of the visual relationship prediction layer to obtain the visual relationship detection result output by the output layer.

[0173] Among them, the output layer is used to discriminate the visual relationship detection results based on the visual relationship category probability.

[0174] In one embodiment, the visual relationship with the highest probability may be used as the visual relationship detection result.

[0175] According to the visual relationship detection method of an embodiment of the present invention, the reconstructed features are mapped to the semantic space layer through the semantic space mapping layer of the visual relationship detection model, and then in the same semantic space, the distance between the semantic mapping features and the visual relationship labels is calculated to obtain the visual relationship category probability, and then the visual relationship detection result is obtained based on the visual relationship category probability, thereby achieving accurate detection of images containing invisible classes and ultimately achieving zero-sample visual relationship detection.

[0176] The visual relationship detection device provided by the present invention is described below. The visual relationship detection device described below and the visual relationship detection method described above can be referenced to each other.

[0177] In this embodiment, the visual relationship detection device includes the following determination module and detection module:

[0178] A determination module, used for determining an image to be detected;

[0179] A detection module, used for inputting the image to be detected into a visual relationship detection model to obtain a visual relationship detection result output by the visual relationship detection model;

[0180] The visual relationship detection model is trained based on visible class sample images and their labels, as well as invisible class labels.

[0181] Figure 3 A schematic diagram of the structure of an electronic device provided by the present invention, such as Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the visual relationship detection method, which includes: determining an image to be detected; inputting the image to be detected into a visual relationship detection model to obtain a visual relationship detection result output by the visual relationship detection model; wherein the visual relationship detection model is trained based on visible class sample images and their labels, as well as invisible class labels.

[0182] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0183] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the visual relationship detection method provided by the above methods, which method includes: determining an image to be detected; inputting the image to be detected into a visual relationship detection model to obtain a visual relationship detection result output by the visual relationship detection model; wherein the visual relationship detection model is trained based on visible class sample images and their labels, as well as invisible class labels.

[0184] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the visual relationship detection method provided by the above-mentioned methods, the method comprising: determining an image to be detected; inputting the image to be detected into a visual relationship detection model, and obtaining a visual relationship detection result output by the visual relationship detection model; wherein the visual relationship detection model is trained based on visible class sample images and their labels, as well as invisible class labels.

[0185] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0186] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0187] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A visual relationship detection method, characterized in that: include: Determine an image to be detected; Inputting the image to be detected into a visual relationship detection model to obtain a visual relationship detection result output by the visual relationship detection model; The visual relationship detection model is obtained by training based on visible class sample images and their labels, as well as invisible class labels; The step of inputting the image to be detected into a visual relationship detection model to obtain a visual relationship detection result output by the visual relationship detection model includes: Inputting the image to be detected into the object feature representation layer of the visual relationship detection model to obtain multiple object feature representations output by the object feature representation layer; Inputting visual relationship representations corresponding to any two object feature representations among the multiple object feature representations into a feature reconstruction layer of the visual relationship detection model to obtain reconstructed features output by the feature reconstruction layer, wherein the feature reconstruction layer is obtained by training based on a first loss function determined based on a visual relationship feature center of an image training set of the visual relationship detection model and reconstructed feature training data corresponding to the image training set; Inputting the reconstructed features into the visual relationship prediction layer of the visual relationship detection model to obtain a visual relationship detection result output by the visual relationship prediction layer, wherein the visual relationship prediction layer is trained based on a second loss function determined based on a visual relationship category probability and a visual relationship label of the visible class sample image, and the visual relationship category probability is obtained by performing distance judgment based on the reconstructed feature training data and the visual relationship label of the visible class sample image; The step of inputting the visual relationship representations corresponding to any two object feature representations among the plurality of object feature representations into the feature reconstruction layer of the visual relationship detection model to obtain the reconstructed features output by the feature reconstruction layer comprises: Inputting the visual relationship representation into the associative vector mapping layer of the feature reconstruction layer to obtain the associative vector output by the associative vector mapping layer; The association vector is input to the memory feature extraction layer of the feature reconstruction layer to obtain the memory feature output by the memory feature extraction layer, wherein the memory feature extraction layer is used to determine the memory feature based on the association vector and the visual relationship feature center of the image training set, the visual relationship feature center of the visible class sample image is obtained by performing cluster center calculation based on the visual relationship label of the visible class sample image, and the visual relationship feature center of the invisible class is generated based on the semantic association between the visible class and the invisible class; Inputting the visual relationship representation into the category selection layer of the feature reconstruction layer to obtain a category vector output by the category selection layer, wherein the category vector is used to represent the dependency scores of different visual relationship categories on the memory feature; The visual relationship representation, the memory feature and the category vector are input into the feature calculation layer of the feature reconstruction layer to obtain the reconstructed features output by the feature calculation layer.

2. The visual relationship detection method according to claim 1, characterized in that: The step of inputting the image to be detected into the object feature representation layer of the visual relationship detection model to obtain a plurality of object feature representations output by the object feature representation layer includes: Inputting the image to be detected into the object detection layer of the object feature representation layer to obtain a plurality of visual features, a plurality of category labels and a plurality of coordinate information output by the object detection layer; The multiple visual features, the semantic features of the multiple category labels and the position features of the multiple coordinate information are input into the feature fusion layer of the object feature representation layer to obtain multiple object feature representations output by the feature fusion layer.

3. The visual relationship detection method according to claim 2, characterized in that: The step of inputting the plurality of visual features, the semantic features of the plurality of category labels, and the position features of the plurality of coordinate information into the feature fusion layer of the object feature representation layer to obtain the plurality of object feature representations output by the feature fusion layer includes: Inputting the multiple visual features into the attention layer of the object feature representation layer, performing feature transformation, and obtaining multiple object attention features output by the attention layer, wherein the attention layer is obtained by training based on the semantic feature training data corresponding to the image training set and a third loss function determined by preset grammatical knowledge, and the preset grammatical knowledge is the hyponym relationship between object categories; The multiple object attention features, the semantic features of the multiple category labels, and the position features of the multiple coordinate information are input into the feature fusion layer to obtain the object feature representation output by the feature fusion layer.

4. The visual relationship detection method according to claim 1, characterized in that: The visual relationship representation is obtained by splicing any two object feature representations from the multiple object feature representations.

5. The visual relationship detection method according to claim 1, characterized in that: The step of inputting the reconstructed features into a visual relationship prediction layer of the visual relationship detection model to obtain a visual relationship detection result output by the visual relationship prediction layer includes: Inputting the reconstructed features into the semantic space mapping layer of the visual relationship prediction layer to obtain the semantic mapping features output by the semantic space mapping layer; Inputting the semantic mapping features into the semantic space layer of the visual relationship prediction layer to obtain the visual relationship category probability output by the semantic space layer, wherein the semantic space layer is used to perform distance calculation based on the semantic mapping features and the visual relationship labels in the same semantic space to obtain the visual relationship category probability; The visual relationship category probability is input into the output layer of the visual relationship prediction layer to obtain the visual relationship detection result output by the output layer.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the visual relationship detection method according to any one of claims 1 to 5 are implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the visual relationship detection method according to any one of claims 1 to 5 are implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the visual relationship detection method according to any one of claims 1 to 5 are implemented.

Images