Fine-grained multi-mode prompt-guided visual relation identification method and device

Abstract

The invention discloses a fine-grained multi-modal prompt-guided visual relation identification method and device, and the method comprises the steps: carrying out the modeling of a plurality of groups of obtained relation weight vectors through multivariate Gaussian distribution, deducing the relation classification loss based on a small number of training samples in combination with the features of an output composite image, and calculating the semantic diversity loss, optimizing the text prompt by minimizing the total training loss; taking a relation average weight vector, respectively calculating similarity with the composite image features of the target object pair, carrying out weighted fusion, and carrying out normalization operation to obtain relation prediction probability distribution; and verifying the relation predicate with the maximum prediction probability, and detecting the abnormal behavior of the large public place in real time by the intelligent monitoring system according to the output relation category. The device comprises a processor and a memory. According to the method, the visual language model is accurately adjusted to the visual relation detection task, and the visual relation is accurately detected, so that the abnormal behavior detection capability of the intelligent monitoring system is improved, and an alarm is timely triggered to inform related personnel.

Description

Technical Field

[0001] The present invention relates to the field of visual relationship detection, and in particular to a fine-grained multimodal prompt-guided visual relationship recognition method and device. Background Art

[0002] The visual relationship detection task aims to identify and classify the relationship between pairs of objects in an image, which is structured as a <subject-predicate-object> relationship triple, such as <girl-riding-horse>. Visual relationship detection can mine the rich semantic information contained in the image and provide a comprehensive scene understanding for the image. This structured representation helps promote the development of various high-level visual language tasks, such as image description. [1][2] , Visual Question Answering [3][4] and visual reasoning [5][6] Etc. Therefore, the visual relationship detection task has always been a research focus in the multimodal field, and has great application value in the fields of intelligent monitoring, autonomous driving, and human-computer interaction.

[0003] In recent years, the field of visual relationship detection has made significant progress and a large number of excellent research results have emerged. For example: a method based on a bidirectional long short-term memory network [7] Mainly uses bidirectional long short-term memory networks to calculate and propagate visual context information to predict the relationship between object pairs; causal reasoning-based methods [8] Constructing causal graphs in traditional biased training, removing bad bias effects by inferring counterfactual causal relationships, and using the total direct effect as the final predicate score; a prototype-based embedding network method [9] The entities and predicates are mainly modeled using representations based on class prototypes, thereby establishing matches between entity pairs and predicates in a common embedding space for relationship identification. However, the above methods rely on supervised training on a large amount of manually annotated data, and it is very difficult to collect annotations for many uncommon relationships, which leads to an uneven distribution of the number of different relationship annotations in the dataset, making the relationships predicted by these methods often biased. Therefore, these methods are not suitable for real-world scenarios. For example: when the abnormal behavior of "people knocking on the glass" appears in the smart monitoring of a shopping mall, there may not be enough annotations for the negative relationship "knocking" in the dataset. In this case, the above methods lack learning of the "knocking" relationship samples, resulting in this behavior being identified as other common visual relationships, such as "people looking at the glass". This recognition bias will cause the system to fail to accurately trigger an alarm, thereby affecting the actual security monitoring effect.

[0004] Recently, large-scale pre-trained visual language models, such as CLIP

[10] , which has shown significant application prospects in image recognition tasks. The visual language model is pre-trained on large-scale image-text pairs and has strong zero-shot visual reasoning capabilities through contrastive learning. Given an image and a text prompt, such as "a photo of [class]", the pre-trained visual language model can identify the category that best matches the image. Currently, the task of adapting the visual language model to region-level visual relationship detection has rarely been explored. The existing intuitive method is to use the joint region of the object pair cropped from the image and similar manually designed text prompts.

[11] , such as "a photo of [relation]". However, the cropped rectangular area may contain other objects in addition to the target object pair, and the visual language model is easily misled to predict the visual relationship between the noise object pairs. For example: a frame of the intelligent monitoring is captured as Figure 1 As shown in (a), a visual language model is used to detect the relationship between "man" and "chair". The joint region of the target object pair is cropped as the visual input. However, the region also contains noise objects such as computers, which will cause the visual language model to be disturbed and the relationship between "man" and "chair" "sitting" is incorrectly predicted as the relationship between "man" and "computer" "looking". Manually designing a suitable text prompt for visual relationship detection usually takes a lot of time and the results may be suboptimal. In addition, the semantics contained in a single text prompt are very limited and cannot fully cover the multiple semantics of a relationship under different visual representations. For example: a screenshot of an intelligent monitoring system such as Figure 1 As shown in (b) and (c), for the action of "holding", "a man holding a racket" and "a woman holding a glass" have completely different visual representations and emphasize different relational semantics. A simple relation description "a photo of holding" contains limited semantic information and cannot guide the visual language model to correctly predict all different visual representations of "holding".

[0005] References:

[0006] [1] T. Wang, W. Chen, Y. Tian, ​​Y. Song, and Z. Mao, “Improving image captioning via predicting structured concepts,” in EMNLP, 2023, pp. 360–370.

[0007] [2]X.Yang,J.Peng,Z.Wang,H.Xu,Q.Ye,C.Li,S.Huang,F.Huang,Z.Li,andY.Zhang,“Transforming visual scene graphs to image captions,”in ACL,2023,pp.12427–12440.

[0008] [3]B.Souza,M.Aasan,H.Pedrini,and A.R.Rivera,“Selfgraphvqa:A self-supervised graph neural network for scene-based question answering,”in ICCV,2023,pp.4642–4647.

[0009] [4]Y.Wang,M.Yasunaga,H.Ren,S.Wada,and J.Leskovec,“VQA-GNN:reasoningwith multimodal knowledge via graph neural networks for visual questionanswering,”in ICCV,2023,pp.21525–21535.

[0010] [5]J.Shi,H.Zhang,and J.Li,“Explainable and explicit visual reasoningover scene graphs,”in CVPR,2019,pp.8376–8384.

[0011] [6]Z.Wang,H.You,L.H.Li,A.Zareian,S.Park,Y.Liang,K.Chang,and S.Chang,“SGEITL:scene graph enhanced image-text learning for visual commonsensereasoning,”in AAAI,2022,pp.5914–5922.

[0012] [7]R.Zellers,M.Yatskar,S.Thomson,and Y.Choi,“Neural motifs:Scenegraph parsing with global context,”in CVPR,2018,pp.5831–5840.

[0013] [8]K.Tang,Y.Niu,J.Huang,J.Shi,and H.Zhang,“Unbiased scene graphgeneration from biased training,”in CVPR,2020,pp.3713–3722.

[0014] [9]C.Zheng,X.Lyu,L.Gao,B.Dai,and J.Song,“Prototype-based embeddingnetwork for scene graph generation,”in CVPR,2023,pp.22783–22792.

[0015]

[10] A.Radford,J.W.Kim,C.Hallacy,A.Ramesh,G.Goh,S.Agarwal,G.Sastry,A.Askell,P.Mishkin,J.Clark,G.Krueger,and I.Sutskever,“Learning transferablevisual models from natural language supervision,”in ICML,2021,pp.8748–8763.

[0016]

[11] L.Li,J.Xiao,G.Chen,J.Shao,Y.Zhuang,and L.Chen,“Zero-shot visualrelation detection via composite visual cues from large language models,”inNeurIPS,2023. Summary of the Invention

[0017] The present invention provides a method and device for visual relationship recognition guided by fine-grained multimodal prompts. The present invention designs fine-grained composite visual prompts by using semantic masks, introduces multiple learnable text prompts, and uses multivariate Gaussian distribution to model multiple weight vectors of each relationship category, thereby guiding the pre-trained visual language model to focus on specific areas and capture the intra-class diversity of visual relationships, accurately adjusting the visual language model to the visual relationship detection task, and realizing accurate detection of visual relationships, thereby improving the detection capability of abnormal behaviors of intelligent monitoring systems, and timely triggering alarms to notify relevant personnel, as described below:

[0018] A fine-grained multimodal cue-guided visual relationship recognition method, the method comprising:

[0019] For the obtained multiple sets of relationship weight vectors, we use multivariate Gaussian distribution to model them, and combine them with the output composite image features. Based on a small number of training samples, we derive the relationship classification loss and calculate the semantic diversity loss, and minimize the total training loss to optimize the text prompts.

[0020] Take the average weight vector of the relationship, calculate the similarity and weighted fusion with the composite image features of the target object, and obtain the relationship prediction probability distribution after normalization operation;

[0021] The relation predicates with the maximum prediction probability are verified, and the intelligent monitoring system detects abnormal behaviors in large public places in real time based on the output relation categories.

[0022] Wherein, the step of outputting the composite image features is:

[0023] The output object position coordinates and the output object category label are used to crop the joint area of ​​the target object pair, and the SAM model and Gaussian blur operation are used to generate a blurred inverse mask to form a fine-grained composite visual cue and input it into the image encoder of the pre-trained visual language model to output composite image features.

[0024] The steps of obtaining the multiple groups of relationship weight vectors are as follows:

[0025] Initialize multiple learnable context vector groups, concatenate them with the word embeddings of the relations as text prompts, input them into the text encoder of the pre-trained visual language model, and output multiple groups of relation weight vectors.

[0026] The joint area of ​​the cropped target object pair is generated by using the SAM model and Gaussian blur operation to generate a blurred reverse mask:

[0027] Crop the image according to the coordinates to get the object pair o i and j The joint area of ​​​​

[0028] x=CROP(I,b i ∪b j )

[0029] Among them, x represents the cropped object o i and j The joint area of ​​​​the CROP (·) represents the cropping operation, I represents the image, and b i ∪b j Represents the coordinates of the minimum rectangular area covering the i-th and j-th objects;

[0030] Generate semantic masks for the i-th and j-th objects:

[0031] M i +M j =SAM(I,b i ,b j )

[0032] Among them, M i +M j Represents the semantic mask containing the i-th and j-th objects, according to the joint region coordinates b of the i-th and j-th objects i ∪b j , the semantic mask M of the cropped output i +M j Object pair o i and j The joint area of ​​​​

[0033] M ij =CROP(M i +M j ,b i ∪b j )

[0034] Among them, M ij Indicates the clipped object pair o i and j The joint area of ​​the semantic mask; use the Gaussian blur operation to blur the part of x except the i-th and j-th object pixels to obtain the blurred inverse mask:

[0035]

[0036] in, Indicates the object to o i and j The fuzzy inverse mask of Represents a Gaussian blur operation.

[0037] The initialization of multiple learnable context vector groups is respectively concatenated with the word embedding of the relation as the text prompt:

[0038] Initialize multiple vector groups randomly with normal distribution as model parameters to provide contextual information for the relation description in the text prompt; for a predefined set of relation categories, use the built-in tokenizer in the pre-trained visual language model to encode each predicate into a symbolic representation as a word embedding;

[0039] According to the position of the word embedding of the relationship in the text prompt: front, middle, and back, the vector group is concatenated with the word embedding of the relationship in a ratio of 1:1:2.

[0040] Among them, the multiple groups of relationship weight vectors are:

[0041]

[0042] in, represents the set of relation weight vectors corresponding to the k-th text prompt, where is the weight vector of the i-th relation category corresponding to the k-th text prompt, C represents the number of predefined relation categories, and f T () represents the text encoder of the pre-trained visual language model CLIP, represents the kth textual hint containing all relation category descriptions, where represents the description of the cth relation category in the kth text prompt, is the context vector group of the kth text prompt, M is the number of context vectors in each text prompt, r c is the word embedding of the c-th relation category, “;” represents the concatenation operation on the dimension, and K represents the number of text prompts.

[0043] The verification of the relation predicate with the maximum prediction probability and the real-time detection of abnormal behavior in large public places by the intelligent monitoring system according to the output relation category are specifically as follows:

[0044] Input the thought chain prompt content into the large language model Llama 2. According to the instructions in the prompt, the large language model Llama 2 judges the rationality of the specified relationship triple based on prior knowledge and whether it conforms to common sense cognition;

[0045] If "No" is returned, the specified object pairs in the test image will be regarded as unrelated object pairs and the relationship between them will be regarded as the background category;

[0046] If “Yes” is returned, the relation predicate with the maximum probability in the output relation prediction probability distribution is identified as the visual relation category between the specified object pair in the test image and output to the intelligent monitoring system.

[0047] In a second aspect, a fine-grained multimodal cue-guided visual relationship recognition device comprises: a processor and a memory, wherein the memory stores program instructions, and the processor calls the program instructions stored in the memory to enable the device to execute any one of the methods described in the first aspect.

[0048] In a third aspect, a computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes any one of the methods described in the first aspect.

[0049] The beneficial effects of the technical solution provided by the present invention are:

[0050] 1. When designing fine-grained composite visual cues, the present invention crops the joint area of ​​the object pair to provide rich contextual information, and uses the pre-trained SAM (Segment Anything) model and Gaussian blur operation to generate a fuzzy reverse mask to segment the target object pair and the background area. The combination of the two helps the visual language model to accurately locate the target object pair, make full use of the surrounding contextual information, and realize accurate recognition of visual relationships;

[0051] 2. The present invention introduces multiple learnable text prompts, uses multivariate Gaussian distribution to model multiple weight vectors of relationships and derives relationship classification loss, uses a small number of samples for training, minimizes the loss function to tune multiple text prompts to include multiple semantic aspects of the relationship, and combines fine-grained composite visual prompts to accurately adjust the pre-trained visual language model to the visual relationship detection task, so that it can accurately identify multiple different visual representations of a relationship, further improving the accuracy of visual relationship recognition;

[0052] 3. The present invention designs a text prompt template based on thought chain and introduces the pre-trained large language model Llama2 to verify the rationality of the predicted relationship between object pairs; by filtering the predictions that do not conform to common sense, this method solves the limitations of the visual language model in predicting background categories when processing unrelated object pairs, and effectively avoids false alarms in the intelligent monitoring system;

[0053] 4. In the present invention, the image encoder and text encoder of the pre-trained visual language model are always kept frozen. The method only needs to use a small number of training samples for each relationship to tune the context vectors in multiple text prompts, avoiding full supervision training on large-scale human-annotated relationship data and biased predictions caused by the long-tail distribution of relationship annotations in the data set; at the same time, the method can effectively detect untrained and novel relationship categories by utilizing prior knowledge from the visual language model, so that the intelligent monitoring system can cope with more diverse and complex scenarios, enhancing its adaptability;

[0054] 5. Applying the generated results of the present invention to the intelligent monitoring system can help the system to more accurately identify the visual relationship between objects (people) in real time in large public places, such as shopping malls, parking lots and banks, improve the ability to respond to abnormal behaviors, trigger alarms in time, notify relevant personnel, and effectively protect the safety of life and property. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Schematic diagram of visual relationship detection for different object pairs;

[0056] Figure 2 Flowchart of a fine-grained multimodal cue-guided visual relationship recognition method. DETAILED DESCRIPTION

[0057] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0058] Example 1

[0059] In order to solve the problem that the existing intuitive methods cannot accurately locate and identify the visual relationship between target object pairs when adjusting the visual language model for visual relationship detection, the visual relationship recognition accuracy of the prompt-guided visual language model is improved. The commonly used indicator is the recall rate, see Figure 2 , an embodiment of the present invention provides a fine-grained multimodal prompt-guided visual relationship recognition method, the method comprising the following steps:

[0060] 101: For the visual relationship detection database used, use the target detection framework to obtain the position coordinates, visual features and category prediction probability distribution of objects in the image;

[0061] The database used in the embodiment of the present invention needs to mark the location and category of objects and the relationship category between objects in the images. The database used is the Visual Genome database, but it is not limited to this database. Other databases that meet the labeling conditions can be used.

[0062] Existing target detection frameworks include Faster-RCNN, Mask-RCNN, YOLO, etc. The embodiment of the present invention uses the Faster-RCNN (fast regional convolutional neural network) framework to extract information about objects in an image. The feature extraction network in the framework can be a network structure such as GoogLeNet, ResNet-101, VGG-16, etc. The embodiment of the present invention uses the VGG-16 convolutional network. The image in the database is input into the above target detection framework, and the output features include: the position coordinates of the object in the image, that is, the coordinates of the minimum rectangular area covering the object B = {b1, b2, ..., bN}, where b i represents the concatenation of the coordinates of the upper left corner and the lower right corner of the smallest rectangular area covering the i-th object in the image, N is the number of objects in the image; the visual features of the objects in the image, that is, the features F = {f1,f2,...,f N}, where f i is the visual feature of the i-th object in the image; the category prediction probability distribution of the object in the image, that is, the feature L={l1,l2,...,l N}, where l i Predict the probability distribution vector for the category of the i-th object in the image.

[0063] 102: For the visual features and category prediction probability distribution of the object output in step 101, they are jointly input into the encoder to obtain context information of the object, and then the context information is input into the decoder to output the predicted object category label;

[0064] The object category prediction probability distribution vector outputted in step 101 is mapped to the same dimension as the visual feature, and then concatenated with the visual feature in dimension, and then input into a bidirectional long short-term memory network (BiLSTM) encoder to obtain object context information, and then the context information is input into a long short-term memory network (LSTM) decoder to output the predicted object category:

[0065]

[0066] Among them, W l represents the learnable feature mapping matrix, BiLSTM represents the encoder bidirectional long short-term memory network, C o ={c1,c2,...,c N} represents the context information of the object, where c i is the context information of the i-th object in the image, LSTM represents the decoder long short-term memory network, O = {o1, o2, ..., o N} represents the predicted object category label, where o i is the category of the i-th object in the predicted image, Input represents the input process, and Output represents the output process.

[0067] 103: using the position coordinates of the object outputted in step 101 and the object category label outputted in step 102, cropping the joint region of the target object pair, using the SAM model and Gaussian blur operation to generate a fuzzy reverse mask to form a fine-grained composite visual cue, inputting it into the image encoder of the pre-trained visual language model, and outputting a composite image feature;

[0068] For the pre-trained visual language model used, which includes an image encoder and a text encoder, the embodiment of the present invention uses a CLIP (Contrastive Language-Image Pre-training) model, but is not limited to this visual language model, and may also be other similar visual language models.

[0069] Detect the i-th object o in the image output by step 102 i and the jth object o j First, we construct fine-grained composite visual cues, including object pairs. i and j The joint region of , and the fuzzy inverse mask.

[0070] Specifically, according to the i-th object o output in step 101 i and the jth object o j The position coordinates b i and b j , extract the minimum value of the horizontal and vertical coordinates of the upper left corner, and the maximum value of the horizontal and vertical coordinates of the lower right corner, and combine them to obtain the joint area coordinates b of the i-th and j-th objects i ∪b j , according to this coordinate, crop the image to get the object pair o i and j The joint area of ​​​​

[0071] x=CROP(I,b i ∪b j )(2)

[0072] Among them, x represents the cropped object o i and j The joint area of ​​​​the CROP (·) represents the cropping operation, I represents the image, and b i ∪b j Represents the coordinates of the minimum rectangular area covering the i-th and j-th objects.

[0073] Then, input the image and the position coordinates b of the i-th and j-th objects i 、b j To the pre-trained SAM model, generate semantic masks for the i-th and j-th objects:

[0074] M i +M j =SAM(I,b i ,b j )(3)

[0075] Among them, M i Represents the semantic mask of the i-th object. Its width and height are consistent with the image. In the mask, the value of the pixel position corresponding to the i-th object in the image is 1, and the values ​​of the other positions are 0.j represents the semantic mask of the jth object, M i +M j Represents a semantic mask containing the i-th and j-th objects. Its width and height are consistent with the image. The values ​​of the pixel positions corresponding to the i-th and j-th objects in the mask are 1, and the values ​​of the remaining positions are 0. SAM(·) represents the Segment Anything Model.

[0076] Next, according to the joint region coordinates b of the i-th and j-th objects i ∪b j , the semantic mask M output by cropping formula (3) i +M j Object pair o i and j The joint area of ​​​​

[0077] M ij =CROP(M i +M j ,b i ∪b j )(4)

[0078] Among them, M ij Indicates the clipped object pair o i and j Joint regions of semantic masks.

[0079] The semantic mask M output by formula (4) ij The object output from formula (2) is o i and j The joint area x corresponds to M ij The values ​​of the pixels in x that are the same as the i-th and j-th object pixels are 1, and the values ​​of the other positions are 0. ij The value in is used to blur the part of x except the i-th and j-th object pixels using Gaussian blur operation, thus obtaining the blurred reverse mask:

[0080]

[0081] in, Indicates the object to o i and j The fuzzy inverse mask of Represents a Gaussian blur operation.

[0082] Finally, the object pair o output by formula (2) i and j The joint region x of the fuzzy inverse mask output by formula (5) Input to the image encoder of the pre-trained visual language model CLIP to obtain composite image features:

[0083]

[0084] Where z represents the object pair o output by formula (2) i and j The image features of the joint region x, The fuzzy reverse mask output of formula (5) is represented by The image features, f I (·) represents the image encoder of the pre-trained visual-language model CLIP.

[0085] 104: Initialize multiple learnable context vector groups, concatenate them with the word embedding of the relationship as text prompts, input them into the text encoder of the pre-trained visual language model, and output multiple groups of relationship weight vectors;

[0086] First, multiple vector groups are randomly initialized with normal distribution as model parameters to provide contextual information for the relationship description in the text prompt. For the predefined relationship category set, the built-in tokenizer in the pre-trained visual language model CLIP is used to encode each predicate into a symbolic representation as a word embedding, which is easy for the model to understand and process.

[0087] Then, according to the position where the word embedding of the relationship appears in the text prompt: front, middle, and back, the vector group is dimensionally spliced ​​with the word embedding of the relationship in a ratio of 1:1:2, so that the multiple text prompts have position diversity, so as to more fully learn the semantics of the relationship.

[0088] Finally, multiple text prompts are input into the text encoder of the pre-trained visual language model to obtain multiple sets of relationship weight vectors:

[0089]

[0090] in, represents the set of relation weight vectors corresponding to the k-th text prompt, where is the weight vector of the i-th relation category corresponding to the k-th text prompt, C represents the number of predefined relation categories, and f T (·) represents the text encoder of the pre-trained visual language model CLIP, represents the kth textual hint containing all relation category descriptions, where represents the description of the cth relation category in the kth text prompt, where is the context vector group of the kth text prompt, M is the number of context vectors in each text prompt, r cis the word embedding of the c-th relation category, “;” represents the concatenation operation on the dimension, and K represents the number of text prompts.

[0091] 105: for the relation weight vector outputted from step 104, a multivariate Gaussian distribution is used to model it, and combined with the composite image features outputted from step 103, a relation classification loss is derived based on a small number of training samples and a semantic diversity loss is calculated, and the total training loss is minimized to optimize the text prompt;

[0092] The small amount of training samples used in the embodiment of the present invention is to extract the first 16 representative object pairs of each relationship category from the training set of the database used. Specifically, all object pairs of each relationship category are extracted from the training set of the database used, and according to the position information of the object pairs in the image, the SAM model and Gaussian blur operation are used to segment the object pairs and the background to generate a fuzzy reverse mask, which is input into the image encoder of the pre-trained visual language model CLIP to obtain image features, and the default text prompt template "a photo of [relation]" is used as a text prompt, which is input into the text encoder of the pre-trained visual language model CLIP to obtain the weight vectors of all relations, and the dot product similarity between the image features of the object pairs and each relationship weight vector is calculated. After normalization operation, the relationship prediction probability distribution is obtained, and the relationship category with the maximum probability is taken as the prediction result. The object pairs whose predicted categories are consistent with the actual relationship categories are screened out, and these object pairs are sorted from large to small according to the predicted probability, and the first 16 object pairs are selected as training samples of the relationship category.

[0093] The set of relation weight vectors for the kth text prompt output by formula (7) includes weight vectors of all relation categories corresponding to the kth text prompt. There are K text prompts in total, so each relation category has K weight vectors.

[0094] Multiple weight vectors of a relation category are inherently consistent and represent multiple semantic aspects of the same relation. In the output embedding space, they tend to be closely clustered together. Multivariate Gaussian distribution is suitable for fitting the distribution of multiple weight vectors of a relation category.

[0095] In the process of multiple text prompts tuning, it is assumed that the K weight vectors for each relation category obey a multivariate Gaussian distribution, whose parameters are estimated by calculating the mean and covariance of the K weight vectors for each relation category:

[0096]

[0097] Among them, μ c represents the mean of the K weight vectors of the cth relationship category, represents the weight vector of the cth relation category corresponding to the kth text prompt, Σ c Represents the covariance matrix of the K weight vectors of the c-th relationship category.

[0098] In the small number of object pairs used for training, each object pair contains the image features of its joint region and fuzzy reverse mask, as well as the basic fact relationship category between them. In order to facilitate the derivation and simplification of the relationship classification loss, the image features of the joint region of the object pair and its basic fact relationship category, as well as the relationship weight vector set of K text prompts, are used to calculate the negative likelihood logarithm of the average probability of all training samples being predicted as the basic fact relationship category, and then the average relationship classification loss for each object pair is obtained:

[0099]

[0100] in, represents the relation classification loss, T represents the number of training samples in a batch, and p(y i |x i ,W k ) represents the probability that the set of relation weight vectors of the k-th text prompt will be used to predict the i-th object pair for training as its basic fact relation category, where x i is the joint area of ​​the cropped i-th object pair, y i is the index of the basic fact relationship category between the i-th object pair, z i represents the image features of the joint area of ​​the ith object pair, Indicates the yth corresponding to the kth text prompt i The weight vector of the relationship categories, τ represents the temperature parameter.

[0101] Based on the following formula:

[0102]

[0103] in, represents the averaging operation, and X represents the variable.

[0104] The relation classification loss can be further derived as:

[0105]

[0106] Among them, the variable Obey Gaussian distribution in Indicates the yth i The mean of K weight vectors of relationship categories, Indicates the yth i The covariance matrix of K weight vectors of relationship categories, Represents the K weight vectors of the cth relationship category and the yth i The covariance matrix of K weight vectors of relationship categories, is the covariance matrix The transpose of .

[0107] Therefore, based on the moment generating function of the following normal distribution:

[0108]

[0109] in, Indicates the average operation, X indicates that the mean is μ and the variance is σ 2 A normally distributed random variable, t is a variable.

[0110] Formula (11) can be further simplified as:

[0111]

[0112] Using the upper limit of formula (13) and jointly using the composite image features output from step 103 to replace the image features of the joint region of the object pair, the final relation classification loss is calculated as:

[0113]

[0114] Among them, α, β represent hyper parameters, Represents the image features of the blurred inverse mask of the i-th object pair.

[0115] By continuously reducing the relation classification loss and optimizing the context vector groups in multiple text prompts, the true distribution of relation weights is estimated. In order to make the context vector groups in multiple text prompts fully capture the various semantic aspects of the relationship, they are input separately into the text encoder of the pre-trained visual language model CLIP to obtain the output embedding of each context vector group, and the average absolute dot product similarity of the output embeddings of any two context vector groups is calculated as the semantic diversity loss:

[0116]

[0117] in, represents the semantic diversity loss, V i represents the context vector group in the i-th text prompt, which is Abbreviation of V j represents the context vector group in the jth text prompt, Abbreviation of .

[0118] By continuously reducing the semantic diversity loss, the distance between the context vector groups of different text prompts is increased, thereby improving the semantic diversity of text prompts. Therefore, the total training loss is:

[0119]

[0120] in, Denotes the total training loss, which is minimized to fine-tune the text prompts for subsequent relational reasoning.

[0121] 106: Take the average weight vector of the relationship, calculate the similarity with the composite image features of the target object pair, and then perform weighted fusion and obtain the relationship prediction probability distribution after normalization operation;

[0122] The multiple text prompts corresponding to the minimum total training loss in step 105 are input into the text encoder of the pre-trained visual language model CLIP to obtain multiple groups of relationship weight vectors output by formula (7). For the K weight vectors of each relationship category, their mean is taken as the final relationship weight vector for visual relationship reasoning.

[0123] In the inference phase, the pre-trained visual language model CLIP is used to detect the visual relationship between all pairs of objects in the database test set images used. For the j-th object pair in a test image, the composite image features of the j-th object pair are obtained through step 103, including the image features of its joint area and the image features of the fuzzy inverse mask.

[0124] The similarity is calculated using the mean of multiple weight vectors of each relationship category and the composite image features of the j-th object pair, and then weighted fusion is performed. After normalization, the predicted probability distribution of the relationship between the j-th object pair is obtained:

[0125]

[0126] Among them, p(c) represents the probability that the relationship between the jth object pair is predicted as the cth relationship category, z j represents the image features of the joint region of the jth object pair, Represents the image features of the blurred inverse mask of the jth object pair.

[0127] 107: Using the thought chain prompt and the large language model Llama 2, the rationality of the relation predicate with the maximum prediction probability in the relation prediction probability distribution output in step 106 is verified. According to the output relation category, the intelligent monitoring system detects abnormal behaviors in large public places in real time, thereby improving security prevention capabilities and response speed.

[0128] In specific implementation, the embodiment of the present invention uses the Llama 2 model as the large language model, but is not limited to this large language model, and may also use other large language models that can interact.

[0129] For the predicted probability distribution of the relationship between the j-th object pair output in step 106, take the relationship predicate with the largest predicted probability and the category labels of the subject and object objects in the j-th object pair output in step 102, and fill in the corresponding positions in the following thought chain prompt template: {subject}, {predicate}, {object}:

[0130] System:Your task is to help user determine whether the relationbetween objects is reasonable.After thinking about it,only answer a word'Yes'or'No',don't give any reason.

[0131] User:Can the{subject}be{predicate}{object}?

[0132] Assistant: Let's think step by step!

[0133] Among them, System, User, and Assistant represent three different roles: system, user, and assistant respectively.

[0134] Then, the above thought chain prompt content is input into the large language model Llama 2. According to the instructions in the prompt, the large language model Llama 2 judges the rationality of the specified relationship triple based on prior knowledge and whether it conforms to common sense cognition. After the judgment is completed, the large language model Llama 2 returns a signal "Yes" or "No".

[0135] If "No" is returned, it means that the relationship predicate with the highest predicted probability occurs between the j-th object pair, which is not in line with common sense. The j-th object pair in the test image will be regarded as an unrelated object pair and the relationship between them will be regarded as a background category, and will not be output to the intelligent monitoring system.

[0136] If "Yes" is returned, it means that the relation predicate with the largest predicted probability is reasonable and may occur between the jth object pair. Therefore, the relation predicate with the largest probability in the relation prediction probability distribution output by formula (17) is identified as the visual relationship category between the jth object pair in the test image and output to the intelligent monitoring system.

[0137] In summary, the embodiments of the present invention avoid the problem that existing intuitive methods cannot accurately locate and identify the visual relationship between target object pairs when adjusting the visual language model for visual relationship detection, improve the visual relationship recognition accuracy of the prompt-guided visual language model, that is, the recall rate, and can be applied to any visual relationship detection database.

[0138] Example 2

[0139] A fine-grained multimodal prompt-guided visual relationship recognition device, the device comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to enable the device to execute the following method steps in embodiment 1:

[0140] For the obtained multiple sets of relationship weight vectors, we use multivariate Gaussian distribution to model them, and combine them with the output composite image features. Based on a small number of training samples, we derive the relationship classification loss and calculate the semantic diversity loss, and minimize the total training loss to optimize the text prompts.

[0141] Take the average weight vector of the relationship, calculate the similarity and weighted fusion with the composite image features of the target object, and obtain the relationship prediction probability distribution after normalization operation;

[0142] The relation predicates with the maximum prediction probability are verified, and the intelligent monitoring system detects abnormal behaviors in large public places in real time based on the output relation categories.

[0143] Among them, the steps of outputting composite image features are:

[0144] The output object position coordinates and the output object category label are used to crop the joint area of ​​the target object pair, and the SAM model and Gaussian blur operation are used to generate a blurred inverse mask to form a fine-grained composite visual cue and input it into the image encoder of the pre-trained visual language model to output composite image features.

[0145] Among them, the steps for obtaining multiple groups of relationship weight vectors are:

[0146] Initialize multiple learnable context vector groups, concatenate them with the word embeddings of the relations as text prompts, input them into the text encoder of the pre-trained visual language model, and output multiple groups of relation weight vectors.

[0147] Among them, the joint area of ​​the target object pair is cropped, and the blurred reverse mask is generated by using the SAM model and Gaussian blur operation:

[0148] Crop the image according to the coordinates to get the object pair o i and j The joint area of ​​​​

[0149] x=CROP(I,bi ∪b j )

[0150] Among them, x represents the cropped object o i and j The joint area of ​​​​the CROP (·) represents the cropping operation, I represents the image, and b i ∪b j Represents the coordinates of the minimum rectangular area covering the i-th and j-th objects;

[0151] Generate semantic masks for the i-th and j-th objects:

[0152] M i +M j =SAM(I,b i ,b j )

[0153] Among them, M i +M j Represents the semantic mask containing the i-th and j-th objects, according to the joint region coordinates b of the i-th and j-th objects i ∪b j , the semantic mask M of the cropped output i +M j Object pair o i and j The joint area of ​​​​

[0154] M ij =CROP(M i +M j ,b i ∪b j )

[0155] Among them, M ij Indicates the clipped object pair o i and j The joint area of ​​the semantic mask; use the Gaussian blur operation to blur the part of x except the i-th and j-th object pixels to obtain the blurred inverse mask:

[0156]

[0157] in, Indicates the object to o i and j The fuzzy inverse mask of Represents a Gaussian blur operation.

[0158] Among them, multiple learnable context vector groups are initialized and concatenated with the word embedding of the relationship as text prompts:

[0159] Initialize multiple vector groups randomly with normal distribution as model parameters to provide contextual information for the relation description in the text prompt; for a predefined set of relation categories, use the built-in tokenizer in the pre-trained visual language model to encode each predicate into a symbolic representation as a word embedding;

[0160] According to the position of the word embedding of the relationship in the text prompt: front, middle, and back, the vector group is concatenated with the word embedding of the relationship in a ratio of 1:1:2.

[0161] Among them, the weight vectors of multiple groups of relationships are:

[0162]

[0163] in, represents the set of relation weight vectors corresponding to the k-th text prompt, where is the weight vector of the i-th relation category corresponding to the k-th text prompt, C represents the number of predefined relation categories, and f T () represents the text encoder of the pre-trained visual language model CLIP, represents the kth textual hint containing all relation category descriptions, where represents the description of the cth relation category in the kth text prompt, is the context vector group of the kth text prompt, M is the number of context vectors in each text prompt, r c is the word embedding of the c-th relation category, “;” represents the concatenation operation on the dimension, and K represents the number of text prompts.

[0164] Among them, the relation predicate with the maximum prediction probability is verified, and the intelligent monitoring system detects abnormal behaviors in large public places in real time according to the output relation category:

[0165] Input the thought chain prompt content into the large language model Llama 2. According to the instructions in the prompt, the large language model Llama 2 judges the rationality of the specified relationship triple based on prior knowledge and whether it conforms to common sense cognition;

[0166] If "No" is returned, the specified object pairs in the test image will be regarded as unrelated object pairs and the relationship between them will be regarded as the background category;

[0167] If “Yes” is returned, the relation predicate with the maximum probability in the output relation prediction probability distribution is identified as the visual relation category between the specified object pair in the test image and output to the intelligent monitoring system.

[0168] It should be pointed out here that the device description in the above embodiment corresponds to the method description in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0169] The execution subjects of the above-mentioned processor and memory can be devices with computing functions such as computers, single-chip microcomputers, and microcontrollers. In specific implementation, the embodiments of the present invention do not limit the execution subjects and are selected according to the needs of actual applications.

[0170] The data signal is transmitted between the memory and the processor via a bus, which is not described in detail in the embodiment of the present invention.

[0171] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium, the storage medium includes a stored program, and when the program is running, the device where the storage medium is located is controlled to execute the method steps in the above embodiment.

[0172] The computer-readable storage medium includes but is not limited to a flash memory, a hard disk, a solid-state drive, and the like.

[0173] It should be pointed out here that the description of the readable storage medium in the above embodiment corresponds to the description of the method in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0174] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present invention are generated.

[0175] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions may be stored in a computer-readable storage medium or transmitted via a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be accessed by the computer or a data storage device such as a server or a data center that includes one or more available media. The available medium may be a magnetic medium or a semiconductor medium, etc.

[0176] Unless otherwise specified, the models of the components in the embodiments of the present invention are not limited, and any device that can perform the above functions may be used.

[0177] Those skilled in the art will appreciate that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0178] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A fine-grained multimodal cue-guided visual relationship recognition method, characterized in that: The method comprises: For the obtained multiple sets of relationship weight vectors, we use multivariate Gaussian distribution to model them, and combine them with the output composite image features. Based on a small number of training samples, we derive the relationship classification loss and calculate the semantic diversity loss, and minimize the total training loss to optimize the text prompts. Take the average weight vector of the relationship, calculate the similarity and weighted fusion with the composite image features of the target object, and obtain the relationship prediction probability distribution after normalization operation; The relation predicates with the maximum prediction probability are verified, and the intelligent monitoring system detects abnormal behaviors in large public places in real time based on the output relation categories.

2. According to claim 1, a fine-grained multimodal cue-guided visual relationship recognition method is characterized in that: The step of outputting the composite image features is: The output object position coordinates and the output object category label are used to crop the joint area of ​​the target object pair, and the SAM model and Gaussian blur operation are used to generate a blurred inverse mask to form a fine-grained composite visual cue and input it into the image encoder of the pre-trained visual language model to output composite image features.

3. The method for visual relationship recognition guided by fine-grained multimodal cues according to claim 1, characterized in that: The steps of obtaining the multiple groups of relationship weight vectors are: Initialize multiple learnable context vector groups, concatenate them with the word embeddings of the relations as text prompts, input them into the text encoder of the pre-trained visual language model, and output multiple groups of relation weight vectors.

4. The fine-grained multimodal cue-guided visual relationship recognition method according to claim 2, characterized in that: The joint area of ​​the cropped target object pair is generated by using the SAM model and Gaussian blur operation to generate a blurred reverse mask: Crop the image according to the coordinates to get the object pair o i and j The joint area of ​​​​ x=CROP(I,b i ∪b j ) Among them, x represents the cropped object o i and j The joint area of ​​​​the CROP (·) represents the cropping operation, I represents the image, and b i ∪b j Represents the coordinates of the minimum rectangular area covering the i-th and j-th objects; Generate semantic masks for the i-th and j-th objects: M i +M j =SAM(I,b i ,b j ) Among them, M i +M j Represents the semantic mask containing the i-th and j-th objects, according to the joint region coordinates b of the i-th and j-th objects i ∪b j , the semantic mask M of the cropped output i +M j Object pair o i and j The joint area of ​​​​ M ij =CROP(M i +M j ,b i ∪b j ) Among them, M ij Indicates the clipped object pair o i and j The joint area of ​​the semantic mask; use the Gaussian blur operation to blur the part of x except the i-th and j-th object pixels to obtain the blurred inverse mask: in, Indicates the object to o i and j The fuzzy inverse mask of Represents a Gaussian blur operation.

5. The fine-grained multimodal cue-guided visual relationship recognition method according to claim 3, characterized in that: The initialization of multiple learnable context vector groups is respectively concatenated with the word embedding of the relation as text prompts: Initialize multiple vector groups randomly with normal distribution as model parameters to provide contextual information for the relation description in the text prompt; for a predefined set of relation categories, use the built-in tokenizer in the pre-trained visual language model to encode each predicate into a symbolic representation as a word embedding; According to the position of the word embedding of the relationship in the text prompt: front, middle, and back, the vector group is concatenated with the word embedding of the relationship in a ratio of 1:1:

2.

6. The fine-grained multimodal cue-guided visual relationship recognition method according to claim 3, characterized in that: The multiple groups of relationship weight vectors are: in, represents the set of relation weight vectors corresponding to the k-th text prompt, where is the weight vector of the i-th relation category corresponding to the k-th text prompt, C represents the number of predefined relation categories, and f T (·) represents the text encoder of the pre-trained visual language model CLIP, represents the kth textual hint containing all relation category descriptions, where represents the description of the cth relation category in the kth text prompt, is the context vector group of the kth text prompt, M is the number of context vectors in each text prompt, r c is the word embedding of the c-th relation category, ";" represents the concatenation operation on the dimension, and K represents the number of text prompts.

7. The fine-grained multimodal cue-guided visual relationship recognition method according to claim 1, characterized in that: The verification of the relation predicate with the maximum prediction probability and the intelligent monitoring system detecting abnormal behaviors in large public places in real time according to the output relation category are specifically as follows: Input the thought chain prompt content into the large language model Llama 2. According to the instructions in the prompt, the large language model Llama 2 judges the rationality of the specified relationship triple based on prior knowledge and whether it conforms to common sense cognition; If "No" is returned, the specified object pairs in the test image will be regarded as unrelated object pairs and the relationship between them will be regarded as the background category; If "Yes" is returned, the relation predicate with the maximum probability in the output relation prediction probability distribution is identified as the visual relation category between the specified object pair in the test image and output to the intelligent monitoring system.

8. A fine-grained multimodal cue-guided visual relationship recognition device, characterized in that: The device comprises: a processor and a memory, wherein program instructions are stored in the memory, and the processor calls the program instructions stored in the memory to enable the device to execute any one of the methods of claims 1-7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 7.

Images