A fine-grained multi-modal prompt guided visual relationship recognition method and device

Abstract

The application discloses a kind of fine granularity multimodal prompt guide visual relationship identification method and device, method includes: to the multiple groups of relationship weight vectors obtained, using multivariate Gaussian distribution to model it, and combining the output composite image feature, based on a small amount of training samples deduce relationship classification loss and calculate semantic diversity loss, minimize training total loss to optimize text prompt;Take relationship average weight vector, with the composite image feature of target object pair respectively calculate similarity, weighted fusion, obtain relationship prediction probability distribution after normalization operation;The relationship predicate of maximum prediction probability is verified, and the output relationship category is intelligently monitored system Real-time detection of abnormal behavior in large public places.The device includes: processor and memory.The application accurately adjusts visual language model to visual relationship detection task, realizes the accurate detection of visual relationship, to improve the detection capability of abnormal behavior of intelligent monitoring system, timely triggers alarm and informs relevant personnel.

Description

Technical Field

[0001] This invention relates to the field of visual relationship detection, and more particularly to a method and apparatus for visual relationship recognition guided by fine-grained multimodal prompts. Background Technology

[0002] The visual relation detection task aims to identify and classify relationships between object pairs in an image, which are structured as <subject-verb-object> relation triples, such as <girl-riding-horse>. Visual relation detection can uncover rich semantic information contained in images and provide a comprehensive scene understanding. This structured representation helps promote the development of various high-level visual language tasks, such as image description. [1][2] Visual Q&A [3][4] and visual reasoning [5][6] Therefore, visual relationship detection has always been a research focus in the multimodal field, and it has great application value in areas such as intelligent monitoring, autonomous driving, and human-computer interaction.

[0003] Significant progress has been made in the field of visual relationship detection in recent years, with a large number of excellent research results emerging. For example, methods based on bidirectional long short-term memory networks... [7] The primary method utilizes bidirectional long short-term memory networks to compute and propagate visual contextual information to predict relationships between object pairs; it also employs causal reasoning-based approaches. [8] In traditional biased training, a causal graph is constructed, and the influence of bad bias is removed by deriving counterfactual causal relationships. The total direct effect is used as the final predicate score. A method based on class-prototype embedding networks is also proposed. [9] The primary approach uses prototype-based representations to model entities and predicates, establishing matching between entity pairs and predicates in a common embedding space for relation recognition. However, these methods rely on supervised training on large amounts of manually annotated data. Collecting annotations for many uncommon relations is extremely difficult, leading to an uneven distribution of annotations for different relations in the dataset. This often results in biased relation predictions. Therefore, these methods are not suitable for real-world scenarios. For example, when an unusual behavior of "a person knocking on glass" occurs in a shopping mall's smart surveillance system, the dataset may lack sufficient annotations for the negative relation "knocking." In such cases, the methods lack learning from "knocking" relation samples, causing this behavior to be identified as other common visual relations, such as "a person looking at glass." This identification bias can prevent the system from accurately triggering alarms, thus affecting the effectiveness of actual security monitoring.

[0004] Recently, large-scale pre-trained visual language models, such as CLIP

[10] Visual language models have already shown significant application potential in image recognition tasks. Pre-trained on large-scale image-text pairs, visual language models possess powerful zero-shot visual reasoning capabilities through contrastive learning. Given an image and a text cue, such as "a photo of [class]", the pre-trained visual language model can identify the category that best matches the image. Currently, there is little exploration into adapting visual language models to region-level visual relationship detection tasks. Existing intuitive methods use joint regions of object pairs cropped from images and similarly hand-designed text cuees.

[11] For example, "a photo of [relation]". However, the cropped rectangular region may contain other objects besides the target object pair, and the visual language model is easily misled into predicting the visual relationship between noisy object pairs. For example: capturing a frame from a smart surveillance camera... Figure 1 As shown in (a), a visual language model is used to detect the relationship between "man" and "chair." The joint region of the target object pair is cropped as visual input. However, the region also contains noisy objects such as computers, which interferes with the visual language model, incorrectly predicting the relationship between "man" and "chair" ("sitting") as the relationship between "man" and "computer" ("looking"). Manually designing a suitable text cue for visual relationship detection is often time-consuming and may yield suboptimal results. Furthermore, a single text cue contains very limited semantics and cannot comprehensively cover the multiple semantics of a relationship under different visual representations. For example, capturing footage from a smart surveillance system... Figure 1 As shown in (b) and (c), for the action of "taking," "a man taking a racket" and "a woman taking a glass" have completely different visual representations, and the semantic relationships they emphasize are also different. A simple relational description, "a photo of holding," contains limited semantic information and cannot guide the visual language model to correctly predict all the different visual representations of "taking."

[0005] References:

[0006] [1] T. Wang, W. Chen, Y. Tian, ​​Y. Song, and Z. Mao, “Improving image captioning via predicting structured concepts,” in EMNLP, 2023, pp. 360–370.

[0007] [2]X.Yang,J.Peng,Z.Wang,H.Xu,Q.Ye,C.Li,S.Huang,F.Huang,Z.Li,andY.Zhang,“Transforming visual scene graphs to image captions,”in ACL,2023,pp.12427–12440.

[0008] [3]B.Souza,M.Aasan,H.Pedrini,and A.R.Rivera,“Selfgraphvqa:A self-supervised graph neural network for scene-based question answering,”in ICCV,2023,pp.4642–4647.

[0009] [4]Y.Wang,M.Yasunaga,H.Ren,S.Wada,and J.Leskovec,“VQA-GNN:reasoningwith multimodal knowledge via graph neural networks for visual questionanswering,”in ICCV,2023,pp.21525–21535.

[0010] [5]J.Shi,H.Zhang,and J.Li,“Explainable and explicit visual reasoningover scene graphs,”in CVPR,2019,pp.8376–8384.

[0011] [6]Z.Wang,H.You,L.H.Li,A.Zareian,S.Park,Y.Liang,K.Chang,and S.Chang,“SGEITL:scene graph enhanced image-text learning for visual commonsensereasoning,”in AAAI,2022,pp.5914–5922.

[0012] [7]R. Zellers, M. Yatskar, S. Thomson, and Y. Choi, “Neural motifs: Scenegraph parsing with global context,” in CVPR, 2018, pp. 5831–5840.

[0013] [8]K. Tang, Y. Niu, J. Huang, J. Shi, and H. Zhang, “Unbiased scene graph generation from biased training,” in CVPR, 2020, pp. 3713–3722.

[0014] [9]C. Zheng, X. Lyu, L. Gao, B. Dai, and J. Song, “Prototype-based embedding network for scene graph generation,” in CVPR, 2023, pp. 22783–22792.

[0015]

[10] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” in ICML, 2021, pp. 8748–8763.

[0016]

[11] L. Li, J. Xiao, G. Chen, J. Shao, Y. Zhuang, and L. Chen, “Zero-shot visual relation detection via composite visual cues from large language models,” in NeurIPS, 2023. Summary of the Invention

[0017] This invention provides a fine-grained, multimodal cue-guided visual relationship recognition method and apparatus. By using semantic masks to design fine-grained composite visual cues, this invention introduces multiple learnable text cues and models multiple weight vectors for each relationship category using a multivariate Gaussian distribution. This guides a pre-trained visual language model to focus on specific regions and capture the intra-class diversity of visual relationships, accurately adjusting the visual language model for the visual relationship detection task. This achieves accurate detection of visual relationships, thereby improving the ability of intelligent monitoring systems to detect abnormal behavior and promptly triggering alarms to notify relevant personnel. See the description below for details:

[0018] A fine-grained, multimodal cue-guided visual relationship recognition method, the method comprising:

[0019] For the obtained multiple sets of relation weight vectors, a multivariate Gaussian distribution is used to model them, and combined with the composite image features of the output, the relation classification loss is derived based on a small number of training samples and the semantic diversity loss is calculated. The total training loss is minimized to optimize the text prompts.

[0020] The average weight vector of the relationship is taken, and the similarity and weighted fusion of the composite image features of the target object pair are calculated respectively. The relationship prediction probability distribution is obtained after normalization.

[0021] The relational predicate with the highest predicted probability is validated, and the intelligent monitoring system detects abnormal behavior in large public places in real time based on the output relational category.

[0022] The step of outputting the composite image features is as follows:

[0023] Using the output object's position coordinates and object category labels, the joint region of the target object pair is cropped. A fuzzy inverse mask is generated using the SAM model and Gaussian blur operation to form a fine-grained composite visual cue, which is then input into the image encoder of a pre-trained visual language model to output composite image features.

[0024] The steps for obtaining the multiple sets of relation weight vectors are as follows:

[0025] Multiple learnable context vector sets are initialized, and each set is concatenated with the word embeddings of the relation as text prompts. These prompts are then input into the text encoder of a pre-trained visual language model, which outputs multiple sets of relation weight vectors.

[0026] The joint region of the cropped target object pair is generated using a SAM model and Gaussian blurring to create a fuzzy inverse mask:

[0027] Obtain the object pair o by cropping the image based on coordinates. i and o j The joint region:

[0028] x = CROP(I,b) i ∪b j )

[0029] Where x represents the object pair o being cut. i and o j The joint region, CROP(·) represents the cropping operation, I represents the image, b i ∪b j Represents the coordinates of the smallest rectangular region covering the i-th and j-th objects;

[0030] Generate semantic masks for the i-th and j-th objects:

[0031] M i +M j =SAM(I,b i ,b j )

[0032] Among them, M i +M j This represents a semantic mask containing the i-th and j-th objects, based on the joint region coordinates b of the i-th and j-th objects. i ∪b j The semantic mask M of the cropped output i +M j object pair o i and o j The joint region:

[0033] M ij =CROP(M i +M j ,b i ∪b j )

[0034] Among them, M ij The object to be cut is o i and o j The joint region of the semantic mask; using Gaussian blurring, the portion of x other than the i-th and j-th object pixels is blurred to obtain the blurred inverse mask:

[0035]

[0036] in, Indicates the object's relationship to o i and o j The blurred reverse mask, This indicates a Gaussian blur operation.

[0037] The initialization of multiple learnable context vector groups, which are concatenated with the word embeddings of the relations to serve as text prompts, is as follows:

[0038] Multiple vector groups are randomly initialized with a normal distribution as model parameters to provide contextual information for relation descriptions in text prompts; for a predefined set of relation categories, each predicate is encoded into a symbolic representation as a word embedding using the built-in word segmenter in the pre-trained visual language model.

[0039] Based on the position of the word embeddings in the text prompts according to their relation (before, in the middle, after), the vector groups are concatenated with the word embeddings of the relation in a 1:1:2 ratio.

[0040] The multiple sets of relation weight vectors are as follows:

[0041]

[0042] in, Let represent the set of relation weight vectors corresponding to the k-th text prompt, where This is the weight vector corresponding to the i-th relation category of the k-th text prompt, where C represents the number of predefined relation categories, and f T () represents the text encoder of the pre-trained visual language model CLIP. This represents the k-th text hint containing descriptions of all relation categories, where This represents the description of the c-th relation category in the k-th text prompt. Let M be the context vector set for the k-th text prompt, and r be the number of context vectors in each text prompt. c For the word embedding of the c-th relation category, ";" indicates a concatenation operation on the dimension, and K represents the number of text prompts.

[0043] Specifically, the verification of the relation predicate with the maximum prediction probability, and the real-time detection of abnormal behavior in large public places by the intelligent monitoring system based on the output relation category, are as follows:

[0044] The thought chain prompts are input into the large language model Llama 2. Following the instructions in the prompts, the large language model Llama 2 judges the rationality of the specified relation triples based on prior knowledge and whether they conform to common sense.

[0045] If "No" is returned, the specified object pairs in the test image will be treated as unrelated object pairs and the relationship between them will be considered as the background category;

[0046] If the return value is "Yes", the relation predicate with the highest probability in the output relation prediction probability distribution will be identified as the visual relation category between the specified object pairs in the test image and output to the intelligent monitoring system.

[0047] In a second aspect, a fine-grained multimodal cue-guided visual relationship recognition device, the device comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to cause the device to perform the method described in any one of the first aspects.

[0048] Third aspect, a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in any one of the first aspects.

[0049] The beneficial effects of the technical solution provided by this invention are:

[0050] 1. In designing fine-grained composite visual cues, this invention cropps the joint region of object pairs to provide rich contextual information. At the same time, it uses a pre-trained SAM (Segment Anything) model and Gaussian blur operation to generate a fuzzy inverse mask to segment the target object pair and the background region. The combination of the two helps the visual language model to accurately locate the target object pair, make full use of the surrounding contextual information, and achieve accurate recognition of visual relationships.

[0051] 2. This invention introduces multiple learnable text prompts, uses a multivariate Gaussian distribution to model multiple weight vectors of the relationship and derives the relationship classification loss. It uses a small number of samples for training and minimizes the loss function to optimize multiple text prompts so that they include multiple semantic aspects of the relationship. At the same time, it combines fine-grained composite visual prompts to accurately adjust the pre-trained visual language model for the visual relationship detection task, so that it can accurately identify multiple different visual representations of a relationship, further improving the accuracy of visual relationship recognition.

[0052] 3. This invention designs a text prompt template based on thought chain and introduces a pre-trained large language model Llama2 to verify the rationality of the predicted relationship between object pairs. By filtering out predictions that do not conform to common sense, this method solves the limitation of visual language models in predicting background categories when dealing with unrelated object pairs and effectively avoids false alarms in intelligent monitoring systems.

[0053] 4. In this invention, the image encoder and text encoder of the pre-trained visual language model are always frozen. This method only needs to use a small number of training samples for each relation to fine-tune the context vectors in multiple text prompts, avoiding fully supervised training on large-scale manually annotated relation data and biased predictions caused by the long-tail distribution of relation annotations in the dataset. At the same time, by utilizing prior knowledge from the visual language model, this method can effectively detect novel relation categories that have not been trained, enabling the intelligent monitoring system to cope with more diverse and complex scenarios and enhancing its adaptability.

[0054] 5. Applying the results generated by this invention to an intelligent monitoring system helps the system more accurately identify the visual relationships between objects (people) in large public places, such as shopping malls, parking lots, and banks, in real time, improves the ability to respond to abnormal behavior, triggers alarms in a timely manner, notifies relevant personnel, and effectively protects life and property safety. Attached Figure Description

[0055] Figure 1 A schematic diagram for detecting visual relationships between different object pairs;

[0056] Figure 2 This is a flowchart of a fine-grained, multimodal cue-guided visual relationship recognition method. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0058] Example 1

[0059] To address the issue that existing intuitive methods cannot accurately locate and identify visual relationships between target object pairs when adjusting visual language models for visual relationship detection, and to improve the accuracy of visual relationship recognition using prompt-guided visual language models, a commonly used metric is recall. (See [link to relevant documentation]). Figure 2 This invention provides a fine-grained, multimodal cue-guided visual relationship recognition method, which includes the following steps:

[0060] 101: For the visual relationship detection database used, the object detection framework is used to obtain the position coordinates, visual features and category prediction probability distribution of objects in the image;

[0061] The database used in this embodiment of the invention requires images to be labeled with the location and category of objects, as well as the relationship categories between objects. The database used is the Visual Genome database, but it is not limited to this database; other databases that meet the labeling conditions can also be used.

[0062] Existing object detection frameworks include Faster-RCNN, Mask-RCNN, and YOLO. This embodiment of the invention uses the Faster-RCNN (Fast Region Convolutional Neural Network) framework to extract information about objects in images. The feature extraction network in this framework can be a network structure such as GoogLeNet, ResNet-101, or VGG-16. This embodiment of the invention uses a VGG-16 convolutional network. Images from the database are input into the aforementioned object detection framework, and the output features include: the position coordinates of objects in the image, i.e., the coordinates of the smallest rectangular region covering the object, B = {b1, b2, ..., b...}.N}, where b i This represents the concatenation of the top-left and bottom-right coordinates of the smallest rectangular region covering the i-th object in the image, where N is the number of objects in the image; the visual features of the objects in the image are the features F = {f1, f2, ..., f...} output by the fully connected layer in the VGG-16 feature extraction network. N}, where f i Let L represent the visual features of the i-th object in the image; L represent the category prediction probability distribution of objects in the image, i.e., the features output by the classification network in the object detection framework, L = {l1, l2, ..., l...}. N}, where l i This is the probability distribution vector for predicting the category of the i-th object in the image.

[0063] 102: For the visual features and category prediction probability distribution of the object output in step 101, input them together into the encoder to obtain the context information of the object, and then input the context information into the decoder to output the predicted object category label.

[0064] The object category prediction probability distribution vector output in step 101 is mapped to the same dimension as the visual features. Then, it is concatenated dimensionally with the visual features and input into a Bidirectional Long Short-Term Memory (BiLSTM) encoder to obtain object context information. This context information is then input into an LSTM decoder to output the predicted object category.

[0065]

[0066] Among them, W l C represents the learnable feature mapping matrix, BiLSTM represents the encoder bidirectional long short memory network, and C o ={c1,c2,...,c N} represents the context information of the object, where c i For the context information of the i-th object in the image, LSTM represents the Long Short-Term Memory network of the decoder, O = {o1, o2, ..., o N} represents the predicted object category label, where o i Let Input be the category of the i-th object in the predicted image, and Output be the output.

[0067] 103: Using the object position coordinates output in step 101 and the object category labels output in step 102, the joint region of the target object pair is cropped, and a fuzzy inverse mask is generated using the SAM model and Gaussian blur operation to form a fine-grained composite visual cue. This cue is then input into the image encoder of the pre-trained visual language model to output composite image features.

[0068] For the pre-trained visual language model used, which includes an image encoder and a text encoder, this embodiment of the invention uses the CLIP (Contrastive Language-Image Pre-trained) model, but is not limited to this visual language model, and may also be other similar visual language models.

[0069] Detecting the i-th object o in the image output by step 102 i and the j-th object o j The visual relationships between them are first constructed by creating fine-grained, complex visual cues, including object pairs. i and o j The joint region, and the fuzzy reverse mask.

[0070] Specifically, based on the i-th object o output in step 101 i and the j-th object o j Position coordinates b i and b j Extract the minimum x and y coordinates of the top left corner and the maximum x and y coordinates of the bottom right corner, and combine them to obtain the joint region coordinates b of the i-th and j-th objects. i ∪b j Based on these coordinates, the image is cropped to obtain the object's position o. i and o j The joint region:

[0071] x = CROP(I,b) i ∪b j (2)

[0072] Where x represents the object pair o being cut. i and o j The joint region, CROP(·) represents the cropping operation, I represents the image, b i ∪b j Represents the coordinates of the smallest rectangular region covering the i-th and j-th objects.

[0073] Then, input the image and the position coordinates b of the i-th and j-th objects. i b j From the pre-trained SAM model, generate semantic masks for the i-th and j-th objects:

[0074] M i +M j =SAM(I,b i ,b j (3)

[0075] Among them, M i Let M represent the semantic mask for the i-th object. Its width and height are the same as the image. In the mask, the value corresponding to the pixel position of the i-th object in the image is 1, and the values ​​at other positions are 0.j M represents the semantic mask of the j-th object. i +M j SAM(·) represents a semantic mask containing the i-th and j-th objects. Its width and height are the same as the image. The values ​​of the pixels corresponding to the i-th and j-th objects in the image are 1, and the values ​​of the other pixels are 0. SAM(·) represents the Segment Anything Model.

[0076] Next, based on the joint region coordinates b of the i-th and j-th objects... i ∪b j The semantic mask M output by the trimming formula (3) i +M j object pair o i and o j The joint region:

[0077] M ij =CROP(M i +M j ,b i ∪b j (4)

[0078] Among them, M ij The object to be cut is o i and o j The joint region of the semantic mask.

[0079] The semantic mask M output by formula (4) ij The object pair o output by formula (2) i and o j The joint region x corresponds to M ij The values ​​at the same positions as the i-th and j-th pixels in x are 1, and the values ​​at other positions are 0, according to M. ij The values ​​in x are used to blur the portion of x other than the i-th and j-th object pixels using a Gaussian blur operation, thus obtaining a blurred inverse mask:

[0080]

[0081] in, Indicates the object's relationship to o i and o j The blurred reverse mask, This indicates a Gaussian blur operation.

[0082] Finally, the object pair o output by formula (2) i and o j The joint region x and the fuzzy inverse mask output by formula (5) The image encoder, fed into the pre-trained visual language model CLIP, is used to obtain composite image features:

[0083]

[0084] Where z represents the object pair o output by formula (2) i and o j Image features of the joint region x, This represents the fuzzy inverse mask output by formula (5). Image features, f I (·) represents the image encoder of the pre-trained visual language model CLIP.

[0085] 104: Initialize multiple learnable context vector sets, concatenate them with the word embeddings of the relations to serve as text prompts, input them into the text encoder of the pre-trained visual language model, and output multiple sets of relation weight vectors;

[0086] First, multiple vector groups are randomly initialized using a normal distribution as model parameters to provide contextual information for the relation descriptions in the text prompts. For a predefined set of relation categories, the built-in tokenizer in the pre-trained visual language model CLIP encodes each predicate into a symbolic representation as a word embedding, facilitating model understanding and processing.

[0087] Then, according to the position of the word embeddings of the relation in the text prompts: before, in the middle, and after, the vector groups are concatenated with the word embeddings of the relation in a 1:1:2 ratio, so that the multiple text prompts have positional diversity, so as to learn the semantics of the relation more fully.

[0088] Finally, multiple text prompts are input into the text encoder of the pre-trained visual language model to obtain multiple sets of relation weight vectors:

[0089]

[0090] in, Let represent the set of relation weight vectors corresponding to the k-th text prompt, where This is the weight vector corresponding to the i-th relation category of the k-th text prompt, where C represents the number of predefined relation categories, and f T (·) represents the text encoder of the pre-trained visual language model CLIP. This represents the k-th text hint containing descriptions of all relation categories, where This represents the description of the c-th relation category in the k-th text prompt, where Let M be the context vector set for the k-th text prompt, and r be the number of context vectors in each text prompt. cFor the word embedding of the c-th relation category, ";" indicates a concatenation operation on the dimension, and K represents the number of text prompts.

[0091] 105: For the relation weight vector output in step 104, model it using a multivariate Gaussian distribution, and combine it with the composite image features output in step 103. Based on a small number of training samples, derive the relation classification loss and calculate the semantic diversity loss, and minimize the total training loss to optimize the text prompt.

[0092] The limited number of training samples used in this embodiment of the invention consists of the top 16 representative object pairs for each relation category extracted from the training set of the database used. Specifically, all object pairs for each relation category are extracted from the training set of the database used. Based on the position information of the object pairs in the image, the SAM model and Gaussian blur operation are used to segment the object pairs and the background to generate a blurred inverse mask, which is input into the image encoder of the pre-trained visual language model CLIP to obtain image features. The default text prompt template "a photo of [relation]" is used as the text prompt and input into the text encoder of the pre-trained visual language model CLIP to obtain the weight vectors of all relations. The dot product similarity between the image features of the object pairs and the weight vector of each relation is calculated. After normalization, the relation prediction probability distribution is obtained, and the relation category with the highest probability is taken as the prediction result. Object pairs whose predicted categories match the actual relation categories are selected, and these object pairs are sorted from largest to smallest according to the predicted probabilities. The top 16 object pairs are selected as the training samples for their respective relation categories.

[0093] The set of relation weight vectors for the k-th text prompt output by formula (7) contains the weight vectors for all relation categories corresponding to the k-th text prompt. There are a total of K text prompts, so each relation category has K weight vectors.

[0094] Multiple weight vectors for a relation category are inherently consistent, representing multiple semantic aspects of the same relation. They tend to cluster tightly together in the output embedding space. A multivariate Gaussian distribution is suitable for fitting the distribution of multiple weight vectors for a relation category.

[0095] In the process of optimizing multiple text suggestions, it is assumed that the K weight vectors for each relation category follow a multivariate Gaussian distribution, and their parameters are estimated by calculating the mean and covariance of the K weight vectors for each relation category:

[0096]

[0097] Where, μ c Let represent the mean of the K weight vectors for the c-th relation category. Σ represents the weight vector corresponding to the c-th relation category of the k-th text prompt. c Let represent the covariance matrix of the K weight vectors of the c-th relation category.

[0098] In the small number of object pairs used for training, each object pair contains image features of its joint region and blurred inverse mask, as well as the basic fact relationship category between them. To facilitate the derivation and simplification of the relationship classification loss, the negative likelihood logarithm of the average probability that all training samples are predicted as the basic fact relationship category is calculated using the image features of the joint region of the object pair, its basic fact relationship category, and a set of K text prompt relationship weight vectors. Then, the average relationship classification loss for each object pair is obtained.

[0099]

[0100] in, Let p(y) represent the relation classification loss, T represent the number of training samples in a batch, and p(y) represent the loss. i |x i W k ) represents the probability that the set of relation weight vectors of the k-th text prompt will be used to predict the i-th object pair as its basic fact relation category during training, where x i For the joint region of the i-th object pair being clipped, y i z is the index of the basic fact relationship category between the i-th object pair. i This represents the image features of the joint region for the i-th object. This represents the y-th word corresponding to the k-th text prompt. i A weight vector for each relation category, where τ represents the temperature parameter.

[0101] Based on the following formula:

[0102]

[0103] in, This indicates the operation of averaging, where X represents the variable.

[0104] The relation classification loss can be further derived as:

[0105]

[0106] Among them, variables Follows Gaussian distribution in Indicates the yth i The mean of the K weight vectors for each relation category. Indicates the yth i The covariance matrix of K weight vectors for each relation category. Represents the sum of the K weight vectors of the c-th relation category and the y-th relation category. i The covariance matrix of K weight vectors for each relation category. Covariance matrix The transpose of .

[0107] Therefore, the moment generating function is based on the following normal distribution:

[0108]

[0109] in, This represents the averaging operation, where X represents a value with a mean of μ and a variance of σ. 2 Let t be a normally distributed random variable.

[0110] Formula (11) can be further simplified to:

[0111]

[0112] Using the upper limit of formula (13) and combining it with the composite image features output in step 103 to replace the image features of the joint region of the object pair, the final relation classification loss is calculated as follows:

[0113]

[0114] Where α and β represent hyperparameters. This represents the image features of the i-th object against the blurred inverse mask.

[0115] By continuously reducing the relation classification loss and optimizing the context vector groups in multiple text prompts, the true distribution of relation weights is estimated. To ensure that the context vector groups in multiple text prompts comprehensively capture various semantic aspects of the relations, they are individually input into the text encoder of the pre-trained visual language model CLIP to obtain the output embedding of each context vector group. The mean absolute dot product similarity of the output embeddings of any two context vector groups is calculated as the semantic diversity loss.

[0116]

[0117] in, V represents the loss of semantic diversity. i Let represent the context vector group in the i-th text prompt, as abbreviation, V j Let represent the context vector group in the j-th text prompt, . The abbreviation of .

[0118] By continuously reducing the semantic diversity loss and increasing the distance between the context vector groups of different text prompts, the semantic diversity of the text prompts is improved. Therefore, the total training loss is:

[0119]

[0120] in, This represents the total training loss. By minimizing the total training loss, we can fine-tune the text prompts for subsequent relational reasoning.

[0121] 106: Take the average weight vector of the relationship, calculate the similarity with the composite image features of the target object pair, then weight and fuse them, and obtain the relationship prediction probability distribution after normalization.

[0122] When the total training loss in step 105 is minimized, the multiple text prompts corresponding to this step are input into the text encoder of the pre-trained visual language model CLIP to obtain multiple sets of relation weight vectors output by formula (7). For each relation category, the average of the K weight vectors is taken as the final relation weight vector for visual relation reasoning.

[0123] During the inference phase, the pre-trained visual language model CLIP is used to detect the visual relationships between all object pairs in the test set images of the database. For the j-th object pair in a test image, step 103 obtains the composite image features of the j-th object pair, including the image features of its joint region and the image features of the blurred inverse mask.

[0124] The similarity is calculated using the mean of multiple weight vectors for each relationship category and the composite image features of the j-th object pair, then weighted and fused, and normalized to obtain the predicted probability distribution of the relationship between the j-th object pairs:

[0125]

[0126] Where p(c) represents the probability that the relationship between the j-th object pair is predicted to be the c-th relationship category, and z j This represents the image features of the joint region for the j-th object. This represents the image features of the j-th object against the blurred inverse mask.

[0127] 107: Using thought chain prompts and the large language model Llama 2, the rationality of the relation predicate with the highest predicted probability in the relation prediction probability distribution output in step 106 is verified. Based on the output relation category, the intelligent monitoring system can detect abnormal behavior in large public places in real time, thereby improving security capabilities and response speed.

[0128] In specific implementation, the embodiments of the present invention use the Llama 2 model as the large language model, but are not limited to this large language model, and can also be any other large language model that can interact with it.

[0129] For the predicted probability distribution of the relationship between the j-th object pair output in step 106, take the relation predicate with the highest predicted probability and the category labels of the subject and object objects in the j-th object pair output in step 102, and fill them into the corresponding positions in the following thought chain prompt template: {subject}, {predicate}, {object}:

[0130] System:Your task is to help user determine whether the relationbetween objects is reasonable.After thinking about it,only answer a word'Yes'or'No',don't give any reason.

[0131] User:Can the{subject}be{predicate}{object}?

[0132] Assistant: Let's think step by step!

[0133] Among them, System, User, and Assistant represent three different roles: system, user, and assistant, respectively.

[0134] Then, the above thought chain prompts are input into the large language model Llama 2. Following the instructions in the prompts, Llama 2 uses prior knowledge to determine the rationality of the specified relation triples and whether they conform to common sense. After the determination, Llama 2 returns a signal "Yes" or "No".

[0135] If "No" is returned, it means that the relationship predicate with the highest prediction probability occurring between the j-th object pair is contrary to common sense. The j-th object pair in the test image will be regarded as an unrelated object pair and the relationship between them will be regarded as the background category, and will not be output to the intelligent monitoring system.

[0136] If the return value is "Yes", it means that the relation predicate with the highest predicted probability is reasonable and may occur between the j-th object pair. Therefore, the relation predicate with the highest probability in the relation prediction probability distribution output by formula (17) is identified as the visual relation category between the j-th object pair in the test image and output to the intelligent monitoring system.

[0137] In summary, the embodiments of the present invention avoid the problem that existing intuitive methods cannot accurately locate and identify the visual relationship between target object pairs when adjusting the visual language model for visual relationship detection, improve the visual relationship recognition accuracy, i.e., recall rate, of the prompt-guided visual language model, and can be applied to any visual relationship detection database.

[0138] Example 2

[0139] A fine-grained, multimodal cue-guided visual relationship recognition device includes a processor and a memory. The memory stores program instructions, and the processor invokes the program instructions stored in the memory to cause the device to execute the following method steps in Embodiment 1:

[0140] For the obtained multiple sets of relation weight vectors, a multivariate Gaussian distribution is used to model them, and combined with the composite image features of the output, the relation classification loss is derived based on a small number of training samples and the semantic diversity loss is calculated. The total training loss is minimized to optimize the text prompts.

[0141] The average weight vector of the relationship is taken, and the similarity and weighted fusion of the composite image features of the target object pair are calculated respectively. The relationship prediction probability distribution is obtained after normalization.

[0142] The relational predicate with the highest predicted probability is validated, and the intelligent monitoring system detects abnormal behavior in large public places in real time based on the output relational category.

[0143] The steps for outputting composite image features are as follows:

[0144] Using the output object's position coordinates and object category labels, the joint region of the target object pair is cropped. A fuzzy inverse mask is generated using the SAM model and Gaussian blur operation to form a fine-grained composite visual cue, which is then input into the image encoder of a pre-trained visual language model to output composite image features.

[0145] The steps for obtaining multiple sets of relation weight vectors are as follows:

[0146] Multiple learnable context vector sets are initialized, and each set is concatenated with the word embeddings of the relation as text prompts. These prompts are then input into the text encoder of a pre-trained visual language model, which outputs multiple sets of relation weight vectors.

[0147] The joint region of the cropped target object pair is generated using the SAM model and Gaussian blurring operation to create a fuzzy inverse mask:

[0148] Obtain the object pair o by cropping the image based on coordinates. i and o j The joint region:

[0149] x = CROP(I,b)i ∪b j )

[0150] Where x represents the object pair o being cut. i and o j The joint region, CROP(·) represents the cropping operation, I represents the image, b i ∪b j Represents the coordinates of the smallest rectangular region covering the i-th and j-th objects;

[0151] Generate semantic masks for the i-th and j-th objects:

[0152] M i +M j =SAM(I,b i ,b j )

[0153] Among them, M i +M j This represents a semantic mask containing the i-th and j-th objects, based on the joint region coordinates b of the i-th and j-th objects. i ∪b j The semantic mask M of the cropped output i +M j object pair o i and o j The joint region:

[0154] M ij =CROP(M i +M j ,b i ∪b j )

[0155] Among them, M ij The object to be cut is o i and o j The joint region of the semantic mask; using Gaussian blurring, the portion of x other than the i-th and j-th object pixels is blurred to obtain the blurred inverse mask:

[0156]

[0157] in, Indicates the object's relationship to o i and o j The blurred reverse mask, This indicates a Gaussian blur operation.

[0158] In this process, multiple learnable context vector sets are initialized and concatenated with the word embeddings of the relations to provide text prompts:

[0159] Multiple vector groups are randomly initialized with a normal distribution as model parameters to provide contextual information for relation descriptions in text prompts; for a predefined set of relation categories, each predicate is encoded into a symbolic representation as a word embedding using the built-in word segmenter in the pre-trained visual language model.

[0160] Based on the position of the word embeddings in the text prompts according to their relation (before, in the middle, after), the vector groups are concatenated with the word embeddings of the relation in a 1:1:2 ratio.

[0161] Among them, the multiple sets of relation weight vectors are:

[0162]

[0163] in, Let represent the set of relation weight vectors corresponding to the k-th text prompt, where This is the weight vector corresponding to the i-th relation category of the k-th text prompt, where C represents the number of predefined relation categories, and f T () represents the text encoder of the pre-trained visual language model CLIP. This represents the k-th text hint containing descriptions of all relation categories, where This represents the description of the c-th relation category in the k-th text prompt. Let M be the context vector set for the k-th text prompt, and r be the number of context vectors in each text prompt. c For the word embedding of the c-th relation category, ";" indicates a concatenation operation on the dimension, and K represents the number of text prompts.

[0164] Specifically, the relational predicate with the highest prediction probability is validated, and the intelligent monitoring system detects abnormal behavior in large public places in real time based on the output relational category.

[0165] The thought chain prompts are input into the large language model Llama 2. Following the instructions in the prompts, the large language model Llama 2 judges the rationality of the specified relation triples based on prior knowledge and whether they conform to common sense.

[0166] If "No" is returned, the specified object pairs in the test image will be treated as unrelated object pairs and the relationship between them will be considered as the background category;

[0167] If the return value is "Yes", the relation predicate with the highest probability in the output relation prediction probability distribution will be identified as the visual relation category between the specified object pairs in the test image and output to the intelligent monitoring system.

[0168] It should be noted that the device descriptions in the above embodiments correspond to the method descriptions in the embodiments, and the embodiments of the present invention will not be repeated here.

[0169] The execution entities of the aforementioned processor and memory can be devices with computing functions such as computers, microcontrollers, and single-chip microcomputers. In specific implementations, the embodiments of the present invention do not limit the execution entities and can select them according to the needs of actual applications.

[0170] Data signals are transmitted between the memory and the processor via a bus, which will not be elaborated upon in this embodiment of the invention.

[0171] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium, the storage medium including a stored program, which, when the program is running, controls the device where the storage medium is located to execute the method steps in the above embodiments.

[0172] The computer-readable storage medium includes, but is not limited to, flash memory, hard disk, solid-state drive, etc.

[0173] It should be noted that the description of the readable storage medium in the above embodiments corresponds to the description of the method in the embodiments, and the embodiments of the present invention will not be repeated here.

[0174] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated.

[0175] A computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in or transmitted through a computer-readable storage medium. A computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic or semiconductor, etc.

[0176] Unless otherwise specified, the model numbers of the various devices in this embodiment of the invention are not limited, and any device that can perform the above functions is acceptable.

[0177] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0178] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A fine-grained, multimodal cue-guided visual relationship recognition method, characterized in that, The method includes: For the obtained multiple sets of relation weight vectors, a multivariate Gaussian distribution is used to model them, and combined with the composite image features of the output, the relation classification loss is derived based on a small number of training samples and the semantic diversity loss is calculated. The total training loss is minimized to optimize the text prompts. The average weight vector of the relationship is taken, and the similarity and weighted fusion of the composite image features of the target object pair are calculated respectively. The relationship prediction probability distribution is obtained after normalization. The relational predicate with the highest prediction probability is validated, and the intelligent monitoring system detects abnormal behavior in large public places in real time based on the output relational category. The step of outputting the composite image features is as follows: Using the output object's position coordinates and object category labels, the joint region of the target object pair is cropped. A fuzzy inverse mask is generated using the SAM model and Gaussian blur operation to form a fine-grained composite visual cue, which is then input into the image encoder of the pre-trained visual language model to output composite image features. The steps for obtaining the multiple sets of relation weight vectors are as follows: Multiple learnable context vector sets are initialized, and each set is concatenated with the word embeddings of the relation as text prompts. These prompts are then input into the text encoder of a pre-trained visual language model, which outputs multiple sets of relation weight vectors.

2. The fine-grained multimodal cue-guided visual relationship recognition method according to claim 1, characterized in that, The joint region of the cropped target object pair is generated using the SAM model and Gaussian blurring operation to create a fuzzy inverse mask as follows: Object pairs are obtained by cropping the image based on coordinates. and The joint region: ； in, Represents the object pair to be clipped and The joint region, This represents a cropping operation; 'I' represents the image. Represents the coordinates of the smallest rectangular region covering the i-th and j-th objects; Generate semantic masks for the i-th and j-th objects: ； in, This represents a semantic mask containing the i-th and j-th objects, based on the joint region coordinates of the i-th and j-th objects. The semantic mask of the output is trimmed. object pairs and The joint region: ； in, Represents the object pair to be clipped and Joint region of semantic mask; using Gaussian blur operation to... The portion of the image outside the i-th and j-th object pixels is blurred to obtain a blurred inverse mask: ； in, Represents object pairs and The blurred reverse mask, This indicates a Gaussian blur operation.

3. The fine-grained multimodal cue-guided visual relationship recognition method according to claim 1, characterized in that, The initialization of multiple learnable context vector groups, which are then concatenated with the word embeddings of the relations to serve as text prompts, is as follows: Multiple vector groups are randomly initialized with a normal distribution as model parameters to provide contextual information for relation descriptions in text prompts; for a predefined set of relation categories, each predicate is encoded into a symbolic representation as a word embedding using the built-in word segmenter in the pre-trained visual language model. Based on the position of the word embeddings in the text prompts according to their relation (before, in the middle, after), the vector groups are concatenated with the word embeddings of the relation in a 1:1:2 ratio.

4. The fine-grained multimodal cue-guided visual relationship recognition method according to claim 1, characterized in that, The multiple sets of relation weight vectors are: ； in, Let represent the set of relation weight vectors corresponding to the k-th text prompt, where Let C be the weight vector corresponding to the i-th relation category of the k-th text prompt, and let C represent the number of predefined relation categories. This represents the text encoder of the pre-trained visual language model CLIP. This represents the k-th text hint containing descriptions of all relation categories, where This represents the description of the c-th relation category in the k-th text prompt. Let M be the context vector set for the k-th text prompt, and M be the number of context vectors in each text prompt. For the word embedding of the c-th relation category, ";" represents the concatenation operation on the dimension, and K represents the number of text prompts.

5. The fine-grained multimodal cue-guided visual relationship recognition method according to claim 1, characterized in that, The verification of the relation predicate with the maximum predicted probability, and the real-time detection of abnormal behavior in large public places by the intelligent monitoring system based on the output relation category, specifically involves: The thought chain prompts are input into the large language model Llama 2. Following the instructions in the prompts, the large language model Llama 2 judges the rationality of the specified relation triples based on prior knowledge and whether they conform to common sense. If "No" is returned, the specified object pairs in the test image will be treated as unrelated object pairs and the relationship between them will be considered as the background category; If the return value is "Yes", the relation predicate with the highest probability in the output relation prediction probability distribution will be identified as the visual relation category between the specified object pairs in the test image and output to the intelligent monitoring system.

6. A fine-grained, multimodal cue-guided visual relationship recognition device, characterized in that, The device includes a processor and a memory, the memory storing program instructions, the processor invoking the program instructions stored in the memory to cause the device to perform the method according to any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in any one of claims 1-5.

Images