Dynamic scene graph generation method based on visual semantic reasoning

Abstract

The invention belongs to the field of dynamic scene graph generation in scene graph generation sub-problems, and particularly relates to a dynamic scene graph generation method based on visual semantic reasoning, which comprises the following steps of: acquiring a video, inputting the video into a trained dynamic scene graph generation model, obtaining objects and relationship categories among the objects, and generating a dynamic scene graph; constructing a scene graph structure of the video according to the objects and the relationship categories among the objects; the dynamic scene graph generation model comprises a pre-trained object detector, a time relation encoding module, a relation feature extraction module, a spatial knowledge encoder, a time-space relation decoding module and a classification module; according to the method, visual feature differences and semantic feature differences between adjacent video frames are calculated, loss calculation is carried out, and a direction is provided for correct prediction of a model; according to the method, object space information and time dependence are fused through a space encoder, correct relation labels are screened through priori knowledge, and the accuracy of relation prediction is improved.

Description

Technical Field

[0001] The present invention belongs to the field of dynamic scene graph generation in the scene graph generation sub-problem, and specifically relates to a dynamic scene graph generation method based on visual semantic reasoning. Background Art

[0002] The scene graph generation task is a classic computer vision downstream task. The purpose of dynamic scene graph generation is to identify objects in images or videos and infer the relationship between them. The dynamic scene graph generation system mainly includes three stages: object detection, temporal dependency extraction, and relationship recognition. Generally, we think that the first stage is an independent computer vision task. The dynamic scene graph generation work is mainly concentrated in the latter two stages, which can also be considered as a classification problem. In the video, due to the movement of objects and the movement of the camera, objects will be blurred, and there will be misdetection and missed detection of objects, so that objects cannot appear continuously in the video frames. In this case, it is difficult for downstream tasks to correctly understand the semantic information in the video, so it is very important to extract the temporal dependency between objects for relationship recognition.

[0003] However, video is an information-intensive media form in which temporal and spatial information are seriously entangled, making it difficult to clearly capture temporal dependencies. Moreover, as time goes by, objects in some frames become blurred and occluded, making it impossible to provide high-quality visual features. In addition, the relationships between objects will also change. Dynamic scene graph generation faces the following problems: objects in the video are moving, and the objects detected in the video frames will be affected by factors such as motion blur or occlusion, resulting in misdetection or missed detection of the same object in video frames at different time nodes, making it impossible to provide continuous high-quality feature information for the relationship detection stage; the movement of objects may mean that the relationship between objects will also change, and this slight visual change is difficult for object detectors to accurately extract.

[0004] Existing dynamic scene graph generation methods are mostly divided into two types according to the data segmentation granularity: video-level dynamic scene graph generation methods and frame-level dynamic scene graph generation methods. Video-level dynamic scene graph generation methods divide the input video into multiple small video clips, and each video clip generates a scene graph; frame-level dynamic scene graph generation methods divide the input video into a sequence of key video frames, and each key frame generates a scene graph. The former requires that long videos be accurately cut into short video clips (e.g., 10 frames) according to the precise time division of the relationship, but due to the fuzzy boundaries of the temporal relationship, cutting is very difficult. In contrast, the latter provides a more flexible implementation mechanism for the relationship representation in continuous video frames. However, most of the current frame-level dynamic scene graph generation methods are still not ideal in terms of consistency detection of the same objects between consecutive frames and dynamic relationship transformation of the same object pairs.

[0005] Therefore, how to construct consistent objects in videos and accurately identify changes in relationships between objects is an urgent problem to be solved. Summary of the invention

[0006] In order to solve the above-mentioned problems in the prior art, the present invention adopts a dynamic scene graph generation method based on visual semantic reasoning, comprising: obtaining a video, inputting the video into a trained dynamic scene graph generation model, obtaining objects and relationship categories between objects, and constructing a scene graph structure of the video according to the objects and relationship categories between objects; the dynamic scene graph generation model comprises: a pre-trained object detector, a temporal relationship encoding module, a relationship feature extraction module, a spatial knowledge encoder, a spatiotemporal relationship decoding module and a classification module;

[0007] The training process of the dynamic scene graph generation model includes:

[0008] S1. Obtain a training set, where the training set includes multiple training samples, each training sample includes a video and its prior knowledge; each video includes multiple video frames; a knowledge matrix of object pairs is constructed based on the prior knowledge of the training set; the prior knowledge of the video includes: object pairs in each video frame and their relationship categories and relationship labels in each relationship type;

[0009] S2, input each video frame of the video into a pre-trained object detector to obtain the object of the detected video frame and its object information; the object information includes: object features, object category and object detection box;

[0010] S3, inputting the object information of the video frame into the temporal relationship encoding module to obtain the temporal object feature Z1 of the video;

[0011] S4, inputting the object information and the temporal object feature Z1 of the video frame of the video into a relational feature extraction module to obtain the relational feature X2 of the object pair of the video;

[0012] S5, input the relationship feature X2 and the knowledge matrix into the spatial knowledge encoder to obtain the spatial relationship feature Z2 of the object pair in the video;

[0013] S6: Input the relationship feature X2 and the spatial relationship feature Z2 of the video object pair into the spatiotemporal relationship decoding module to obtain the spatiotemporal relationship feature Z2 of the video object pair. Fin ;

[0014] S7, the spatiotemporal relationship feature Z Fin And the time object feature Z1 is input into the classification module to obtain the probability distribution vector P of the predicted object pair in each relationship type v and the relationship category in each relationship type and the category probability distribution vector P of the predicted object O ; v is the index of the relationship type;

[0015] S8, calculating the visual feature difference and the semantic feature difference according to the relationship features of the object pairs in the video and the predicted relationship categories of the object pairs, and calculating the probability distribution vector P according to the visual feature difference, the semantic feature difference and the probability distribution vector P of the video. v And the probability distribution vector P O The loss function value is calculated, and the dynamic scene graph generation model parameters are updated according to the loss function value. When the loss function value is minimized, the trained dynamic scene graph generation model is obtained.

[0016] Beneficial effects:

[0017] 1. The present invention calculates the visual feature differences and semantic feature differences between adjacent video frames, performs loss calculation on the visual feature differences and semantic feature differences, uses the difference in semantic features as a guide for the visual feature differences, and provides direction for the correct prediction of the model; 2. The present invention inputs the relationship feature X2 into the spatial knowledge encoder to fuse the object spatial information and time dependency, and corrects the false detection and missed detection problems of the object detector; 3. The present invention inputs the relationship feature into the spatial knowledge encoder, and uses prior knowledge to screen the correct relationship label, thereby improving the accuracy of relationship prediction; 4. The temporal decoder of the present invention has two self-attention mechanisms, one of which uses the temporal relationship mask to prevent future objects from affecting the calculation of the current relationship features generated by the relationship, thereby improving the quality of relationship prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flow chart of a method for generating a dynamic scene graph based on visual semantic reasoning provided by an embodiment of the present invention

[0019] Figure 2 A flow chart of prior knowledge extraction provided by an embodiment of the present invention;

[0020] Figure 3 A flowchart of object trajectory extraction provided by an embodiment of the present invention;

[0021] Figure 4 A schematic diagram of the network structure of an object trajectory encoder provided by an embodiment of the present invention;

[0022] Figure 5 A flow chart of calculating the visual feature difference and semantic feature difference of a relational feature provided by an embodiment of the present invention;

[0023] Figure 6 A schematic diagram of a spatial knowledge encoder network structure provided by an embodiment of the present invention;

[0024] Figure 7 A schematic diagram of a time decoder network structure provided by an embodiment of the present invention;

[0025] Figure 8A schematic diagram of the result of generating a video relationship scene graph provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] like Figure 1 As shown, the present invention adopts a dynamic scene graph generation method based on visual semantic reasoning, including obtaining a video, inputting the video into a trained dynamic scene graph generation model, obtaining objects and relationship categories between objects, and constructing a scene graph structure of the video according to the objects and relationship categories between objects; the dynamic scene graph generation model includes: a pre-trained object detector, a temporal relationship encoding module, a relationship feature extraction module, a spatial knowledge encoder, a spatiotemporal relationship decoding module and a classification module;

[0028] The training process of the dynamic scene graph generation model includes:

[0029] S1. Obtain a training set, where the training set includes multiple training samples, each training sample includes a video and its prior knowledge; each video includes multiple video frames; and construct a knowledge matrix of object pairs based on the prior knowledge of the training set;

[0030] The dataset of the present invention is the Action Genome dataset, which contains 36 object categories, 26 relationship tags, 476229 object boxes and 135848 object pairs; the relationship categories are divided into three different types: attention relationship (Attention), spatial relationship (Spatial) and contact relationship (Contacting); the Action Genome dataset contains 7464 videos with 166785 frames for training and 1737 videos with 54371 frames for testing; the video frame sequence V of each video is [I1, I2···, I T ], where I t is the video frame, and T represents the number of video frames in the video.

[0031] The prior knowledge of each video includes the objects of each video frame and their object categories and object frames, as well as the object pairs of each video frame and their relationship categories and relationship labels in each relationship type; wherein the object pairs of the video frames are the combination of any two objects in the video frames.

[0032] The attention relationship indicates whether the object's eyes are focused on a certain object, and contains 3 relationship tags; the contact relationship indicates whether the object is in contact with a certain object, and contains 17 relationship tags; the spatial relationship indicates the position of the object in another object, and contains 6 relationship tags.

[0033] like Figure 2 As shown, constructing a knowledge matrix based on the prior knowledge of the training set includes:

[0034] S11. According to the relationship labels of the object pairs in each relationship type (there are 26 relationship labels in total for the three types), the dimension of each group of object pairs in each video frame in the training set is constructed as R. 1×26 The relationship vector of

[0035] Specifically, if there are objects o1, *2, and o3 in a frame, and then the objects are combined in pairs, there will be object pairs (o1, *2), (*1, *3), and (*2, *2).

[0036] An array of 26 elements records all the relationship labels that appear in the object pair (o1, o2). Subscripts 0-2 represent attention relationships, subscripts 3-8 represent spatial relationships, and subscripts 9-25 represent contact relationships. For example, in the first video frame, (o1, o2) appears. The relationship category of (o1, o2) in the attention relationship a is a1, and the relationship category in the spatial relationship s is s1. There is no corresponding relationship label in the contact relationship; the subscript occupied by a1 is 1, and the subscript occupied by s1 is 4. Since there is no relationship label in the contact relationship, the relationship vector subscripts 9 to 25 are all 0, so the relationship vector of (o1, o2) in the first video frame is [0,1,0,0,1,0,0,0,0,…,0];

[0037] S12, in all video frames of all videos in the training set, accumulating the relationship vectors belonging to the same group of object pairs to obtain the accumulated relationship vectors of each group of object pairs;

[0038] S13, calculating the distribution probability of the relationship label of each group of object pairs in the corresponding relationship type according to the accumulated relationship vector of each group of object pairs, and obtaining the relationship knowledge vector K of each group of object pairs;

[0039] Specifically, calculating the distribution probability of the relationship label in the relationship type a includes: there are N in the relationship type a a There are relationship labels, Then the probability distribution of object pairs in relation type a is and The same applies to relationship types s and c.

[0040] The probability distribution of each object pair in the three relationship categories is concatenated to obtain the relationship knowledge vector K of each object pair. Specifically:

[0041]

[0042] in, Respectively represent the distribution probability of the object pair in the relationship type a, s, c. Concat() represents the concatenation operation. N r =26 represents the number of relationship tags.

[0043] S14. Superimpose the relationship knowledge vectors of all object pairs to obtain the knowledge matrix of the object pairs Among them, N p is the number of object pairs;

[0044] S2, input each video frame of the video into a pre-trained object detector to obtain the object of each video frame and its object information;

[0045] Each video frame I t The detected object information includes: object features Object Class And object detection box Among them, N t is the number of objects detected.

[0046] S3, inputting the object information of the video frame into the temporal relationship encoding module to obtain the temporal object features of the video;

[0047] Object detection box B t Including coordinate information E t ;The temporal relationship encoding module includes: an object trajectory extraction module, a position encoding embedding module and an object trajectory encoder;

[0048] like Figure 4 As shown, the processing of the temporal relationship encoding module includes:

[0049] S31, input the object features of the video frame into the object trajectory extraction module to obtain the object trajectory feature matrix F o ;

[0050] like Figure 3 As shown, the object trajectory extraction module extracts the object trajectory feature matrix including:

[0051] Construct object trajectory features for all object categories from the object information of video frames

[0052] s l ={s l,1 ,s l,2 ,…,s l,T}, where sl is the object trajectory feature of object category l, s l,t N represents the features of objects belonging to object category l in video frame t, u Represents the number of object categories: 36.

[0053] The object trajectory feature of object category l is the trajectory feature of the object belonging to object category l in the video frame. If there are multiple objects of the same object category in a video frame, the trajectory feature of the object with the highest confidence is selected as the trajectory feature of the object category. The object information obtained in step S2 also includes the detected object category L t Confidence with the object,Confidence refers to the matching score between the object and the object category.

[0054] The object trajectory features of each object category are superimposed to obtain the object trajectory feature matrix F o , specifically:

[0055]

[0056] Among them, σ represents a function that fills a variable list with filling values, aligning the object trajectory features of different lengths (empty positions are filled with 0) and superimposing them together.

[0057] S32, the object trajectory feature matrix F o The coordinate information of its object detection box is input into the position encoding embedding module to obtain the position encoding feature X1:

[0058] X1=κ(F o ,W f (E)

[0059] Where κ represents the position encoding embedding function, W g represents the linear transformation function, and E represents the coordinate information of the object detection box.

[0060] S33, input the position coding feature X1 into the object trajectory encoder to obtain the temporal object feature of the video Among them, N A represents the number of objects detected in the entire video, d model The dimension representing each object feature detected.

[0061] The object trajectory feature matrix and the object detection frame coordinate information containing the object's spatial position information are position-encoded and input into the object trajectory encoder in order to extract the temporal dependency of objects between multiple frames, correct and supplement the objects that are misdetected or missed by the object detector in step S2, and provide support for the subsequent formation of correct relationship object pairs.

[0062] The object trajectory encoder uses a standard encoder based on a multi-head attention mechanism, which consists of n stacked Transformer Encoder Layers. The Transformer Encoder Layer consists of a multi-head attention mechanism, a residual connection and normalization layer (Add&Norm), and a feedforward neural network (FNN). The feedforward network (FNN) consists of a linear layer, a RELU activation layer, and a random loss layer Dropout.

[0063] Specifically, X1 is input into the first coding layer, the output of the first coding layer is input into the second coding layer, and the output of the k-1th coding layer is input into the kth coding layer, where k is the index of the coding layer. The object trajectory feature Z1 containing time-dependent information obtained after passing through n layers of encoders is expressed as:

[0064]

[0065] Z1=LayerNorm(Z″1+FFN(Z″1))

[0066] Among them, Z′1 is the result of the multi-head attention mechanism, Z″1 is the output result of the residual connection and normalization layer, is the output of the n-1th encoding layer; the object trajectory feature Z1 containing time-dependent information is expanded in time order in frames to obtain the time object feature of each video frame t Among them, N A Indicates the number of objects detected in the entire video, d model Represents the feature dimension of each object, M O represents the filling mask, i.e. the location of the filling value, which is represented by the object trajectory feature F o Construction can avoid having the object features at the fill value positions participate in the calculation of the attention mechanism in the encoder.

[0067] S4, inputting object information and temporal object features of video frames of the video into a relational feature extraction module to obtain relational features of object pairs of the video;

[0068] The relationship feature extraction module calculates the relationship features of the object pairs in the video, including:

[0069] S41, using the word vector model to identify the object category of each object i detected in the video frame t Processing to obtain the semantic features of the object

[0070]

[0071] Among them, Glove represents the pre-trained word embedding model Glove-200d, which maps object categories into 200-dimensional semantic embedding vectors. Represents the object category of object i.

[0072] S42, combining the objects detected in the video frame t in pairs to obtain multiple groups of object pairs<i,j> ; Object detection box based on object i, j Calculate the object pair<i,j> The joint box feature Object detection box Combine with the joint frame feature to get the detection frame feature

[0073]

[0074] S43, the object<i,j> Temporal object characteristics Detection box features And object semantic features Combine to get the object pair<i,j> Relationship characteristics

[0075]

[0076] Where t = [1, 2···, T] represents the t frame in the video, i, j, k∈O t , i≠j≠k, is the index of the object, O t are all objects detected in frame t, P t Represents the object pair feature of frame t, and the feature of an object pair in frame t is Concat(,) represents the concatenation operation. Indicates a flattening operation. represents the addition of corresponding elements, W s ,W o ,W u represents the linear transformation matrix, represents the features of the joint box calculated by RoIAlign, h box Indicates that the detection boxes of object i and object j are transformed into Features of the same dimension,

[0077] Calculate the joint box features of each set of object pairs through region alignment RoIAlign Includes: extracting the box area where two objects in the object pair exist at the same time, called the union box feature Union Boxes.

[0078] S44, combining the relationship features of all object pairs in the video frame t to obtain the relationship feature P of the video frame tt ; Combine the relational features of all video frames to obtain the relational features X2 of the object pairs in the video;

[0079] Since the number of object pairs obtained in each video frame is not exactly the same, when combining, it is necessary to fill invalid values ​​into the relational features of each video frame so that the relational features of each video frame have the same shape.

[0080] S5, inputting the relationship features of the object pairs in the video and the knowledge matrix into the spatial knowledge encoder to obtain the spatial relationship features of the object pairs in the video;

[0081] like Figure 6 As shown, the processing of the spatial knowledge encoder includes:

[0082] The spatial knowledge encoder adopts a standard encoder based on the multi-head attention mechanism, which is composed of n stacked TransformerEncoder Layer layers; the Transformer Encoder Layer consists of Multi-Head Attention, Add&Norm and FNN.

[0083] In the spatial knowledge encoder, X2 and X are input into the first encoding layer. The query vector Q and key vector K of the first encoding layer are obtained by linear transformation after X2 fuses the knowledge matrix X of the object pair. The value vector V of the first layer is directly obtained by linear transformation of X2 with the same dimension. Specifically:

[0084]

[0085] Among them, W q ,W k ,W v represents the linear transformation matrix, Indicates the addition of corresponding elements.

[0086] The output of the first coding layer is input into the second coding layer, and the output of the k-1th coding layer is input into the kth coding layer, where k is the index of the coding layer; the spatial relationship feature Z2 obtained after passing through n layers of encoders is expressed as:

[0087]

[0088] Z2=LayerNorm(Z″2+FFN(Z″2))

[0089] Among them, Z′2 is the result of the multi-head attention mechanism, Z″2 is the output result of the residual connection and normalization layer, is the output of the n-1th encoding layer,

[0090] Inputting relation features into the spatial knowledge encoder can not only extract the spatial information in the relation features, but also use the prior knowledge of the relationship to restrict the relationship selection of object pairs, eliminate impossible relationship labels, and distribute the relationship probability on possible relationship labels, which will be conducive to the selection of correct relationship labels.

[0091] S6, inputting the relationship feature X2 of the object pair of the video and the spatial relationship feature of the object pair into the spatiotemporal relationship decoding module to obtain the spatiotemporal relationship feature of the object pair of the video; the spatiotemporal relationship decoding module includes: a time decoder;

[0092] The specific process of the spatiotemporal relationship decoding module includes: constructing the object pair trajectory feature sequence based on the object pair relationship feature X2, generating the temporal relationship mask matrix based on the object pair trajectory feature sequence and sequence mask matrix; input the spatial relationship features, the temporal relationship mask matrix and the sequence mask matrix into the temporal decoder to obtain the spatiotemporal relationship features of the video.

[0093] Constructing the object pair trajectory feature sequence includes: finding the object pair in the object pair relationship feature X2<i,j> , the object pairs in each frame<i,j> By splicing the relationship features of the two objects, we can obtain the trajectory features of the object with time information.

[0094] The rows and columns of the temporal relationship mask matrix and the sequence mask matrix are the features of the object pairs in the object pair trajectory feature sequence. The values ​​on the diagonal of the temporal relationship mask matrix and the sequence mask matrix are all 1. In the object pair trajectory feature sequence, when the object pair p t b has appeared before the object pair p t d, the position of the temporal relationship mask matrix (p t b,p t d) is 1, and the rest are 0; when the object is p t b has appeared before the object pair p t d and b is equal to d, the position (p t b,p t d) is 1, and the rest are 0; t b is the feature of object pair b in video frame t.

[0095] like Figure 7 As shown, the process of the time decoder includes:

[0096] The temporal decoder uses a standard decoder based on a multi-head attention mechanism, which consists of n stacked TransformerDecoder Layer layers. The Transformer Decoder Layer consists of Multi-HeadAttention, Add&Norm, and FNN.

[0097] In the temporal decoder, specifically, Z2 is input into the first coding layer, the output of the first coding layer is input into the second coding layer, and the output of the k-1th coding layer is input into the kth coding layer, where k is the index of the coding layer; the output Z3 obtained after Z2 passes through the n-layer decoder is expressed as:

[0098] X′2=MultiHeadAtt(Q=W q Z2 (n-1) ,K=W k Z2 (n-1) ,V=W v Z2 (n-1) ,mask=M P )

[0099] Z′3=MultiHeadAtt(Q=W q X′2,K=W k X′2,V=W v X′2,mask=M S 0

[0100] Z″3=LayerNorm(Z′3+X′2)

[0101] Z3=LayerNorm(Z″3+FFN(Z″3))

[0102] Among them, Z′3 is the result of the multi-head attention mechanism, Z″3 is the output result of the residual connection and normalization layer, and Z2 (n-1) is the output of the n-1th encoding layer, is the spatiotemporal relationship feature, and Z3 is simply recorded as Z Fin , N p represents the number of object pairs detected in the entire video, d model Represents the dimension of each relationship feature, M P represents the temporal relation mask matrix, M S Represents the sequence mask matrix, which acts on each relation feature so that the decoder cannot see future information.

[0103] There are two self-attention mechanisms in the temporal decoder. The first one is to calculate the relational features of the same object pairs through the temporal relation mask M P Do not let future objects affect the calculation of current relationship features on the generated relationship. The second is the self-attention mechanism in the classic decoder, using the sequence mask M S Generate relation labels for each object pair in turn.

[0104] In one embodiment, the Transformer Encoder Layer structure of the temporal decoder includes: a first layer of MultiheadAttention, with an input and output channel number of 1936; a second layer of Dropout, with an input and output channel number of 1936; a third layer of MultiheadAttention, with an input channel number of 1936; a fourth layer of Dropout, with an input and output channel number of 1936; a fifth layer of Layer Norm, with an input channel number of 1936; a sixth layer of Linear, with input and output channel numbers of 1936 and 2048 respectively; a seventh layer of RELU, with an input and output channel number of 2048; an eighth layer of Dropout, with an input and output channel number of 2048; a ninth layer of Linear, with input and output channels of 2048 and 1936 respectively; a tenth layer of Dropout, with an input and output channel number of 1936;

[0105] S7, input the spatiotemporal relationship features and temporal object features Z1 of the object pair into the classification module, and obtain the predicted probability distribution vector of the object pair in each relationship type and the relationship category in each relationship type as well as the predicted category probability distribution vector P of the object O ;

[0106] Classification includes: the spatiotemporal relationship feature Z of the object pair Fin Three types of relationship label classification are performed to classify the temporal object feature Z1, specifically:

[0107]

[0108] Among them, Pa, Ps, and Pc are the probability distribution vectors of the relationship labels of the object pairs in the three types, respectively. O is the category probability distribution vector of the predicted object, η a ,η s ,η c ,η O represents the linear mapping function, N a ,N s ,N c Represents the number of relation labels in the three relation types, N O is the number of object category labels, and the sigmoid function maps the output to a range of 0 to 1.

[0109] S8. Calculate the visual feature differences and semantic feature differences of the relationship features of the video according to the object information of the video frames, calculate the loss function value according to the visual feature differences, semantic feature differences, object relationship category prediction results and true labels of the video, and update the dynamic scene graph generation model parameters according to the loss function value. When the loss function value is minimum, obtain the trained dynamic scene graph generation model.

[0110] Calculating the visual feature differences of the video includes:

[0111] Construct a matrix based on the relationship features of the video Save the relationship features of each pair of objects in the video; where N p , T and d model Represent the number of object pairs, the number of video frames, and the dimension of the relational features of object pairs respectively;

[0112] The matrix M u According to the second dimension, the offset subtraction represents the visual features of the object pair in the next frame minus the visual features of the same object pair in the previous frame, as follows:

[0113]

[0114] There are three types of relations between each object pair in the dataset, so the semantic category difference needs to be calculated separately for each relation type of each object pair.

[0115] like Figure 5 As shown, the semantic feature differences of the calculated videos include:

[0116] The pre-trained word vector model is used to process the relationship categories of the predicted object pairs in the relationship types a, s, and c to obtain the semantic features of the object pairs in the relationship types a, s, and c:

[0117] When the relation category consists of phrases, the word vector of each word is calculated and added together to obtain the final semantic features. Specifically:

[0118]

[0119] in, Relationship classes representing the three relationship types, represents a phrase segmentation function, Glove represents the pre-trained Glove-200d model, which maps the relation category to a 200-dimensional word embedding vector. The final semantic feature is obtained by adding the word vectors of multiple words in the three relationship types.

[0120] Construct three semantic feature matrices Separately save the semantic features of each set of object pairs in relation types a, s, and c; where N p , T represent the number of object pairs and the number of video frames, respectively, N a ,N s ,N c The number of relation categories representing relation types a, s, and c respectively;

[0121] The three matrices M o,a , M o,s , M o,c According to the second dimension, the semantic features of each relationship type of the object pair in the next frame are subtracted from the semantic features of the same object pair in the previous frame:

[0122]

[0123] The calculation process of the loss function is:

[0124] The predicted object category probability distribution vector P is calculated using the cross entropy loss O and the object’s true category label T O The difference is:

[0125]

[0126] Among them, Linear, BatchNorm and ReLU represent linear transformation function, batch normalization function and nonlinear activation function respectively, i is the index of the object, N A is the number of objects.

[0127] The mean square error loss is used to calculate the difference between the visual feature difference and the semantic feature difference of the object pair in the relation type a, s, c. diff,a 、Loss diff,s 、Loss diff,c , the formula is

[0128]

[0129] Among them, L a ,L s ,L c They represent linear transformation functions respectively.

[0130] Use cross entropy loss to calculate the predicted distribution P of relation type a a and the true category label vector T a The gap between The formula is:

[0131]

[0132] Where i is the index of the relationship category of the object pair in the relationship type a.

[0133] Calculate the predicted distribution P of relation type s using multi-label marginal loss s and the true category label vector T s The gap between The formula is:

[0134]

[0135] Among them, i and j are the true category label vectors T s The index of .

[0136] Calculate the predicted distribution P of relation type c using multi-label marginal loss c and the true category label vector T c The gap between The formula is:

[0137]

[0138] Among them, i and j are the true category label vectors T c The index of .

[0139] The total loss L of the model is:

[0140]

[0141] in, is the cross entropy loss CE of the object category, is the mean square error loss BCE of the visual feature difference and the semantic feature difference, is the cross entropy loss of relation type a, and The multi-label margin losses for relation types s and c, respectively.

[0142] The present invention also provides a dynamic scene graph generation system based on visual semantic reasoning, comprising: a visualization interface, a video input module, a priori knowledge extraction module, an object detection module, a temporal relationship encoding module, a spatial knowledge encoding module, a spatiotemporal relationship decoding module and a relationship classification module;

[0143] The video input module is used to input video;

[0144] The prior knowledge extraction module is used to count all the relationship labels existing in each relationship category in the training set. It can be used in the encoder to narrow the range of relationship labels that can be selected for each relationship category and improve the matching probability of the correct relationship label.

[0145] The object detection module is used to detect objects and related information in video frames;

[0146] The temporal relationship encoding module is used to group objects of the same object category together and feed them into the encoder. The temporal dependency of objects between adjacent frames can be encoded in the object features.

[0147] The spatial knowledge encoding module is used to encode the spatial position relationship of the object pairs in the video frame in the relationship feature;

[0148] The spatiotemporal relationship decoding module is used to extract the temporal dependency of the same object pairs and decode the spatiotemporal relationship features of each object pair in time order;

[0149] The relationship classification module classifies the spatiotemporal relationship features and outputs three different types of relationship labels;

[0150] like Figure 8 As shown, the visualization interface includes: a weight path selection interface, a video path selection interface and a video-generated relationship graph structure display interface; the weight path selection interface is used to select a dynamic scene graph generation model locally, the video path selection interface is used to select a video, and the video-generated relationship graph structure display interface is used to display the scene graph structure output by the dynamic scene graph generation model; the scene graph structure includes objects: person, bed, laptop and the relationships between objects: behind, touching, lying on, not looking at, and looking at.

[0151] The dynamic scene graph generation based on visual semantic reasoning of the present invention can be applied to tasks that require video analysis, understanding and early warning. For example, in the medical field, video scene graph generation can be applied to dynamic medical image analysis, such as cardiac ultrasound or abdominal examination videos. It can help doctors automatically annotate and analyze key objects (such as organs, tissues, etc.) in medical videos and their relationships, thereby improving diagnostic accuracy.

[0152] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating a dynamic scene graph based on visual semantic reasoning, characterized in that: include: Obtain a video, input the video into a trained dynamic scene graph generation model, obtain objects and relationship categories between objects, and construct a scene graph structure of the video based on the objects and relationship categories between objects; The dynamic scene graph generation model includes: a pre-trained object detector, a temporal relationship encoding module, a relation feature extraction module, a spatial knowledge encoder, a spatiotemporal relationship decoding module, and a classification module; The training process of the dynamic scene graph generation model includes: S1. Obtain a training set, where the training set includes multiple training samples, each training sample includes a video and its prior knowledge; each video includes multiple video frames; a knowledge matrix of object pairs is constructed based on the prior knowledge of the training set; the prior knowledge of the video includes: object pairs in each video frame and their relationship categories and relationship labels in each relationship type; S2, input each video frame of the video into a pre-trained object detector to obtain the object of the detected video frame and its object information; the object information includes: object features, object category and object detection box; S3, inputting the object information of the video frame into the temporal relationship encoding module to obtain the temporal object feature Z1 of the video; S4, inputting the object information and the temporal object feature Z1 of the video frame of the video into a relational feature extraction module to obtain the relational feature X2 of the object pair of the video; S5, input the relationship feature X2 and the knowledge matrix into the spatial knowledge encoder to obtain the spatial relationship feature Z2 of the object pair in the video; S6: Input the relationship feature X2 and the spatial relationship feature Z2 of the video object pair into the spatiotemporal relationship decoding module to obtain the spatiotemporal relationship feature Z2 of the video object pair. Fin ; S7, the spatiotemporal relationship feature Z Fin And the time object feature Z1 is input into the classification module to obtain the probability distribution vector P of the predicted object pair in each relationship type v and the relationship category and the predicted object category probability distribution vector P O ; v is the index of the relationship type; S8, according to the relationship feature X2 of the object pair in the video and the relationship category of the predicted object pair, the visual feature difference and the semantic feature difference are calculated, and the probability distribution vector P is calculated according to the visual feature difference, the semantic feature difference and the probability distribution vector P of the video. v And the probability distribution vector P O The loss function value is calculated, and the dynamic scene graph generation model parameters are updated according to the loss function value. When the loss function value is minimized, the trained dynamic scene graph generation model is obtained.

2. The method for generating a dynamic scene graph based on visual semantic reasoning according to claim 1, characterized in that: Constructing a knowledge matrix based on prior knowledge of the training set includes: S11, constructing a relationship vector for each group of object pairs in each video frame in the training set according to the relationship labels of the object pairs in each relationship type; S12, in all video frames of all videos in the training set, accumulating the relationship vectors belonging to the same group of object pairs to obtain the accumulated relationship vectors of each group of object pairs; S13, calculating the distribution probability of the relationship label of each group of object pairs in the corresponding relationship type according to the accumulated relationship vector of each group of object pairs, and obtaining the relationship knowledge vector of each group of object pairs; S14. Superimpose the relationship knowledge vectors of all object pairs to obtain the knowledge matrix X of the object pairs.

3. The method for generating a dynamic scene graph based on visual semantic reasoning according to claim 1, characterized in that: The temporal relationship coding module includes: an object trajectory extraction module, a position coding embedding module and an object trajectory encoder; the temporal relationship coding module processes the object information of the video frame including: S31, input the object features of the video frame into the object trajectory extraction module to obtain the object trajectory feature matrix F o ; S32, the object trajectory feature matrix F o The coordinate information of the object detection box is input into the position encoding embedding module to obtain the position encoding feature X1; S33, input the position coding feature X1 into the object trajectory encoder to obtain the temporal object feature of the video.

4. The method for generating a dynamic scene graph based on visual semantic reasoning according to claim 1, characterized in that: The relational feature extraction module processes the object information and temporal object features of the video frames of the video including: S41, using the word vector model to identify the object category of each object i detected in the video frame t Processing to obtain the semantic features of the object S42, combining the objects detected in the video frame t in pairs to obtain multiple groups of object pairs<i,j> ; Object detection box based on object i, j Calculate the object pair<i,j> The joint box feature Object detection box Combine with the joint frame feature to get the detection frame feature S43, the object<i,j> Temporal object characteristics Detection box features And object semantic features Combine to get the object pair<i,j> Relationship characteristics S44, combining the relationship features of all object pairs in the video frame t to obtain the relationship feature P of the video frame t t ; Combine the relational features of all video frames to obtain the relational features X2 of the video.

5. The method for generating a dynamic scene graph based on visual semantic reasoning according to claim 1, characterized in that: The spatial knowledge encoder includes multiple encoding layers; The spatial knowledge encoder processes the relational features and knowledge matrix including: S51, fusing the relational features of the video and the knowledge matrix, calculating the query vector and key vector of the first coding layer according to the fusion result, calculating the value vector of the first coding layer according to the relational features of the video, and calculating the output of the first coding layer according to the value vector, the query vector and the key vector; S52, inputting the output of the first coding layer into the second coding layer; S53, inputting the output of the k-1th coding layer into the kth coding layer; S54, repeat step S53 until the output of the last coding layer is obtained, that is, the spatial relationship features of the object pairs in the video; wherein k is the index of the coding layer.

6. The method for generating a dynamic scene graph based on visual semantic reasoning according to claim 1, characterized in that: The spatiotemporal relationship decoding module includes a time decoder; the spatiotemporal relationship decoding module processes the relationship features and spatial relationship features of object pairs, including: constructing an object pair trajectory feature sequence based on the object pair relationship feature X2, generating a temporal relationship mask matrix and a sequence mask matrix based on the object pair trajectory feature sequence, inputting the spatial relationship features, the temporal relationship mask matrix and the sequence mask matrix into the time decoder to obtain the spatiotemporal relationship features of the video.

7. The method for generating a dynamic scene graph based on visual semantic reasoning according to claim 1, characterized in that: Calculating visual feature differences includes: constructing a matrix based on the relationship features of object pairs in the video Save the relationship features of each pair of objects in the video, where N p , T and d model Represent the number of object pairs, the number of video frames, and the feature dimensions of the object pairs respectively; the matrix M u According to the second dimension, the difference in visual features M is obtained by subtracting the displacement. Fin .

8. The method for generating a dynamic scene graph based on visual semantic reasoning according to claim 1, characterized in that: The relationship types include: attention relationship a, spatial relationship s, and contact relationship c; calculating the semantic feature difference includes: using the pre-trained word vector model to process the relationship categories of the predicted object pairs in relationship types a, s, and c, obtain the semantic features of the object pairs in relationship types a, s, and c, and construct three semantic feature matrices Separately save the semantic features of each set of object pairs in relation types a, s, and c; where N p , T represent the number of object pairs and the number of video frames, respectively, N a 、N s 、N c Respectively represent the number of relationship categories of relationship types a, s, and c; the three matrices M o,a 、M o,s 、M o,c Subtract the second dimension by displacement to get the semantic feature difference 9. The method for generating a dynamic scene graph based on visual semantic reasoning according to claim 1, characterized in that: The loss function is: in, is the cross entropy loss of the object category, is the mean square error loss of the visual feature difference and semantic feature difference of the object pair in the relation type v, is the cross entropy loss of the relation type a of the object pair, and are the multi-label edge losses for the relationship types s and c of the object pairs, respectively, and λ and β are weight parameters.

10. The method for generating a dynamic scene graph based on visual semantic reasoning according to claim 9, characterized in that: The calculation process includes: Among them, L v represents the linear transformation function, T is the number of video frames, BCE is the mean square error loss function, M Fin is the difference in visual features, is the semantic feature difference of relation type v.

Images