A dynamic scene graph generation method based on visual semantic reasoning

Abstract

The application belongs to the field of dynamic scene graph generation in the sub-problem of scene graph generation, and particularly relates to a dynamic scene graph generation method based on visual semantic reasoning, comprising: acquiring a video, inputting the video into a trained dynamic scene graph generation model, obtaining object and relationship category between objects, and constructing a scene graph structure of the video according to the object and the relationship category between objects; the dynamic scene graph generation model comprises: a pre-trained object detector, a time relationship coding module, a relationship feature extraction module, a spatial knowledge encoder, a space-time relationship decoding module and a classification module; the application calculates the visual feature difference and the semantic feature difference between adjacent video frames and performs loss calculation, thereby providing a direction for correct prediction of the model; the application fuses object spatial information and time dependence by using a spatial encoder, and filters correct relationship labels by using prior knowledge, thereby improving the accuracy of relationship prediction.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of dynamic scene graph generation in the sub-problem of scene graph generation, and particularly relates to a dynamic scene graph generation method based on visual semantic reasoning. BACKGROUND

[0002] The scene graph generation task is a classic computer vision downstream task, and the purpose of dynamic scene graph generation is to identify objects in images or videos and infer the relationships between them. The dynamic scene graph generation system mainly includes three stages: object detection, temporal dependency extraction, and relationship identification. Generally, we consider the first stage to be an independent computer vision task, and the dynamic scene graph generation work mainly focuses on the last two stages, which can also be considered as a classification problem. Due to the movement of objects and the movement of the camera in the video, the objects may be blurred, resulting in false detection and missed detection of objects, and the objects cannot appear continuously in the video frames. In this case, it is difficult for the downstream task to correctly understand the semantic information in the video, so it is very important to extract the temporal dependency between objects for relationship identification.

[0003] However, video is an information-intensive media form, in which time and space information are severely intertwined, making it difficult to explicitly capture temporal dependencies from them, and as time goes on, objects in some frames appear blurred and occluded, etc., which cannot provide high-quality visual features, in addition, the relationship between objects will also change. Dynamic scene graph generation faces the following problems: due to the movement of objects in the video, the detected objects in the video frames will be affected by factors such as motion blur or occlusion, resulting in false detection or missed detection of the same object in different time points of the video frames, which cannot provide continuous high-quality feature information for the relationship detection stage; the movement of objects may mean that the relationship between objects will also change, and this small change in vision is difficult for the object detector to accurately extract.

[0004] Existing dynamic scene graph generation methods are mostly divided into two types according to the data segmentation granularity: video-level dynamic scene graph generation method and frame-level dynamic scene graph generation method. The video-level dynamic scene graph generation method divides the input video into multiple small video clips, and generates a scene graph for each video clip; the frame-level dynamic scene graph generation method divides the input video into a sequence of key video frames, and generates a scene graph for each key frame. The former needs to accurately cut long videos into short video clips (e.g. 10 frames) according to the precise time of the relationship, but it is very difficult to cut due to the ambiguity of the time relationship. In contrast, the latter provides a more flexible implementation mechanism for the relationship representation in continuous video frames. However, most of the current frame-level dynamic scene graph generation methods are still not ideal in terms of consistent detection of the same object between consecutive frames and dynamic relationship conversion of the same object pair.

[0005] Therefore, how to construct consistency in the video and accurately identify the change of the relationship between objects is an urgent problem to be solved. SUMMARY

[0006] To solve the above-mentioned prior art problems, the application adopts a dynamic scene graph generation method based on visual semantic reasoning, comprising: acquiring a video, inputting the video into a trained dynamic scene graph generation model to obtain object and relationship category between objects, and constructing a scene graph structure of the video according to the object and the relationship category between objects; the dynamic scene graph generation model comprises: a pre-trained object detector, a temporal relationship encoding module, a relationship feature extraction module, a spatial knowledge encoder, a spatio-temporal relationship decoding module and a classification module;

[0007] The training process of the dynamic scene graph generation model comprises:

[0008] S1, acquiring a training set, the training set comprising a plurality of training samples, each training sample comprising a video and prior knowledge thereof; each video comprising a plurality of video frames; constructing a knowledge matrix of object pairs according to the prior knowledge of the training set; the prior knowledge of the video comprising: object pairs in each video frame and their relationship category and relationship label in each relationship type;

[0009] S2, inputting each video frame of the video into the pre-trained object detector to obtain the detected objects of the video frame and object information thereof; the object information comprising: object features, object categories and object detection boxes;

[0010] S3, inputting the object information of the video frame into the temporal relationship encoding module to obtain the temporal object features Z1 of the video;

[0011] S4, inputting the object information of the video frame of the video and the temporal object features Z1 into the relationship feature extraction module to obtain the relationship features X2 of the object pairs of the video;

[0012] S5, inputting the relationship features X2 and the knowledge matrix into the spatial knowledge encoder to obtain the spatial relationship features Z2 of the object pairs of the video;

[0013] S6, inputting the relationship features X2 and the spatial relationship features Z2 of the object pairs of the video into the spatio-temporal relationship decoding module to obtain the spatio-temporal relationship features Z Fin of the object pairs of the video;

[0014] S7, inputting the spatio-temporal relationship features Z Fin and the temporal object features Z1 into the classification module to obtain the predicted probability distribution vector P v of the object pairs in each relationship type and the relationship category in each relationship type and the class probability distribution vector P O of the predicted objects; v is the index of the relationship type;

[0015] S8, calculate the visual feature difference and the semantic feature difference according to the relationship feature of the object pair of the video and the predicted relationship category of the object pair, calculate the visual feature difference and the semantic feature difference according to the visual feature difference, the semantic feature difference, the probability distribution vector P v And the probability distribution vector P O Calculate the loss function value, update the dynamic scene graph generation model parameter according to the loss function value, and obtain the trained dynamic scene graph generation model when the loss function value is minimum.

[0016] Advantages:

[0017] 1、The visual feature difference and the semantic feature difference between adjacent video frames are calculated, the visual feature difference and the semantic feature difference are loss calculated, the difference of the semantic feature is used as the guide of the visual feature difference, and the direction for correct prediction of the model is provided; 2、The relationship feature X2 is input into the spatial knowledge encoder to fuse object spatial information and time dependence, and the false detection and missed detection problems of the object detector are corrected; 3、The relationship feature is input into the spatial knowledge encoder, and correct relationship labels are screened by using prior knowledge, and the accuracy of relationship prediction is improved; 4、The time decoder has two self-attention mechanisms, one of which does not allow the relationship of the future object pair to affect the calculation of the current relationship feature through the time relationship mask, and the relationship prediction quality is improved. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 A flowchart of a dynamic scene graph generation method based on visual semantic reasoning provided for an embodiment of the present application

[0019] Figure 2 A prior knowledge extraction flowchart provided for an embodiment of the present application

[0020] Figure 3 A flowchart of object trajectory extraction provided for an embodiment of the present application

[0021] Figure 4 An object trajectory encoder network structure schematic diagram provided for an embodiment of the present application

[0022] Figure 5 A flowchart of calculating the visual feature difference and the semantic feature difference of the relationship feature provided for an embodiment of the present application

[0023] Figure 6 A spatial knowledge encoder network structure schematic diagram provided for an embodiment of the present application

[0024] Figure 7 A time decoder network structure schematic diagram provided for an embodiment of the present application

[0025] Figure 8This is a schematic diagram illustrating the result of generating a video relationship scene graph according to an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] like Figure 1 As shown, the present invention employs a dynamic scene graph generation method based on visual semantic reasoning, including: acquiring a video; inputting the video into a trained dynamic scene graph generation model to obtain objects and the relationship categories between objects; and constructing a scene graph structure of the video based on the objects and the relationship categories between objects. The dynamic scene graph generation model includes: a pre-trained object detector, a temporal relationship encoding module, a relationship feature extraction module, a spatial knowledge encoder, a spatiotemporal relationship decoding module, and a classification module.

[0028] The training process of the dynamic scene graph generation model includes:

[0029] S1. Obtain the training set, which includes multiple training samples. Each training sample includes a video and its prior knowledge. Each video includes multiple video frames. Construct a knowledge matrix of object pairs based on the prior knowledge of the training set.

[0030] The dataset used in this invention is the Action Genome dataset, which contains 36 object categories, 26 relation labels, 476,229 object boxes, and 135,848 object pairs. The relation categories are divided into three different types: Attention, Spatial, and Contacting. The Action Genome dataset contains 7,464 videos with 166,785 frames for training and 1,737 videos with 54,371 frames for testing. Each video's frame sequence is V = [I1, I2, ..., I...]. T ], where I t For each video frame, T represents the number of video frames in the video.

[0031] The prior knowledge for each video includes the objects in each video frame, their object categories and object frames, as well as the object pairs in each video frame and their relation categories and relation labels in each relation type; wherein, the object pair in a video frame is a combination of any two objects in the video frame.

[0032] Attention relation represents whether the object's eyes focus on a certain object, including 3 relation labels; contact relation represents whether the object contacts a certain object, including 17 relation labels; spatial relation represents the position of the object relative to another object, including 6 relation labels.

[0033] As shown in Figure 2 , constructing a knowledge matrix according to prior knowledge of the training set includes:

[0034] S11, according to the relation label of the object pair in each relation type (26 relation labels in total for 3 types), a relation vector of each group of object pairs in each video frame in the training set is constructed, with a dimension of R 1×26 ;

[0035] Specifically, as shown in a frame, there are o1, o2 and o3, and then the objects are combined in pairs, so there are object pairs (o1, o2), (o1, o3) and (o2, o2).

[0036] An array of 26 elements records all the relation labels of the object pair (o1, o2), and subscripts 0-2 represent attention relations, subscripts 3-8 represent spatial relations, and subscripts 9-25 represent contact relations. For example, in the first video frame, (o1, o2) appears, the relation category of (o1, o2) in the attention relation a is a1, the relation category in the spatial relation s is s1, and there is no corresponding relation label in the contact relation; a1 occupies subscript 1, s1 occupies subscript 4, and since there is no relation label in the contact relation, subscripts 9 to 25 of the relation vector are all 0, so the relation vector of (o1, o2) in the first video frame is [0, 1, 0, 0, 1, 0, 0, 0, 0, …, 0];

[0037] S12, in all video frames of all videos in the training set, the relation vectors belonging to the same group of object pairs are accumulated to obtain the accumulated relation vector of each group of object pairs;

[0038] S13, according to the accumulated relation vector of each group of object pairs, the distribution probability of the relation label of each group of object pairs in the corresponding relation type is calculated to obtain the relation knowledge vector K of each group of object pairs;

[0039] Specifically, calculating the distribution probability of the relation label in the relation type a includes: there are N a relation labels in the relation type a, respectively , then the probability distribution of the object pair in the relation type a is , and the same in the relation type s and the relation type c.

[0040] The probability distribution of each group of object pairs in the three relation categories is spliced to obtain the relation knowledge vector K of each group of object pairs, specifically:

[0041]

[0042] wherein, respectively represent the distribution probability of the relationship label in the relationship type a, s, c for the object pair, Concat() represents the concatenation operation, N r = 26 represents the number of relationship labels.

[0043] S14, superimpose the relationship knowledge vectors of all object pairs to obtain the knowledge matrix of the object pair wherein, N p is the number of object pairs;

[0044] S2, input each video frame of the video into a pre-trained object detector to obtain the object and the object information of each video frame;

[0045] Each video frame I t The detected object information includes: object features object category and object detection box wherein, N t is the number of detected objects.

[0046] S3, input the object information of the video frame into a time relationship encoding module to obtain the time object features of the video;

[0047] The object detection box B t includes coordinate information E t ; The time relationship encoding module includes: an object trajectory extraction module, a position coding embedding module and an object trajectory encoder;

[0048] As shown in the figure, Figure 4 the processing process of the time relationship encoding module includes:

[0049] S31, input the object features of the video frame into the object trajectory extraction module to obtain the object trajectory feature matrix F o ;

[0050] As shown in the figure, Figure 3 the object trajectory extraction module extracts the object trajectory feature matrix includes:

[0051] In the object information of the video frame, the object trajectory features of all object categories are constructed s l = {s l,1 , s l,2 , …, s l,T}, wherein s l is the object trajectory feature of the object category l, sl,t representing the features of the object belonging to the object class l in the video frame t, N u representing the number of object classes 36.

[0052] The object trajectory feature of the object class l is the trajectory feature of the object belonging to the object class l in the video frame. If there are multiple objects of the same object class in a video frame, the trajectory feature of the object with the highest confidence is selected as the trajectory feature of the object class. The object information obtained in step S2 further includes the detected object class L t and the confidence of the object, which refers to the matching score of the object and the object class.

[0053] The object trajectory features of each object class are superimposed to obtain the object trajectory feature matrix F o , specifically:

[0054]

[0055] wherein σ represents a function of filling a variable list with a filling value, and the object trajectory features with different lengths are aligned (the empty positions are filled with 0) and superimposed together.

[0056] S32, input the object trajectory feature matrix F o and the coordinate information of the object detection frame into a position encoding embedding module to obtain the position encoding feature X1:

[0057] X1=κ(F o ,W f (E))

[0058] wherein κ represents a position encoding embedding function, W g represents a linear transformation function, and E represents the coordinate information of the object detection frame.

[0059] S33, input the position encoding feature X1 into an object trajectory encoder to obtain the temporal object feature of the video wherein N A represents the number of detected objects in the entire video, and d model represents the dimension of each detected object feature.

[0060] The object trajectory feature matrix and the coordinate information of the object detection frame containing the spatial position information of the object are input into the object trajectory encoder after position encoding, in order to extract the temporal dependence of the object between multiple frames, correct and supplement the objects missed or misdetected by the object detector in step S2, and provide support for forming correct relationship object pairs later.

[0061] The object trajectory encoder adopts a standard multi-head attention mechanism-based encoder, which is composed of n Transform encoder layers stacked together, and each Transform encoder layer is composed of a multi-head attention mechanism, a residual connection and a normalization layer (Add&Norm), and a feedforward neural network (FNN). The feedforward network (FNN) is composed of a linear layer Linear, a RELU activation layer, and a random dropout layer Dropout.

[0062] Specifically, X1 is input into the first layer of the encoding layer, the output of the first layer of the encoding layer is input into the second layer of the encoding layer, the output of the k-1 layer of the encoding layer is input into the k layer of the encoding layer, k is the index of the encoding layer, and the object trajectory feature Z1 containing time-dependent information obtained after the n layers of the encoder is represented as:

[0063]

[0064] Z1=LayerNorm(Z″1+FFN(Z″1))

[0065] wherein Z'1 is the result of the multi-head attention mechanism, Z''1 is the output result of the residual connection and the normalization layer, is the output of the n-1 layer of the encoding layer; the object trajectory feature Z1 containing time-dependent information is unfolded in time sequence in units of frames to obtain the time object feature of each video frame t wherein N A represents the number of objects detected in the entire video, d model represents the feature dimension of each object, M O represents the padding mask, i.e., the position of the padding value, which is constructed from the object trajectory feature F o , so as to avoid the object feature at the position of the padding value from participating in the calculation of the attention mechanism in the encoder.

[0066] S4, input the object information and the time object feature of the video frame of the video into a relationship feature extraction module to obtain the relationship feature of the object pair of the video;

[0067] The relationship feature extraction module calculates the relationship feature of the object pair of the video, which includes:

[0068] S41, process the object class of each object i detected in the video frame t by using a word vector model to obtain the object semantic feature

[0069]

[0070] Here, Glove represents the pre-trained word embedding model Glove-200d, which maps object categories to 200-dimensional semantic embedding vectors. Indicates the object category of object i.

[0071] S42. Combine the objects detected in video frame t in pairs to obtain multiple sets of object pairs.<i,j> ; based on the object detection boxes of objects i and j Computational object pairs<i,j> Union box features Object detection box The detection box features are concatenated with the combined bounding box features to obtain the detection box features.

[0072]

[0073] S43, Pair objects<i,j> Time object characteristics Detection box features and object semantic features Combine them to obtain object pairs<i,j> Relationship characteristics

[0074]

[0075] Where t = [1, 2, ..., T] represents frame t in the video, i, j, k ∈ O. t i≠j≠k, where i is the index of the object, O t It refers to all objects detected in frame t, P t Let the object pair feature in frame t be denoted as . Concat(,) represents a concatenation operation. This indicates a flattening operation. W represents the addition of corresponding elements. s W o W u Represents a linear transformation matrix. h represents the feature of the union box calculated by RoIAlign. box This means changing the bounding boxes of objects i and j to those of objects j. Features with the same dimensions

[0076] The joint box features for each pair of objects are calculated using RoIAlign region alignment. This includes: extracting the bounding box regions where both objects in an object pair exist simultaneously, which are called Union Boxes.

[0077] S44. Combine the relational features of all object pairs in video frame t to obtain the relational feature P of video frame t.t combining the relational features of all video frames to obtain the relational features X2 of the object pairs of the video;

[0078] Since the number of object pairs obtained in each video frame is not exactly the same, when combining, the relational features of each video frame need to be padded with invalid values so that the shape of the relational features of each video frame is the same.

[0079] S5, inputting the relational features of the object pairs of the video and the knowledge matrix into a spatial knowledge encoder to obtain the spatial relational features of the object pairs of the video;

[0080] As shown in Figure 6 , the processing process of the spatial knowledge encoder includes:

[0081] The spatial knowledge encoder adopts a standard encoder based on a multi-head attention mechanism, which is composed of n TransformerEncoder Layer layers; the TransformerEncoder Layer layer is composed of Multi-Head Attention, Add&Norm and FNN.

[0082] In the spatial knowledge encoder, X2 and X are input into the first encoding layer, the query vector Q and the key vector K of the first encoding layer are obtained by linear transformation after X2 fusing the knowledge matrix X of the object pairs, and the value vector V of the first layer is obtained by linear transformation of the same dimension of X2, and specifically:

[0083]

[0084] wherein, W q ,W k ,W v represents a linear transformation matrix, represents the addition of corresponding elements.

[0085] The output of the first encoding layer is input into the second encoding layer, the output of the k-1th encoding layer is input into the kth encoding layer, and k is the index of the encoding layer; the spatial relational features Z2 obtained after n layers of encoder are represented as:

[0086]

[0087] Z2=LayerNorm(Z″2+FFN(Z″2))

[0088] wherein, Z′2 is the result of the multi-head attention mechanism, Z″2 is the output result of the residual connection and the normalization layer, is the output of the n-1th encoding layer,

[0089] The input of the relationship feature into the spatial knowledge encoder can not only extract spatial information in the relationship feature, but also limit the relationship selection of the object pair by using the prior knowledge of the relationship, so as to eliminate impossible relationship labels and make the relationship probability distribution on possible relationship labels, which will be beneficial to the selection of correct relationship labels.

[0090] S6, input the relationship feature X2 of the object pair of the video and the spatial relationship feature of the object pair into a space-time relationship decoding module to obtain the space-time relationship feature of the object pair of the video; the space-time relationship decoding module comprises a time decoder;

[0091] The specific process of the space-time relationship decoding module comprises: constructing an object pair trajectory feature sequence according to the relationship feature X2 of the object pair, generating a time relationship mask matrix and a sequence mask matrix according to the object pair trajectory feature sequence and the sequence mask matrix; inputting the spatial relationship feature, the time relationship mask matrix and the sequence mask matrix into the time decoder to obtain the space-time relationship feature of the video.

[0092] The construction of the object pair trajectory feature sequence comprises: finding the object pair <i,j> in the relationship feature X2 of the object pair, and splicing the relationship feature of the object pair <i,j> in each frame to obtain the object pair trajectory feature with time information.

[0093] The time relationship mask matrix and the sequence mask matrix are both the features of the object pair in the object pair trajectory feature sequence, the values on the diagonal lines of the time relationship mask matrix and the sequence mask matrix are all 1, and when the object pair p t b appears before the object pair p t d in the object pair trajectory feature sequence, the position (p t b, p t d) of the time relationship mask matrix is 1, and the rest positions are all 0; when the object pair p t b appears before the object pair p t d and b is equal to d, the position (p t b, p t d) of the sequence mask matrix is 1, and the rest positions are all 0; wherein p t b is the feature of the object pair b in the video frame t.

[0094] As shown in Figure 7 , the process of the time decoder comprises:

[0095] The time decoder adopts a standard decoder based on a multi-head attention mechanism, which is stacked by n TransformerDecoder Layer layers, and each TransformerDecoder Layer layer is composed of Multi-HeadAttention, Add&Norm and FNN.

[0096] In the temporal decoder, specifically, Z2 is input into the first encoding layer, the output of the first encoding layer is input into the second encoding layer, the output of the k-1th encoding layer is input into the kth encoding layer, k is the index of the encoding layer; the output Z3 obtained after Z2 passes through the n-layer decoder is represented as:

[0097] X'2 = MultiHeadAtt(Q = W q Z2 (n-1) ,K = W k Z2 (n-1) ,V = W v Z2 (n-1) ,mask = M P )

[0098] Z'3 = MultiHeadAtt(Q = W q X'2,K = W k X'2,V = W v X'2,mask = M S 0

[0099] Z"3 = LayerNorm(Z'3 + X'2)

[0100] Z3 = LayerNorm(Z"3 + FFN(Z"3))

[0101] wherein Z'3 is the result of the multi-head attention mechanism, Z"3 is the output result of the residual connection and normalization layer, Z2 (n-1) is the output of the n-1th encoding layer, is the spatio-temporal relationship feature, Z3 is simply denoted as Z Fin , N p represents the number of object pairs detected in the entire video, d model represents the dimension of each relationship feature, M P represents the temporal relationship mask matrix, M S represents the sequence mask matrix, which acts on each relationship feature to prevent the decoder from seeing future information.

[0102] There are two self-attention mechanisms in the temporal decoder, the first one is to calculate the relationship features of the same object pairs, and the temporal relationship mask M P is used to prevent the relationship generated by future object pairs from affecting the calculation of the current relationship feature, the second one is the self-attention mechanism in the classic decoder, which uses the sequence mask M S to generate the relationship label of each object pair in turn.

[0103] In one embodiment, the Transformer Encoder Layer structure of the time decoder includes: a first layer MultiheadAttention, the input and output channel number is 1936; a second layer Dropout, the input and output channel number is 1936; a third layer MultiheadAttention, the input channel number is 1936; a fourth layer Dropout, the input and output channel number is 1936; a fifth layer Layer Norm, the input channel number is 1936; a sixth layer Linear, the input and output channel number is 1936 and 2048 respectively; a seventh layer RELU, the input and output channel number is 2048; an eighth layer Dropout, the input and output channel number is 2048; a ninth layer Linear, the input and output channel number is 2048 and 1936 respectively; a tenth layer Dropout, the input and output channel number is 1936;

[0104] S7, input the spatio-temporal relationship features and the temporal object features Z1 of the object pair into the classification module to obtain a predicted probability distribution vector of the object pair in each relationship type and a relationship class in each relationship type and a predicted class probability distribution vector P of the object O ;

[0105] The classification includes: performing three types of relationship label classification on the spatio-temporal relationship features Z Fin of the object pair, classifying the temporal object features Z1, and specifically:

[0106]

[0107] wherein Pa, Ps and Pc are respectively probability distribution vectors of the object pair in the three types of relationship labels, P O is a predicted class probability distribution vector of the object, η a , η s , η c , η O represents a linear mapping function, N a , N s , N c respectively represent the number of relationship labels in the three types of relationship, N O is the number of object class labels, and the sigmoid function represents mapping the output to the range of 0 to 1.

[0108] S8, calculate the visual feature difference and the semantic feature difference of the relationship features of the video according to the object information of the video frames of the video, calculate the loss function value according to the visual feature difference, the semantic feature difference, the object relationship class prediction result and the real label of the video, update the dynamic scene graph generation model parameter according to the loss function value, and obtain the trained dynamic scene graph generation model when the loss function value is the minimum.

[0109] The visual feature difference of the video is calculated, including:

[0110] A matrix is constructed according to the relationship features of the video The relationship features of each group of object pairs in the video are saved; wherein, N p , T and d model respectively represent the number of object pairs, the number of video frames and the dimension of the relationship features of the object pairs;

[0111] The matrix M u is subtracted according to the second dimension, which represents the visual feature of the object pair in the next frame minus the visual feature of the same object pair in the previous frame, specifically as follows:

[0112]

[0113] There are three types of relationships between each object pair in the data set, so the semantic class difference needs to be calculated separately for each relationship type of each object pair.

[0114] As Figure 5 shown, the semantic feature difference of the video is calculated, including:

[0115] The predicted relationship class of the object pair in the relationship types a, s and c is processed by using the pre-trained word vector model, to obtain the semantic feature of the object pair in the relationship types a, s and c:

[0116] When the relationship class is composed of phrases, the word vector of each word is calculated and added to obtain the final semantic feature, specifically:

[0117]

[0118] wherein, represents the relationship class of the three relationship types, represents a phrase segmentation function, Glove represents a pre-trained Glove-200d model, which maps the relationship class to a 200-dimensional word embedding vector, represents the final semantic feature obtained by adding the word vectors of multiple words in the three relationship types.

[0119] Three semantic feature matrices are constructed The semantic features of each group of object pairs in the relationship types a, s and c are saved respectively; wherein N p , T respectively represent the number of object pairs, the number of video frames, N a , N s , N c respectively represent the number of relationship classes of the relationship types a, s and c;

[0120] Three matrices M o,a , M o,s , M o,c are subtracted in the second dimension, representing the semantic features of the object in the next frame minus the semantic features of the same object in the previous frame:

[0121]

[0122] The calculation process of the loss function is as follows:

[0123] The cross-entropy loss is used to calculate the difference between the predicted object class probability distribution vector P O and the real class label T O of the object, and the formula is:

[0124]

[0125] where Linear, BatchNorm and ReLU represent linear transformation function, batch normalization function and nonlinear activation function respectively, i is the index of the object, and N A is the number of objects.

[0126] The mean square error loss is used to calculate the difference between the visual feature difference and the semantic feature difference of the object pair in the relationship types a, s and c Loss diff,a , Loss diff,s , Loss diff,c , and the formula is

[0127]

[0128] where L a , L s , L c represent linear transformation functions respectively.

[0129] The cross-entropy loss is used to calculate the difference between the predicted distribution P a and the real class label vector T a of the relationship type a , and the formula is:

[0130]

[0131] where i is the index of the relationship class of the object pair in the relationship type a.

[0132] The multi-label margin loss is used to calculate the difference between the predicted distribution P s and the real class label vector T s of the relationship type s , and the formula is:

[0133]

[0134] where i, j are indexes of the real class label vector T s .

[0135] The multi-label margin loss is used to calculate the prediction distribution P c of the relationship type c c and the real class label vector T The formula is as follows:

[0136]

[0137] where i, j are indexes of the real class label vector T c .

[0138] The total loss L of the model is as follows:

[0139]

[0140] where, is the cross-entropy loss CE of the object class, is the mean square error loss BCE of the visual feature difference and the semantic feature difference, is the cross-entropy loss of the relationship type a, and are the multi-label margin losses of the relationship types s and c respectively.

[0141] The application further provides a dynamic scene graph generation system based on visual semantic reasoning, comprising a visual interface, a video input module, a priori knowledge extraction module, an object detection module, a time relationship coding module, a spatial knowledge coding module, a space-time relationship decoding module and a relationship classification module.

[0142] The video input module is used for inputting a video.

[0143] The priori knowledge extraction module is used for counting all relationship labels existing in each relationship category in a training set, so that the range of selectable relationship labels of each relationship category can be reduced in the encoder, and the matching probability of correct relationship labels can be improved.

[0144] The object detection module is used for detecting objects and related information in a video frame.

[0145] The time relationship coding module is used for combining objects of the same object class together and sending the objects into the encoder, so that the time dependence of the objects between adjacent frames can be coded in the object features.

[0146] The spatial knowledge coding module is used for coding the spatial position relationship of object pairs in a video frame in the relationship features.

[0147] The space-time relationship decoding module is configured to extract the time-dependent relationship of the same object pair, and decode the space-time relationship features of each object pair in time sequence from the time-dependent relationship;

[0148] The relationship classification module is configured to classify the space-time relationship features, and output relationship labels of three different categories.

[0149] As shown in Figure 8 The visualization interface includes a weight path selection interface, a video path selection interface, and a relationship graph structure display interface of video generation. The weight path selection interface is configured to locally select a dynamic scene graph generation model. The video path selection interface is configured to select a video. The relationship graph structure display interface of video generation is configured to display a scene graph structure output by the dynamic scene graph generation model. The scene graph structure includes objects: person, bed, laptop, and relationships between objects: behind, touching, lying on, not looking at, and looking at.

[0150] The dynamic scene graph generation based on visual semantic reasoning can be applied to tasks that require video analysis, understanding, and early warning. For example, in the medical field, video scene graph generation can be applied to dynamic medical image analysis, such as heart ultrasound or abdominal examination video. It can help doctors automatically label and analyze key objects (such as organs, tissues, etc.) and their relationships in medical videos, thereby improving diagnostic accuracy.

[0151] The above examples further illustrate the purpose, technical solutions, and advantages of the present application. It should be understood that the above examples are only preferred embodiments of the present application and do not limit the present application. Any modifications, equivalent replacements, improvements, etc. made to the present application within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A dynamic scene graph generation method based on visual semantic reasoning, characterized in that, The method comprises the following steps: acquiring a video, inputting the video into a trained dynamic scene graph generation model to obtain objects and relationship categories between the objects, and constructing a scene graph structure of the video according to the objects and the relationship categories between the objects; the dynamic scene graph generation model comprises a pre-trained object detector, a temporal relationship encoding module, a relationship feature extraction module, a spatial knowledge encoder, a spatio-temporal relationship decoding module and a classification module; the training process of the dynamic scene graph generation model comprises the following steps: S1, acquiring a training set, the training set comprising a plurality of training samples, each training sample comprising a video and prior knowledge thereof; each video comprising a plurality of video frames; constructing a knowledge matrix of object pairs according to the prior knowledge of the training set; the prior knowledge of the video comprising object pairs in each video frame and relationship categories and relationship labels of the object pairs in each relationship type; S2, inputting each video frame of the video into the pre-trained object detector to obtain detected objects of the video frame and object information of the objects; the object information comprising object features, object categories and object detection boxes; S3, inputting the object information of the video frame into the temporal relationship encoding module to obtain temporal object features Z1 of the video; S4, inputting the object information of the video frame and the temporal object features Z1 of the video into the relationship feature extraction module to obtain relationship features X2 of object pairs of the video; S5, inputting the relationship features X2 and the knowledge matrix into the spatial knowledge encoder to obtain spatial relationship features Z2 of the object pairs of the video; S6, input the relationship feature X2 and the spatial relationship feature Z2 of the object pair of the video into the space-time relationship decoding module to obtain the space-time relationship feature Z of the object pair of the video Fin ; S7, the spatio-temporal relation feature Z Fin and the time object feature Z1 into the classification module to obtain a predicted probability distribution vector P of the object pair in each relation type v and the relation class and the predicted class probability distribution vector P of the object O ; v is the index of the relation type; S8, calculate the visual feature difference and the semantic feature difference according to the relationship feature X2 of the object pair of the video and the predicted relationship category of the object pair, calculate the loss function value according to the visual feature difference, the semantic feature difference and the probability distribution vector P of the video v and the probability distribution vector P O calculate the loss function value, update the dynamic scene graph generation model parameter according to the loss function value, and obtain the trained dynamic scene graph generation model when the loss function value is the minimum. 2.The dynamic scene graph generation method based on visual semantic reasoning according to claim 1, wherein, constructing the knowledge matrix according to the prior knowledge of the training set comprises the following steps: S11, constructing relationship vectors of each group of object pairs in each video frame of the training set according to the relationship labels of the object pairs in each relationship type; S12, in all video frames of all videos of the training set, accumulating the relationship vectors of the same group of object pairs to obtain accumulated relationship vectors of each group of object pairs; S13, calculating the distribution probability of the relationship label of each group of object pairs in the corresponding relationship type according to the accumulated relationship vectors of each group of object pairs to obtain relationship knowledge vectors of each group of object pairs; S14, superimposing the relationship knowledge vectors of all object pairs to obtain the knowledge matrix X of the object pairs. 3.The method of claim 1, wherein, The temporal relationship encoding module comprises an object trajectory extraction module, a position encoding embedding module and an object trajectory encoder; the processing of the object information of the video frame by the temporal relationship encoding module comprises the following steps: S31, input the object feature of the video frame into the object trajectory extraction module to obtain an object trajectory feature matrix F o ; S32, inputting the object track feature matrix F o and the coordinate information of the object detection frame into a position encoding embedding module to obtain a position encoding feature X1; S33, inputting the position encoding features X1 into the object trajectory encoder to obtain the temporal object features of the video.

4. The method of claim 1, wherein, The processing of the object information of the video frame and the temporal object features by the relationship feature extraction module comprises the following steps: S41, utilizing the word vector model to obtain the object class of each object i detected in the video frame t processing to obtain the object semantic feature S42, combine the detected objects in the video frame t two by two to obtain a plurality of object pairs <i,j>; according to the object detection boxes of the objects i and j calculate the joint box feature of the object pair <i,j> splice the object detection boxes and the joint box feature to obtain the detection box feature S43, combining the temporal object features of the object pair <i,j> bounding box features and the object semantic features to obtain the relationship features of the object pair <i,j> S44, combine the relationship features of all object pairs in the video frame t to obtain the relationship feature P of the video frame t t ; combine the relationship features of all video frames to obtain the relationship feature X2 of the video.

5. The method of claim 1, wherein, the spatial knowledge encoder comprises a plurality of encoding layers; the processing of the relationship features and the knowledge matrix by the spatial knowledge encoder comprises the following steps: S51, fusing the relationship features of the video and the knowledge matrix, calculating a query vector and a key vector of a first encoding layer according to the fused result, calculating a value vector of the first encoding layer according to the relationship features of the video, and calculating an output of the first encoding layer according to the value vector, the query vector and the key vector; S52, inputting the output of the first encoding layer into a second encoding layer; S53, inputting the output of the k-1th encoding layer into the kth encoding layer; S54, repeating step S53 until the output of the last encoding layer, i.e. the spatial relationship feature of the object pair of the video, is obtained; wherein k is the index of the encoding layer. 6.The method of claim 1, wherein, The space-time relationship decoding module comprises a time decoder; the processing of the relationship feature and the spatial relationship feature of the object pair by the space-time relationship decoding module comprises: constructing an object pair trajectory feature sequence according to the relationship feature X2 of the object pair, generating a time relationship mask matrix and a sequence mask matrix according to the object pair trajectory feature sequence, inputting the spatial relationship feature, the time relationship mask matrix and the sequence mask matrix into the time decoder, and obtaining the space-time relationship feature of the video.

7. The method of claim 1, wherein the method further comprises: The computing the visual feature difference comprises: constructing a matrix according to the relationship feature of the object pair of the video Saving the relationship feature of each group of object pairs in the video, wherein, N p , T and d model respectively represent the number of object pairs, the number of video frames and the feature dimension of the object pairs; subtracting the matrix M u according to the second dimension to obtain the visual feature difference M Fin . 8.The method of claim 1, wherein, The relationship types include an attention relationship a, a space relationship s, and a connection relationship c; the semantic feature difference is calculated by processing the predicted object pair in the relationship categories of the relationship types a, s, and c by using a pre-trained word vector model to obtain semantic features of the object pair in the relationship types a, s, and c, and three semantic feature matrices are constructed The semantic features of each group of object pairs in the relationship types a, s, and c are saved respectively; wherein N p , T represent the number of object pairs and the number of video frames respectively, N a , N s , N c represent the number of relationship categories of the relationship types a, s, and c respectively; the three matrices M o,a , M o,s , M o,c are subtracted by staggering the second dimension to obtain the semantic feature difference 9. The method of claim 1, wherein the method further comprises: The loss function is: wherein, is a cross-entropy loss for the object class, is a mean square error loss for the visual feature difference and the semantic feature difference of the object pair in the relation type v, is a cross-entropy loss for the relation type a of the object pair, and are multi-label margin losses for the relation types s and c of the object pair, respectively, and λ, β are weight parameters.

10. The method of claim 9, wherein the method further comprises: The calculation process includes: wherein, L v represents a linear transformation function, T is the number of frames of the video, BCE is a mean square error loss function, M Fin is the visual feature difference, is the semantic feature difference of the relationship type v.

Images