An online video visual relationship detection method based on multi-modal information fusion

Abstract

The present application relates to the technical field of artificial intelligence understanding and environment interaction, and particularly relates to an online video visual relationship detection method based on multi-modal information fusion, which proposes an ASE-GCN layer, uses three different relevant adjacency matrices to model a latent relationship graph, and the matrices contain visual appearance, physical space and semantic embedding information, extends a fusion method based on bilinear pooling to a fusion method based on trilinear pooling, and designs a hierarchical fusion module to fuse multi-role multi-modal features.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence understanding and environment interaction, and particularly relates to an online video visual relationship detection method based on multi-modal information fusion. BACKGROUND

[0002] With the progress of computer vision recognition and natural language processing technology, the video visual relationship detection task has gradually become a research hotspot. The purpose of the visual relationship detection task of online video visual relationship detection is to detect the interaction between objects in the video, which is the basis of artificial intelligence understanding and environment interaction.

[0003] The multi-modal information commonly used in the video visual relationship detection (VidVRD) method includes visual appearance clues, spatial position clues and semantic embedding clues. At present, the video visual relationship detection mainly adopts a feature-level fusion method when processing multi-modal information, and a few adopts a decision-level fusion method. However, the focus is mainly on the acquisition of a certain feature information, and the exploration of the multi-modal information fusion method is lacking. Only the features of the object pair and the predicate are spliced together, the potential association between the object pair and the predicate is not fully explored, and the method of directly splicing the multi-modal features does not fully utilize the complementarity between the multi-modal information, resulting in weak expression ability of the visual relationship representation and low accuracy of the video visual relationship detection. SUMMARY

[0004] The technical problem of the present application is to overcome the shortcomings of the prior art and provide an online video visual relationship detection method based on multi-modal information fusion.

[0005] The technical solution of the present application is:

[0006] An online video visual relationship detection method based on multi-modal information fusion, the steps of the method comprising:

[0007] Step 1, a hierarchical multi-modal fusion network is constructed, the constructed hierarchical multi-modal fusion network comprising two layers of ASE-GCN layers, a hierarchical multi-modal fusion module, a YOLO target detector and a semantic embedding layer, the hierarchical multi-modal fusion module comprising a first layer of attention-based fusion module and a second layer of three-linear pooling fusion module;

[0008] Step 2, inputting online video data into the hierarchical multi-modal fusion network constructed in step 1;

[0009] Step 3, the online video data input into the hierarchical multi-modal fusion network in step 2 extracts the initial visual features of each object through a deep neural network, and inputs the extracted initial visual features of each object into the two layers of ASE-GCN layers to obtain visual appearance clues;

[0010] Step 4, the frames of the online video data input into the hierarchical multi-modal fusion network in step 2 are input into the YOLO target detector, and according to the object bounding box coordinates expressed by each object generated by the YOLO target detector, static information and action representation are constructed, the static information is a physical space clue of the object pair, and the action representation is a physical space clue of the predicate, and the physical space clue of the object pair and the physical space clue of the predicate are spliced to obtain a spliced physical space clue;

[0011] Step 5, the frames of the online video data input into the hierarchical multi-modal fusion network in step 2 are input into the YOLO target detector, and the category of each target generated by the YOLO target detector is modeled by a semantic embedding layer to model the co-occurrence attribute of the object pair and the predicate, the object category is mapped to a unique corresponding hidden representation in a hidden space, and the hidden representations are connected to obtain a semantic embedding clue for relationship prediction;

[0012] Step 6: input the visual appearance clue obtained in step 3, the spliced physical space clue obtained in step 4, and the semantic embedding clue obtained in step 5 into the hierarchical multi-modal information fusion module, pass through the first layer of attention-based fusion module, and perform multi-role fusion on the objects and the relationships between the objects in the visual appearance clue, the objects and the relationships between the objects in the physical space clue, and the objects and the relationships between the objects in the semantic embedding clue, and finally pass through the second layer of three-linear pooling fusion module to perform multi-modal fusion on the visual appearance clue, the physical space clue, and the semantic embedding clue after respective fusion, to obtain a total fusion result;

[0013] Step 7: detect the interaction relationship between the object pairs in the total fusion result obtained in step 6, the interaction relationship includes up, down, left, right, close, far away, and follow, to obtain an online relationship detection result, and complete the online video visual relationship detection based on multi-modal information fusion.

[0014] In step 1, the two layers of ASE-GCN layers in the hierarchical multi-modal fusion network are a first ASE-GCN layer and a second ASE-GCN layer, and the first ASE-GCN layer and the second ASE-GCN layer are stacked together;

[0015] The ASE-GCN layers respectively explore the associativity between objects from the perspectives of visual appearance, physical space, and semantic prior, to capture all possible potential interaction relationships;

[0016] Each layer of ASE-GCN layers includes a visual appearance related adjacency matrix, a physical space related adjacency matrix, and a semantic prior related adjacency matrix, and the visual appearance related adjacency matrix, the physical space related adjacency matrix, and the semantic prior related adjacency matrix determine the adjacency state and information update weight between nodes;

[0017] The visual appearance related adjacency matrix is obtained by filling the similarity between objects into the corresponding position in the matrix and normalizing, and the similarity between object i and object j is:

[0018]

[0019] Where x i is the visual appearance feature of object i, and x j is the visual appearance feature of object j.

[0020] Fill the similarity value similarity(x i , x j ) into the corresponding position (i _loc , j _loc ) in the matrix, where the i _loc value is equal to i, and the j _loc value is equal to j, and then normalize to obtain the visual appearance related adjacency matrix.

[0021] The physical space related adjacency matrix is obtained by filling the spatial position correlation between object i and object j into the corresponding position in the matrix and normalizing, and the spatial position correlation is:

[0022]

[0023] Where bbox i and bbox j represent the bounding boxes of object i and object j, I(·) represents the intersection area between bbox i and bbox j , and U(·) represents the union area between bbox i and bbox j . The stronger the spatial position correlation between the two objects, the greater the information transmission weight between the two objects.

[0024] The features of each node in the first layer are updated according to the adjacent nodes determined by the visual appearance related adjacency matrix, the physical space related adjacency matrix and the semantic prior related adjacency matrix in the first ASE-GCN layer, and the features are the hidden states of each node. The features of each node in the second layer are updated according to the adjacent nodes determined by the visual appearance related adjacency matrix, the physical space related adjacency matrix and the semantic prior related adjacency matrix in the second ASE-GCN layer, and the features are the hidden states of each node.

[0025] The stacked first ASE-GCN layer and second ASE-GCN layer aggregate the visual features and adjacent node information together to obtain the visual appearance features.

[0026] The semantic embedding layer is used to construct co-occurrence attributes of object pairs and predicates, map object categories to unique corresponding hidden representations in a hidden space, and connect the hidden representations to obtain semantic embedding clues for relationship prediction.

[0027] The hierarchical multi-modal fusion module adopts a tri-linear pooling fusion method, that is, the constructed hierarchical multi-modal fusion network (HMMFnet) extends the original bilinear pooling fusion method to a tri-linear pooling fusion method, utilizes complementary multi-object visual appearance information, physical space information and semantic embedding information, and improves the expression ability of visual relationship features.

[0028] In step 6, the formula for multi-role fusion of objects and relationships between objects in visual appearance clues is:

[0029]

[0030] Among them, represents the visual appearance feature between objects, represents the visual appearance feature of the predicate, and are dimensionally unified; F v (·) represents a visual appearance information fusion function based on an attention mechanism, V f is the output relationship of the visual appearance feature;

[0031] The formula for multi-role fusion of objects and relationships between objects in the physical space clue is:

[0032]

[0033] Among them represents the physical space feature between objects, represents the physical space feature of the predicate, and are dimensionally unified; F s (·) represents a physical space information fusion function based on an attention mechanism, S f is the output relationship physical space feature;

[0034] The formula for multi-modal fusion is:

[0035] R f =F(W,V f ,S f ,E f )

[0036] Among them, R f represents the final relationship feature, E f is the relationship semantic embedding feature; F(·) represents a fusion function; Representing parameter matrix, which means there are O1xO2xO3 parameters to learn, using factorization method to R f Tri-linear pooling decomposition into o channels, where the i _channel th channel formula is:

[0037]

[0038] Where, 1 is an all-one vector, x, y, z are the feature vectors of visual appearance, physical space, semantic embedding respectively, i _channel represents the i _channel th channel, k is a hyperparameter artificially specified, indicating merging into k dimensions, O1, O2, O3 are the lengths of corresponding feature vectors of x, y, z respectively; U i_channel , V i_channel , W i_channel represent the parameter matrix after the original parameter matrix W is decomposed.

[0039] Assuming the length of the output feature vector is o, the set of U i_channel is denoted as U, then U=[U1, U2, … U o ], i _channel =1, 2, 3…o; the set of V i_channel is denoted as V, then V=[V1, V2, … V o ], the set of W i_channel is denoted as W, then W=[W1, W2, … W o ], through the factorization method, the number of parameters is reduced from O1xO2xO3x o to (O1xk+O2xk+O3xk);

[0040] After re-formulating these parameter matrices and merging k dimensions, the factorization method is rewritten as:

[0041]

[0042] Where, SumPool(·) is the sum pooling function, and Norm(·) is the regularization function.

[0043] Beneficial effects

[0044] The application relates to the technical field of artificial intelligence understanding and environmental interaction, and particularly relates to an online video visual relationship detection method based on multi-modal information fusion. The application proposes an ASE-GCN layer, which models a latent relationship graph using three different relevant adjacency matrices. The matrix contains visual appearance, physical space and semantic embedding information. The fusion method based on bilinear pooling is extended to a fusion method based on tri-linear pooling, and a hierarchical fusion module is designed to fuse multi-role multi-modal features. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 The layered multi-modal fusion network is outlined. DETAILED DESCRIPTION

[0046] The application is further described below in conjunction with the accompanying drawings and examples.

[0047] A method for detecting visual relationships in online videos based on multi-modal information fusion, the steps of the method comprising:

[0048] Step 1, constructing a layered multi-modal fusion network, as shown in Figure 1 The constructed layered multi-modal fusion network includes two layers of ASE-GCN layers, a layered multi-modal fusion module, a YOLO target detector, and a semantic embedding layer, the layered multi-modal fusion module including a first layer of attention-based fusion module and a second layer of three-linear pooling fusion module;

[0049] Step 2, inputting online video data into the layered multi-modal fusion network constructed in step 1;

[0050] Step 3, the online video data input into the layered multi-modal fusion network in step 2 extracts the initial visual features of each object through a deep neural network, and inputs the extracted initial visual features of each object into the two layers of ASE-GCN layers to obtain visual appearance clues;

[0051] Step 4, inputting the frames of the online video data input into the layered multi-modal fusion network in step 2 into the YOLO target detector, constructing static information and action representation according to the object bounding box coordinates generated by the YOLO target detector, the static information as the physical space clues of the object pair, the action representation as the physical space clues of the predicate, and the physical space clues of the object pair and the predicate are spliced to obtain the spliced physical space clues;

[0052] Step 5, inputting the frames of the online video data input into the layered multi-modal fusion network in step 2 into the YOLO target detector, the class of each target generated by the YOLO target detector modeling the co-occurrence attributes of the object pair and the predicate through the semantic embedding layer, mapping the object class to a unique corresponding hidden representation in a hidden space, and connecting the hidden representation to obtain the semantic embedding clues for relationship prediction;

[0053] Step 6: input the visual appearance clues obtained in step 3, the spliced physical space clues obtained in step 4 and the semantic embedding clues obtained in step 5 into the hierarchical multi-modal information fusion module, pass through the first layer of attention-based fusion module, perform multi-role fusion on the objects and the relationships between the objects in the visual appearance clues, perform multi-role fusion on the objects and the relationships between the objects in the physical space clues, perform multi-role fusion on the objects and the relationships between the objects in the semantic embedding clues, and finally perform multi-modal fusion on the visual appearance clues, the physical space clues and the semantic embedding clues after respective fusion through the second layer of three-linear pooling fusion module to obtain a total fusion result;

[0054] Step 7: detect the interaction relationship between the objects in the total fusion result obtained in step 6, the interaction relationship includes up, down, left, right, close, far away and follow, obtain an online relationship detection result, and complete the online video visual relationship detection based on multi-modal information fusion.

[0055] In the step 1, the two layers of ASE-GCN layers in the hierarchical multi-modal fusion network are a first ASE-GCN layer and a second ASE-GCN layer, and the first ASE-GCN layer and the second ASE-GCN layer are stacked together.

[0056] The ASE-GCN layers respectively explore the correlation between the objects from the perspectives of visual appearance, physical space and semantic priori to capture all possible potential interaction relationships.

[0057] Each ASE-GCN layer includes a visual appearance related adjacency matrix, a physical space related adjacency matrix and a semantic priori related adjacency matrix, and the visual appearance related adjacency matrix, the physical space related adjacency matrix and the semantic priori related adjacency matrix determine the adjacency state and information update weight between nodes.

[0058] The visual appearance related adjacency matrix is obtained by filling the similarity between the objects into the corresponding position in the matrix and normalizing, and the similarity between the object i and the object j is:

[0059]

[0060] Wherein, x i is the visual appearance feature of the object i, and x j is the visual appearance feature of the object j.

[0061] Fill the similarity value simolarity(x i ,x j ) into the corresponding position (i _loc ,j _loc ) in the matrix, wherein the i _loc value is equal to i, and the j _locThe numerical value is equal to j, and then normalized to obtain the visual appearance related adjacency matrix;

[0062] The physical space related adjacency matrix is obtained by filling the spatial position correlation between object i and object j into the corresponding position in the matrix and normalizing, and the spatial position correlation is:

[0063]

[0064] Wherein, bbox i And bbox j Indicate the bounding box of object i and object j, I(·) indicates the intersection area between bbox i And bbox j , U(·) represents the union area between bbox i And bbox j , the stronger the spatial position correlation of the two objects, the greater the information transmission weight between the two objects;

[0065] The features of each node in the first layer are updated according to the adjacent nodes determined by the visual appearance related adjacency matrix, the physical space related adjacency matrix and the semantic prior related adjacency matrix in the first ASE-GCN layer, and the features are the hidden states of each node, and the features of each node in the second layer are updated according to the adjacent nodes determined by the visual appearance related adjacency matrix, the physical space related adjacency matrix and the semantic prior related adjacency matrix in the second ASE-GCN layer, and the features are the hidden states of each node;

[0066] The stacked first ASE-GCN layer and second ASE-GCN layer aggregate visual features and adjacent node information together to obtain visual appearance features;

[0067] The semantic embedding layer is used to build the co-occurrence attributes of object pairs and predicates, and map object categories to unique corresponding hidden representations in a hidden space, and the hidden representations are connected to obtain semantic embedding clues for relationship prediction;

[0068] The hierarchical multi-modal fusion module adopts a three-linear pooling fusion method, that is, the hierarchical multi-modal fusion network (HMMFnet) constructed expands the original bilinear pooling fusion method to a three-linear pooling fusion method, and utilizes the complement of multi-object visual appearance information, physical space information and semantic embedding information to improve the expression ability of visual relationship features;

[0069] In step 6, the formula for multi-role fusion of objects and relationships between objects in the visual appearance clue is:

[0070]

[0071] Wherein, represents the visual appearance feature between objects, represents the visual appearance feature of the predicate, and are dimensionally unified; F v (·) represents a visual appearance information fusion function based on attention mechanism, V f is the output relationship of visual appearance features;

[0072] The formula when multi-role fusion is performed on objects and the relationship between objects in the physical space clue is:

[0073]

[0074] wherein represents the physical space feature between objects, represents the physical space feature of the predicate, and are dimensionally unified; F s (·) represents a physical space information fusion function based on attention mechanism, S f is the output relationship physical space feature;

[0075] The formula when multi-modal fusion is performed is:

[0076] R f = F(W, V f , S f , E f )

[0077] wherein R f represents the final relationship feature, E f is the relationship semantic embedding feature; F(·) represents a fusion function; represents a parameter matrix, which means that there are O1×O2×O3parameters to be learned, and the factorization method is used to perform three-linear pooling decomposition on R f into o channels, wherein the formula of the i _channel th channel is:

[0078]

[0079] wherein, 1 is an all-one vector, x, y, z are feature vectors of visual appearance, physical space, and semantic embedding respectively, i _channel represents the i _channel th channel, k is a hyperparameter artificially specified, indicating merging into k dimensions, O1, O2, O3 are the lengths of the corresponding feature vectors of x, y, z respectively; U i_channel , V i_channel , W i_channelW represents a parameter matrix after decomposition of the original parameter matrix W.

[0080] Assuming that the length of the output feature vector is o, U i_channel The set of U is represented by U, then U=[U1, U2, … U o ], i _channel =1, 2, 3…o; V i_channel The set of V is represented by V, then V=[V1, V2, … V o ], W i_channel The set of W is represented by W, then W=[W1, W2, … W o ], through the factorization method, the number of parameters is reduced from O1xO2xO3xO to (O1xk+O2xk+O3xk);

[0081] After re-formulating and merging the k-dimensional of these parameter matrices, the factorization method is rewritten as:

[0082]

[0083] Where SumPool(·) is the sum pooling function, and Norm(·) is the regularization function.

[0084] Embodiment

[0085] An online video visual relationship detection method based on multi-modal information fusion, the steps of the method comprising:

[0086] ASE-GCN: The potential relationship between objects in the scene is difficult to represent in Euclidean space, so a directed graph is selected to describe the objects and the relationship between them in the scene. Each object in the scene is represented by a unique node, and the directed edge between two nodes represents the relationship between them. The update formula of the node attribute is:

[0087] Y=AXW

[0088] Where X is the node attribute, A is the adjacency matrix, and W is the learnable parameter matrix. By only calculating the value of the Internet of Things, the existence of the interaction is ignored, and some potential correlations are ignored. Therefore, the ASE-GCN layer is designed to improve the ability to capture relationships, where “ASE” represents “visual appearance, spatial position, and semantic priori”. In order to fully capture the potential relationship between objects, three adjacency matrices are designed: visual appearance related adjacency matrix, spatial position related adjacency matrix, and semantic prior related adjacency matrix.

[0089] (1) Visual appearance related adjacency matrix. The visual appearance related adjacency matrix is related to the attributes of the visual appearance of the objects. First, the object pair visual features are mapped to the hidden space through the shared fully connected layer, and the similarity of these visual appearances indicates the relationship between the objects. The similarity is calculated as follows:

[0090]

[0091] The similarity value similarity(x i ,x j ) is filled into the corresponding position (i _loc ,j _loc ) in the matrix, where i _loc is equal to the value of i, j _loc is equal to the value of j, and then normalized, and the visual appearance related adjacency matrix is obtained.

[0092] (2) Spatial position related adjacency matrix. Some predicates do not depend on the correlation between visual appearances, but depend on the spatial correlation between objects. If two objects are adjacent to each other in space, there is usually interaction between them. Therefore, the spatial position related adjacency matrix is designed to measure the spatial position correlation between two objects. The IoU value between objects is used as the basis for judging the potential spatial correlation between the two. The spatial position correlation between object i and object j is calculated as follows:

[0093]

[0094] where bbox i and bbox j represent the bounding boxes of object i and object j, I(·) represents the intersection area between bbox i and bbox j , and U(·) represents the union area between bbox i and bbox j . Then, through the normalization layer, the spatial position related adjacency matrix is obtained. The stronger the spatial position correlation between two objects, the greater the information transmission weight between the two objects.

[0095] (3) Semantic prior correlation adjacency matrix. Relation triples can be regarded as a combination of object pairs and predicates, and the purpose of the relationship detection network is to distinguish object pairs from appropriate predicates. According to the grammar and the real world, some object pairs and predicates are more likely to appear together than other object pairs and predicate combinations. In order to represent this semantic correlation, a semantic co-occurrence matrix is ​​constructed by counting the relationship triplets in the training set. If the frequency of the object pair in the training set is high, its co-occurrence probability value is high. Then, the prior probability of each object pair is normalized so that the sum of all predicate probabilities is 1. In order to improve the generalization ability of the network, the semantic prior correlation adjacency matrix is ​​assigned a small probability to the triplets that do not appear in the training set so that they can also transmit information.

[0096] At the ASE-GCN layer, the target features extracted by the deep network are propagated based on the weighted fusion of the graph described by the visual appearance association, spatial position association, and semantic prior association adjacency matrix, as well as the information of adjacent nodes;

[0097] Hierarchical Multimodal Fusion Module: Object-pair visual appearance information, object-pair physical space information, predicate visual appearance information, and predicate physical space information can be extracted from the video. This feature information includes multimodal features of multiple characters. To fully utilize these features to represent relation triples, the present invention designs a hierarchical multimodal fusion module with two main functions: multi-character information fusion and multimodal information fusion.

[0098] The hierarchical multimodal fusion module has a two-layer structure. The purpose of the first layer is to fuse the multi-role information of each modality. Assume that the visual appearance feature of the object pair is Vf <S,O> , the visual appearance feature of the predicate is Vf p , where Vf p It is obtained by extracting the union of the subject and object bounding boxes from the frame features of the video through the RoI pooling operation. Since the dimension of the visual appearance features of the object is greater than the dimension of the predicate, in order to reduce the information flooding caused by the difference in feature dimensions, the present invention unifies the feature dimensions before information fusion. Multi-role fusion of visual features:

[0099]

[0100] in, represents the visual appearance features unified by the dimension, F v (·) represents the visual appearance information fusion function based on the attention mechanism, V f is the output relation of visual appearance features.

[0101] Physical space feature similarity. The physical space feature of the object pair is a static position feature, and the physical space feature of the predicate is a dynamic motion feature. Only the fusion of the two can fully reflect the physical space feature of the relationship. Similarly, the two physical space features are also unified by feature dimensions before fusion. Physical space feature fusion:

[0102]

[0103] wherein, represents the physical space feature unified by dimensions, F s (·) represents a physical space information fusion function based on an attention mechanism, S f is the output relationship physical space feature.

[0104] The semantic information part connects the semantic embedding features of the object pair into the relationship semantic embedding feature E f .

[0105] The second layer of the hierarchical multi-modal module adopts a linear pooling fusion method. The invention extends the bilinear pooling fusion method to a trilinear pooling fusion method to fully integrate multi-modal information. The trilinear pooling fusion is shown in the following formula.

[0106] R f =F(W,V f ,S f ,E f )

[0107] wherein, R f represents the final relationship feature, represents a learnable parameter matrix, which means that there are O1×O2×O3 parameters to be learned. Such a large number of learnable parameters will occupy a large amount of storage space and greatly reduce the learning speed. The factorization method is used to decompose the trilinear pooling:

[0108]

[0109] wherein, 1 is an all-one vector. Assuming that the length of the output feature vector is o, then U = [U1, U2, … U O ], V = [V1, V2, … V O ], W = [W1, W2, … W O ]. Through the factorization method, the number of parameters is reduced from O1×O2×O3×o to (O1×k+O2×k+O3×k). Then, these parameter matrices are re-formulated and combined into k dimensions, and then the factorization method can be rewritten as:

[0110]

[0111] where SumPool(·) is a sum-pooling function and Norm(·) is a normalization function. In the present application, k is set to 3.

Claims

1. An online video visual relationship detection method based on multimodal information fusion, characterized by The steps of the method include: Step 1: Construct a hierarchical multimodal fusion network. The constructed hierarchical multimodal fusion network includes two ASE-GCN layers, a hierarchical multimodal fusion module, a YOLO object detector, and a semantic embedding layer. The hierarchical multimodal fusion module includes an attention-based fusion module in the first layer and a trilinear pooling fusion module in the second layer. Step 2: Input the online video data into the hierarchical multi-mode fusion network constructed in step 1; Step 3: The online video data input into the hierarchical multimodal fusion network in step 2 is used to extract the initial visual features of each object through a deep neural network, and the extracted initial visual features of each object are input into two layers of ASE-GCN layers to obtain visual appearance clues; Step 4: Input the frames of online video data input into the hierarchical multimodal fusion network in step 2 into the YOLO object detector. Based on the target bounding box coordinate representation of each object generated by the YOLO object detector, static information and action representation are constructed. The static information is used as the physical space clue of the object pair, and the action representation is used as the physical space clue of the predicate. The physical space clues of the object pair and the physical space clues of the predicate are spliced ​​to obtain the spliced ​​physical space clues. Step 5: Input the frames of online video data input into the hierarchical multimodal fusion network in step 2 into the YOLO object detector. The category of each object generated by the YOLO object detector is modeled through the semantic embedding layer to model the co-occurrence property of object pairs and predicates, mapping the object category to a unique corresponding hidden representation in a latent space. The hidden representations are concatenated to obtain the semantic embedding clues for relationship prediction; Step 6: The visual appearance cues obtained in step 3, the concatenated physical space cues obtained in step 4, and the semantic embedding cues obtained in step 5 are input into the hierarchical multimodal information fusion module. After passing through the first-layer attention-based fusion module, the objects and the relationships between objects in the visual appearance cues are subjected to multi-role fusion, the objects and the relationships between objects in the physical space cues are subjected to multi-role fusion, and the objects and the relationships between objects in the semantic embedding cues are subjected to multi-role fusion. Finally, the second-layer trilinear pooling fusion module is used to perform multimodal fusion on the fused visual appearance cues, physical space cues, and semantic embedding cues to obtain the overall fusion result. Step 7: Detect the interaction relationship between object pairs in the total fusion result obtained in step 6, and complete the online video visual relationship detection based on multimodal information fusion.

2. The online video visual relationship detection method based on multimodal information fusion according to claim 1, characterized in that: In step 1, the two ASE-GCN layers in the layered multi-mode fusion network are respectively a first ASE-GCN layer and a second ASE-GCN layer, and the first ASE-GCN layer and the second ASE-GCN layer are stacked together; The ASE-GCN layer explores the associations between objects from the perspectives of visual appearance, physical space, and semantic priors to capture all possible potential interactions.

3. The online video visual relationship detection method based on multimodal information fusion according to claim 2, characterized in that: Each ASE-GCN layer contains a visual appearance-related adjacency matrix, a physical space-related adjacency matrix, and a semantic prior-related adjacency matrix. The visual appearance-related adjacency matrix, the physical space-related adjacency matrix, and the semantic prior-related adjacency matrix determine the adjacency status and information update weight between nodes.

4. The online video visual relationship detection method based on multimodal information fusion according to claim 3 is characterized by: The visual appearance related adjacency matrix is ​​obtained by filling the similarities between objects into the corresponding positions in the matrix and normalizing them. The similarity between object i and object j is: Among them, x i is the visual appearance feature of object i, x j is the visual appearance feature of object j.

5. The online video visual relationship detection method based on multimodal information fusion according to claim 3 or 4, characterized in that: The physical space correlation adjacency matrix is ​​obtained by filling the spatial position correlation between object i and object j into the corresponding positions in the matrix and normalizing them. The spatial position correlation is: Among them, bbox i and bbox j represents the bounding box of object i and object j, I(·) represents bbox i and bbox j The intersection area between them, U(·) represents the bbox i and bbox j The union area between .

6. The online video visual relationship detection method based on multimodal information fusion according to claim 5, characterized in that: Update the features of each node in the first layer according to the adjacent nodes determined by the visual appearance related adjacency matrix, physical space related adjacency matrix and semantic prior related adjacency matrix in the first ASE-GCN layer. The features are The hidden state of each node updates the features of each node in the second layer according to the adjacent nodes determined by the visual appearance related adjacency matrix, physical space related adjacency matrix and semantic prior related adjacency matrix in the second ASE-GCN layer. The feature is the hidden state of each node.

7. The online video visual relationship detection method based on multimodal information fusion according to claim 6, characterized in that: The stacked first and second ASE-GCN layers aggregate visual features with adjacent node information to obtain visual appearance features; The semantic embedding layer is used to construct co-occurrence attributes of object pairs and predicates, mapping object categories to unique corresponding hidden representations in a latent space, and the hidden representations are connected to obtain semantic embedding clues for relationship prediction.

8. The online video visual relationship detection method based on multimodal information fusion according to claim 7, characterized in that: The hierarchical multimodal fusion module adopts a trilinear pooling fusion method, that is, the constructed hierarchical multimodal fusion network expands the original bilinear pooling fusion method into a trilinear pooling fusion method, and utilizes the visual appearance information, physical space information and semantic embedding information of multiple objects to complement each other, thereby improving the expression ability of visual relationship features.

9. The online video visual relationship detection method based on multimodal information fusion according to claim 1, characterized in that: In step 6, the formula for multi-role fusion of objects and relationships between objects in visual appearance clues is: in, Represents the visual appearance characteristics between objects, Represents the visual appearance characteristics of the predicate, and It is dimensionally unified; F v (·) represents the visual appearance information fusion function based on the attention mechanism, V f is the output relation of visual appearance features; The formula for multi-role fusion of objects and relationships between objects in physical space clues is: in, Represents the physical space characteristics between objects, Represents the physical space characteristics of the predicate, and It is dimensionally unified; F s (·) represents the physical space information fusion function based on the attention mechanism, S f is the relational physical space feature of the output; The formula for multimodal fusion is: R f =F(W,V f ,S f ,E f ) Among them, R f Represents the final relationship feature, E f is the relational semantic embedding feature; F(·) represents the fusion function; Represents the parameter matrix, using factor decomposition method to R f Perform trilinear pooling decomposition into o channels, where the i-th _channel The formula for each channel is: in, 1 is an all-one vector, x, y, z are the feature vectors of visual appearance, physical space, and semantic embedding respectively, i _channel Indicates the i _channel channels, k is a manually specified hyperparameter, indicating merging into k dimensions, O1, O2, and O3 are the lengths of the corresponding feature vectors of x, y, and z respectively; Represents the parameter matrix after the original parameter matrix W is decomposed; Assume that the length of the output feature vector is o, The set of is represented by U, then U=[U1,U2,…U o ],i _channel =1,2,3…o; The set of is represented by V, then V=[V1,V2,…V o ], The set of is represented by W, then W=[W1,W2,…W o ], through the factorization method, the number of parameters is reduced from O1×O2×O3×o to (O1×k+O2×k+O3×k); After reformulating these parameter matrices and merging the k dimensions, the factorization method is rewritten as: Among them, SumPool(·) is the sum pooling function, and Norm(·) is the regularization function.

10. The online video visual relationship detection method based on multimodal information fusion according to claim 1, characterized in that: In step 7, the interactive relationships include up, down, left, right, approach, move away, and follow.

Images