An adaptive temporal aggregation network and method for online video visual relationship detection

Abstract

The present invention relates to artificial intelligence understanding and environmental interaction, and in particular to an adaptive time aggregation network and method for online video visual relationship detection. The dynamic buffer memory used in the present invention to store historical video clips can fix the size of the storage content, and the storage content will not become larger and larger due to time. The step sampling strategy adopted by the present invention sets the sampling frequency by the strength of the correlation between the video frame and the current key frame, which can reduce the computational consumption required by the network while maintaining the accuracy of the detection results of the relationship between objects in the video in the time dimension. The adaptive spatiotemporal activation module and the attention-based knowledge state fusion module proposed in the present invention enable the invention to adaptively extract and fuse historical information and current status, and can detect the dynamic and static relationships of objects.

Description

Technical Field

[0001] The present invention relates to artificial intelligence understanding and environmental interaction, and in particular to an adaptive temporal aggregation network and method for online video visual relationship detection. Background Art

[0002] Video visual relationship detection (VidVRD) is a challenging and important task in robotic scene perception. Its goal is to first detect objects in visual input and then identify their interactions. However, in the real world, the interactions between pairs of objects constantly change over time. This requires the network to update the relationship detection based on real-time input information. Existing methods for offline video visual relationship detection require the entire video as input and cannot process real-time incremental information, making them unsuitable for online visual relationship detection. Summary of the Invention

[0003] The technical problem solved by the present invention is to overcome the shortcomings of the existing technology and propose an adaptive temporal aggregation network and method for online video visual relationship detection, aiming to detect the interaction between target pairs in real time. The adaptive temporal aggregation network (ATAN) for online video visual detection includes an adaptive spatiotemporal activation module and an attention-based knowledge state fusion module. The network can adaptively extract and fuse historical knowledge and current state to detect relationships in real-time video. In addition, our network is evaluated on the ImageNet-VidVRD dataset using online relationship detection and online relationship labeling as evaluation indicators. Experimental results show that the network is effective for online video visual relationship detection.

[0004] The technical solution of the present invention is:

[0005] An adaptive temporal aggregation network for online video visual relationship detection, comprising a dynamic buffer memory, a step sampling module, a 2D backbone module, an adaptive spatiotemporal activation module, an attention-based knowledge-state fusion module, a YOLO module, and a directed graph module; the adaptive spatiotemporal activation module comprises an adaptive spatial activation unit and an adaptive temporal aggregation unit;

[0006] The dynamic buffer memory is used to store historical video clips. When a new video frame arrives, the dynamic buffer memory will forget the first video frame in the storage sequence, so that the entire storage sequence remains at a fixed length. This method can effectively avoid the expansion of storage resources over time.

[0007] The step sampling module is used to sample historical video clips from the dynamic buffer memory to obtain a historical memory sequence, and output the obtained historical memory sequence to the 2D backbone module. The sampling method is: adopting a step sampling strategy, that is, setting the sampling frequency based on the degree of correlation with the current frame. Video frames that are more distant in time have a weaker correlation with the current key frame, while video frames that are closer in time have a stronger correlation with the current key frame, thereby reducing overall computing resources.

[0008] The 2D backbone module is used to receive the historical memory sequence output by the step sampling module and also to receive the current frame, obtain a historical memory feature map formed by frame-level features corresponding to the sampling frame based on the received historical memory sequence, and output it to the adaptive spatiotemporal activation module; obtain a current state formed by the corresponding frame-level features based on the received current frame, and output it to the attention-based knowledge-state fusion module;

[0009] The adaptive spatiotemporal activation module is used to receive the historical memory feature map output by the 2D backbone module. The adaptive spatial activation unit in the adaptive spatiotemporal activation module adaptively activates instructive spatial information in the spatial dimension based on the content of the received historical memory feature map. The adaptive temporal aggregation unit in the adaptive spatiotemporal activation module adaptively aggregates instructive temporal information in the temporal dimension based on the content of the received historical memory feature map. The spatial information and temporal information are combined to form historical knowledge, which is output to the attention-based knowledge-state fusion module.

[0010] The method of the adaptive spatial activation unit adaptively activating instructive spatial information in the spatial dimension according to the content of the received historical memory feature map is as follows:

[0011] Input historical memory feature map Represented as follows, where C is the number of channels, T is the length of the historical memory sequence, H and W are the height and width of the historical memory feature map, respectively. Since the historical memory features include not only object features but also background information, the adaptive spatial activation unit focuses on enhancing object features while weakening background noise. When the historical memory features are known, the input features are first compressed in the time dimension to focus on spatial information.

[0012] TemSqueeze(F M )= <TemAvgPooling(F M ),TemMaxPooling(F M )>

[0013] Where <·> is the feature connection, TemAvgPooling(F M ) is F MGlobal average pooling, TemMaxPooling (F M ) is F M Global maximum pooling, TemSqueeze(F M ) is the compressed input feature in the time dimension;

[0014] First, the input features are globally averaged and globally max-pooled. Next, the spatial pattern of historical memory features is captured using stacked 2D convolutional layers by generating position-sensitive activation masks. Then, position information weights are generated based on the activation masks. Finally, these weights are rescaled to the same shape of C×T×H×W and then multiplied by the input features F. M ;

[0015]

[0016] Among them, rescale(·) means rescaling the features to the same shape as X, conv(·) is implemented by two stacked convolutional layers, and F spatial To generate instructive spatial information;

[0017] The method in which the adaptive time aggregation unit adaptively aggregates instructive time information in the time dimension according to the content of the received historical memory feature map is as follows:

[0018] The adaptive time aggregation unit focuses on acquiring time information. The specific design and method are as follows: In order to effectively learn time correlation, an adaptive time aggregation module is designed to learn the time signal with unchanged position, which is used for the aggregation of time information in convolution operation. Similar to the adaptive spatial activation unit, the adaptive time aggregation unit first compresses the input feature F in the spatial dimension. M Then, a dynamic one-dimensional signal is generated from the remaining temporal features, which is implemented by two stacked fully connected layers. The generation of the temporal kernel is shown in the following formula.

[0019]

[0020] Among them, SpaSqueeze(F M ) is the compressed input feature in the spatial dimension, and fc(·) is implemented by two stacked fully connected layers. is the generated adaptive timing kernel.

[0021] The method for combining the spatial information and the temporal information to form historical knowledge is as follows:

[0022] Aggregate F through convolution operation spatial and Finally, extract historical knowledge F history ;

[0023]

[0024] The attention-based knowledge-state fusion module is used to receive the historical knowledge output by the adaptive spatiotemporal activation module and the current state output by the 2D backbone module, and spontaneously selects appropriate information from the received historical knowledge and current state to focus on, generate visual appearance cues, and output them to the directed graph module;

[0025] The method of generating visual appearance clues is to fuse the features of video clips and current frames to improve the performance of action detection. Based on this idea, the attention mechanism is used to extract relevant information from historical knowledge and current state. The input historical knowledge is recorded as The current state is recorded as F history and F current Cascade on the channel dimension to construct fusion features Fusion feature F fusion Each channel in F represents a mode of obtaining visual information, so different channels focus on different modes. fusion They are fed into query, key and value branches respectively, each branch has an independent mapping function to map the fused features to the new latent space;

[0026] F query =(conv q ( fusion ))

[0027] F key =(conv k ( fusion ))

[0028] F value =(conv v ( fusion ))

[0029] Among them, σ(·) is the activation function, conv(·) is implemented by the convolution layer. After obtaining these three mapping features, the relevant information is activated through the attention mechanism. First, F key Transpose and multiply by F query , aims to mine the potential correlation between regional channels, the channel correlation matrix F mask The calculation formulas for each value in are as follows.

[0030]

[0031] Among them, F mask,(,l) It is F in row c and column l maskThen, the channel correlation matrix is ​​used as the activation weight to activate the mapped features in the value branch to generate the attention-enhanced fusion feature F. f ′ usion .

[0032]

[0033] Among them, F f ′ usion,(,l) It is F in row c and column l f ′ usion In the attention mechanism, the value close to zero in the activation matrix will cause the activation feature to have a zero tensor, thus affecting the back propagation of the loss. To avoid this problem, the weighted residual structure in the proposed module is used to convert the weighted original fusion feature F fusion The fused feature F added to the attention activation f ′ usion In addition, in order to improve the generalization ability of the network, a dropout layer is set at the beginning of the attention-based knowledge state fusion module, and the probability of dropping neurons is p = 0.1.

[0034] F visual =ropout(conv(α·F f ′ usion +(1-) fusion ),p)

[0035] Among them, α is a learnable parameter that will learn an appropriate ratio to balance the two features during training, and dropout(·) is a dropout layer implementation. Finally, using F visual Generate visual appearance cues for relationship detection;

[0036] The YOLO module is used to identify the bounding box and the category of the object from the current frame to obtain spatial position clues and semantic embedding clues and output them to the directed graph module;

[0037] The directed graph module is used to receive the visual appearance clues output by the attention-based knowledge-state fusion module, and is also used to receive the spatial position clues and semantic embedding clues output by the YOLO module, and directly connect the visual appearance clues, spatial position clues, and semantic embedding clues to obtain online relationship detection results.

[0038] A method for detecting visual relationships in online videos, the method comprising the following steps:

[0039] In the first step, a dynamic buffer memory is used to save historical video clips;

[0040] In the second step, the historical video clips saved in the first step are input into the step sampling module to obtain the historical memory sequence;

[0041] In the third step, the historical memory sequence obtained in the second step and the current frame are input into the 2D backbone module to obtain the historical memory feature map and the current state;

[0042] In the fourth step, the historical memory feature map obtained in the third step is input into the adaptive spatiotemporal activation module to obtain historical knowledge;

[0043] In the fifth step, the current state obtained in the third step and the historical knowledge obtained in the fourth step are input into the attention-based knowledge state fusion module to obtain visual appearance clues;

[0044] Step 6: Input the current frame into the YOLO module to obtain spatial position clues and semantic embedding clues;

[0045] In the seventh step, the visual appearance clues obtained in the fifth step and the spatial position clues and semantic embedding clues obtained in the sixth step are input into the directed graph module and directly connected to obtain the online relationship detection results.

[0046] Beneficial effects

[0047] The advantage of the present invention is that it fills the gap in online video visual relationship detection. Existing methods only realize offline video visual relationship detection, which requires downloading the video to globally detect the visual relationship in the video. The present invention can detect the visual relationship in the picture in real time through the camera.

[0048] The dynamic buffer memory used in the present invention to store historical video clips can fix the size of the storage content and will not cause the storage content to become larger and larger over time.

[0049] The step sampling strategy adopted by the present invention sets the sampling frequency by the strength of the correlation between the video frame and the current key frame, which can reduce the computing consumption required by the network while maintaining the accuracy of the detection results of the relationship between objects in the video in the time dimension.

[0050] The adaptive spatiotemporal activation module and attention-based knowledge state fusion module proposed in the present invention enable the invention to adaptively extract and fuse historical information and current state, and to detect dynamic and static relationships between objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 Schematic diagram of the composition of the adaptive time aggregation network of the present invention. DETAILED DESCRIPTION

[0052] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0053] like Figure 1 As shown, an adaptive temporal aggregation network for online video visual relationship detection includes a dynamic buffer memory, a step sampling module, a 2D backbone module, an adaptive spatiotemporal activation module, an attention-based knowledge-state fusion module, a YOLO module, and a directed graph module;

[0054] The dynamic buffer memory is used to store historical video clips. When a new video frame arrives, the dynamic buffer memory forgets the first video frame in the storage sequence, so that the entire storage sequence remains at a fixed length.

[0055] The step sampling module is used to sample the historical video clips from the dynamic buffer memory to obtain a historical memory sequence, and output the obtained historical memory sequence to the 2D backbone module;

[0056] The 2D backbone module is used to receive the historical memory sequence output by the step sampling module and also to receive the current frame, obtain a historical memory feature map formed by frame-level features corresponding to the sampling frame based on the received historical memory sequence, and output it to the adaptive spatiotemporal activation module; obtain a current state formed by the corresponding frame-level features based on the received current frame, and output it to the attention-based knowledge-state fusion module;

[0057] The adaptive spatiotemporal activation module is configured to receive the historical memory feature map output by the 2D backbone module, and adaptively activate instructive spatial information in the spatial dimension and aggregate instructive temporal information in the temporal dimension based on the content of the received historical memory feature map, and then merge the spatial and temporal information to form historical knowledge, which is then output to the attention-based knowledge-state fusion module.

[0058] The attention-based knowledge-state fusion module is used to receive the historical knowledge output by the adaptive spatiotemporal activation module and the current state output by the 2D backbone module, and selects set information from the received historical knowledge and current state to focus on, generate visual appearance clues, and output them to the directed graph module;

[0059] The YOLO module is used to identify the bounding box and the category of the object from the current frame, obtain the spatial position clues and semantic embedding clues and output them to the directed graph module;

[0060] The directed graph module is used to receive visual appearance cues output by the attention-based knowledge-state fusion module, and is also used to receive spatial location cues and semantic embedding cues output by the YOLO module, and directly connects the visual appearance cues, spatial location cues, and semantic embedding cues to obtain an online relationship detection result.

[0061] Stepped Sampling Strategy

[0062] The present invention uses a dynamic buffer memory to save historical video clips. When a new video frame arrives, the dynamic buffer memory will forget the first video frame in the storage sequence, making the entire storage sequence maintain a fixed length. Although this method can effectively avoid the expansion of storage resources over time, using the entire sequence as historical memory also requires excessive computing resources, so a short historical memory needs to be sampled from the dynamic buffer memory.

[0063] Generally, video frames with a longer time distance have a weaker association with the current key frame, while video frames with a shorter time distance have a stronger association with the current key frame. Therefore, the sampling frequency can be set according to the degree of association with the current frame. As the association increases, the sampling frequency also increases. Given a storage sequence denoted by M, first, the entire sequence M is evenly divided into N steps. Each step has its own sampling frequency, and the sampling frequency of the i-th step is denoted as si, i ∈ [1, 2, ···, N]. According to the assumption, the sampling frequency is higher the closer it is to the current frame, so let s1 < s2 < ··· < sN. The memory segment sampled in the i-th step is denoted as mi, i ∈ [1, 2, ···, N]. Then, these segments are concatenated in chronological order to obtain historical memory. After sampling historical memory from the dynamic buffer memory, historical knowledge can be extracted from these discrete frames.

[0064] Adaptive Spatio-Temporal Activation Module

[0065] After obtaining the historical memory sequence using the stepped sampling strategy, we input the video frames of this sequence into the trained two-dimensional backbone to obtain a historical memory feature map formed by the frame-level features corresponding to the sampled frames. Then, historical information is extracted from these discrete features for relationship detection.

[0066] To extract useful information from discrete video frame features, an adaptive spatio-temporal activation module is proposed, which aims to adaptively activate guiding information in the spatio-temporal dimension according to the historical memory content. The adaptive spatio-temporal activation can be expressed by the following formula.

[0067]

[0068] Where X is the input history memory feature, represents the generation of adaptive time kernel, Represents adaptive spatial activation. The adaptive spatiotemporal activation module consists of two parts: the adaptive spatiotemporal activation unit and the adaptive spatiotemporal aggregation unit.

[0069] Input history memory feature , where C is the number of channels, T is the length of the historical memory sequence, and H and W are the height and width of the feature map, respectively. Because historical memory features include not only object features but also background information, the adaptive spatial activation unit focuses on enhancing object features while weakening background noise. Given the historical memory features, the input features are first compressed in the temporal dimension to focus on spatial information.

[0070] TemSqueeze(F M )= <TemAvgPooling(F M ),TemMaxPooling(F M )>

[0071] where <·> is a feature connection.

[0072] As shown in the above formula, the input features are first subjected to global average pooling and global maximum pooling. Next, the spatial pattern of historical memory features is captured using stacked 2D convolutional layers by generating position-sensitive activation masks. Then, position information weights are generated based on the activation masks. Finally, these weights are rescaled to the same shape of C×T×H×W and then multiplied by the input features F. M .

[0073]

[0074] where rescale(·) means rescaling the features to the same shape as X, and conv(·) is implemented by two stacked convolutional layers.

[0075] Different from the adaptive spatial activation unit, the adaptive temporal aggregation unit focuses on modeling temporal correlation. In order to effectively learn temporal correlation, we design an adaptive temporal aggregation module to learn position-invariant temporal signals for the aggregation of temporal information in convolution operations. Similar to the adaptive spatial activation unit, the adaptive temporal aggregation unit first compresses the input features F in the spatial dimension. M We then generate a dynamic one-dimensional signal from the remaining temporal features, which is implemented by two stacked fully connected layers. The temporal kernel is generated as follows.

[0076]

[0077] The weights are then used to aggregate the spatial activation features. As shown in the following formula, the position-sharing weights are aggregated through the convolution operation f spatial , extract historical information F history .

[0078]

[0079] Attention-based Knowledge-State Fusion Module

[0080] The entire network has two input sources for relationship detection: historical knowledge and the current state. Historical knowledge records the movement and changes of objects over time, while the current state focuses on the current visual appearance. Some relationships, such as "flying above" and "running past," rely more heavily on historical knowledge. Furthermore, detecting some relationships, such as "above" and "behind," depends primarily on the current state. At the same time, some relationships require both historical knowledge and the current state for classification, requiring the detection network to be able to autonomously select appropriate information for attention. To this end, an attention-based knowledge state fusion module is proposed.

[0081] The features of the video clip and the current frame are integrated to improve the performance of action detection. Based on this idea, the attention mechanism is used to extract relevant information from historical knowledge and current state. The input historical knowledge is recorded as The current state is recorded as F history and F current Cascade on the channel dimension to construct fusion features Fusion feature F fusion Each channel in F represents a mode of acquiring visual information, so different channels focus on different modes. fusion They are fed into query, key and value branches respectively, and each branch has an independent mapping function to map the fused features to a new latent space.

[0082] F query =(conv q (F fusion ))

[0083] F key =(conv k (F fusion ))

[0084] F value =(conv v (F fusion ))

[0085] Where σ(·) is the activation function and conv(·) is implemented by the convolutional layer.

[0086] After obtaining these three mapping features, we can activate relevant information through the attention mechanism. First, F key Transpose and multiply by F query , which aims to explore the potential correlation between regional channels. Channel correlation matrix F mask The calculation formulas for each value in are as follows.

[0087]

[0088] Among them, F mask,(,l) It is F in row c and column l mask Then, the channel correlation matrix is ​​used as the activation weight to activate the mapped features in the value branch to generate the attention-enhanced fusion feature F. f ′ usion .

[0089]

[0090] Among them, F f ′ usion,(c,l) It is F in row c and column l f ′ usion In the attention mechanism, the value close to zero in the activation matrix will cause the activation feature to have a zero tensor, thus affecting the back propagation of the loss. To avoid this problem, the weighted residual structure in the proposed module is used to convert the weighted original fusion feature F fusion The fused feature F added to the attention activation f ′ usion In addition, in order to improve the generalization ability of the network, a dropout layer is set at the beginning of the attention-based knowledge state fusion module, and the probability of dropping neurons is p = 0.1.

[0091] F visual =dropout(conv(α·F f ′ usion +(1-) fusion ),p)

[0092] Among them, α is a learnable parameter, which will learn an appropriate ratio to balance the two features during the training process. Finally, using F visual Generating visual appearance cues for relationship detection.

Claims

1. An adaptive temporal aggregation network for online video visual relationship detection, characterized by: The adaptive temporal aggregation network includes a dynamic buffer memory, a step sampling module, a 2D backbone module, an adaptive spatiotemporal activation module, an attention-based knowledge-state fusion module, a YOLO module, and a directed graph module; The dynamic buffer memory is used to store historical video clips. When a new video frame arrives, the dynamic buffer memory forgets the first video frame in the storage sequence, so that the entire storage sequence remains at a fixed length. The step sampling module is used to sample the historical video clips from the dynamic buffer memory to obtain a historical memory sequence, and output the obtained historical memory sequence to the 2D backbone module; The 2D backbone module is used to receive the historical memory sequence output by the step sampling module and also to receive the current frame, obtain a historical memory feature map formed by frame-level features corresponding to the sampling frame based on the received historical memory sequence, and output it to the adaptive spatiotemporal activation module; obtain a current state formed by the corresponding frame-level features based on the received current frame, and output it to the attention-based knowledge-state fusion module; The adaptive spatiotemporal activation module is configured to receive the historical memory feature map output by the 2D backbone module, and adaptively activate instructive spatial information in the spatial dimension and aggregate instructive temporal information in the temporal dimension based on the content of the received historical memory feature map, and then merge the spatial and temporal information to form historical knowledge, which is then output to the attention-based knowledge-state fusion module. The attention-based knowledge-state fusion module is used to receive the historical knowledge output by the adaptive spatiotemporal activation module and the current state output by the 2D backbone module, and selects set information from the received historical knowledge and current state to focus on, generate visual appearance clues, and output them to the directed graph module; The YOLO module is used to identify the bounding box and the category of the object from the current frame, obtain the spatial position clues and semantic embedding clues and output them to the directed graph module; The directed graph module is used to receive the visual appearance clues output by the attention-based knowledge-state fusion module, and is also used to receive the spatial position clues and semantic embedding clues output by the YOLO module, and directly connect the visual appearance clues, spatial position clues, and semantic embedding clues to obtain online relationship detection results.

2. The adaptive temporal aggregation network for online video visual relationship detection according to claim 1, characterized in that: The method for sampling historical video clips by the step sampling module is: adopting a step sampling strategy, that is, setting the sampling frequency according to the degree of association with the current frame, the video frame with a longer time distance has a weaker association with the current key frame, and the video frame with a closer time distance has a stronger association with the current key frame.

3. The adaptive temporal aggregation network for online video visual relationship detection according to claim 1 or 2, characterized in that: The adaptive spatiotemporal activation module includes an adaptive spatial activation unit and an adaptive temporal aggregation unit.

4. The adaptive temporal aggregation network for online video visual relationship detection according to claim 3, characterized in that: The adaptive spatial activation unit in the adaptive spatiotemporal activation module adaptively activates instructive spatial information in the spatial dimension according to the content of the received historical memory feature map.

5. The adaptive temporal aggregation network for online video visual relationship detection according to claim 3 or 4, characterized in that: The adaptive time aggregation unit in the adaptive spatiotemporal activation module adaptively aggregates instructive time information in the time dimension according to the content of the received historical memory feature map.

6. The adaptive temporal aggregation network for online video visual relationship detection according to claim 5, characterized in that: The method for the adaptive spatial activation unit to adaptively activate instructive spatial information in the spatial dimension is: Input historical memory feature map Represented as, where C is the number of channels, T is the length of the historical memory sequence, H and W are the height and width of the historical memory feature map respectively; Compress input features in the time dimension to focus on spatial information; TemSqueeze(F M )=<TemAvgPooling(F M ),TemMaxPooling(F M )> Where <·> is the feature connection, TemAvgPooling(F M ) is F M Global average pooling, TemMaxPooling (F M ) is F M Global maximum pooling, TemSqueeze(F M ) is the compressed input feature in the time dimension; First, the input features are globally averaged and globally max-pooled. Then, a position-sensitive activation mask is generated, and a stacked 2D convolution layer is used to capture the spatial pattern of the historical memory features. Position information weights are generated according to the activation mask. Finally, these weights are rescaled to the same shape of C×T×H×W and then multiplied by the input feature F. M ; Among them, rescale(·) means rescaling the features to the same shape as X, conv(·) is implemented by two stacked convolutional layers, and F spatial To generate guiding spatial information.

7. The adaptive temporal aggregation network for online video visual relationship detection according to claim 5, characterized in that: The method for the adaptive time aggregation unit to adaptively aggregate instructive time information in the time dimension is: First, compress the input feature F in the spatial dimension M ,Then, a dynamic one-dimensional signal is generated from the remaining temporal features, which is implemented by two stacked fully connected layers. The generation of the temporal kernel is shown as follows: Among them, SpaSqueeze(F M ) is the compressed input feature in the spatial dimension, and fc(·) is implemented by two stacked fully connected layers. is the generated adaptive timing kernel.

8. The adaptive temporal aggregation network for online video visual relationship detection according to claim 7, characterized in that: The method for combining the spatial information and the temporal information to form historical knowledge is as follows: Aggregate F through convolution operation spatial and Finally, extract historical knowledge F history :

9. The adaptive temporal aggregation network for online video visual relationship detection according to claim 7, characterized in that: The method for generating visual appearance clues is: Fusion of video clip and current frame features, using attention mechanism to extract relevant information from historical knowledge and current state, the input historical knowledge is recorded as The current state is recorded as F history and F current Cascade on the channel dimension to construct fusion features Fusion feature F fusion Each channel in F represents a mode of obtaining visual information. Different channels focus on different modes. fusion They are fed into query, key and value branches respectively, each branch has an independent mapping function to map the fused features to the new latent space; F query =(conv q ( fusion )) F key =(conv k ( fusion )) F value =(conv v ( fusion )) Among them, σ(·) is the activation function, conv(·) is implemented by the convolution layer. After obtaining these three mapping features, the relevant information is activated through the attention mechanism. First, F key Transpose and multiply by F query , channel correlation matrix F mask The calculation formulas for each value in are as follows: Among them, F mask,(,l) It is F in row c and column l mask Then, the channel correlation matrix is ​​used as the activation weight to activate the mapped features in the value branch to generate the attention-enhanced fusion feature F. f ′ usion : Among them, F f ′ usion,(,l) It is F in row c and column l f ′ usion Value, c∈[1,2,…,′], l∈[1,2,…,(H*W)], the weighted original fusion feature F fusion The fused feature F added to the attention activation f ′ usion In [15], a dropout layer is set at the beginning of the attention-based knowledge state fusion module, and the probability of dropping neurons is p = 0.1; F visual =ropout(conv(α·F f ′ usion +(1-)· fusion ),p) Among them, α is a learnable parameter that will learn an appropriate ratio to balance the two features during training, dropout(·) is a dropout layer implementation, and finally, using F visual Generating visual appearance cues for relationship detection.

10. A method for online video visual relationship detection, characterized in that The steps of the method include: In the first step, a dynamic buffer memory is used to save historical video clips; In the second step, the historical video clips saved in the first step are input into the step sampling module to obtain the historical memory sequence; In the third step, the historical memory sequence obtained in the second step and the current frame are input into the 2D backbone module to obtain the historical memory feature map and the current state; In the fourth step, the historical memory feature map obtained in the third step is input into the adaptive spatiotemporal activation module to obtain historical knowledge; In the fifth step, the current state obtained in the third step and the historical knowledge obtained in the fourth step are input into the attention-based knowledge state fusion module to obtain visual appearance clues; Step 6: Input the current frame into the YOLO module to obtain spatial position clues and semantic embedding clues; In the seventh step, the visual appearance clues obtained in the fifth step and the spatial position clues and semantic embedding clues obtained in the sixth step are input into the directed graph module and directly connected to obtain the online relationship detection results.

Images