Video multi-modal representation learning method and device

Abstract

The application provides a video multi-modal representation learning method and device, and relates to the technical field of computer vision. The method comprises the following steps: analyzing the compression encoding of a video to obtain key frames in the video and motion vectors of the key frames; extracting entity visual representations from the key frames, obtaining action visual representations based on the entity visual representations and the motion vectors of the key frames, and extracting action text representations and entity text representations from text descriptions corresponding to the video; fusing the entity visual representations and the entity text representations to obtain video entity multi-modal representations, and fusing the action visual representations and the action text representations to obtain video action multi-modal representations; and fusing the video entity multi-modal representations and the video action multi-modal representations to obtain multi-modal representations of the video. The application decouples the joint learning of the space-time information in the video into entity representation learning and action representation learning, thereby reducing the complexity of the joint learning of the space-time.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and apparatus for learning multimodal representations of videos. Background Technology

[0002] Video multimodal representation learning refers to learning the relationship between the visual representation of a video and the corresponding textual description, enabling computers to understand videos more accurately. It has wide applications in scenarios such as video retrieval, video classification, and video question answering.

[0003] Although significant progress has been made in video multimodal representation learning in recent years thanks to advancements in self-attention network architecture design, the computational complexity of jointly modeling spatiotemporal information in videos hinders this progress, especially for understanding long videos. Long videos contain numerous frames, and existing video multimodal representation learning methods can cause GPU memory overflow when learning from them, making normal learning difficult. Summary of the Invention

[0004] To address the problems existing in the prior art, the present invention provides a video multimodal representation learning method and apparatus.

[0005] In a first aspect, the present invention provides a video multimodal representation learning method, comprising:

[0006] The compressed encoding of the video is parsed to obtain the keyframes in the video and the motion vectors of the keyframes;

[0007] Entity visual representations are extracted from the keyframes. Based on the entity visual representations and the motion vectors of the keyframes, action visual representations are obtained. Action text representations are extracted from the text descriptions corresponding to the video, as well as entity text representations corresponding to the action text representations.

[0008] The entity visual representation and the entity text representation are fused to obtain a video entity multimodal representation, and the action visual representation and the action text representation are fused to obtain a video action multimodal representation.

[0009] The multimodal representation of the video entity and the multimodal representation of the video action are fused to obtain the multimodal representation of the video.

[0010] Optionally, according to the video multimodal representation learning method provided by the present invention, the step of parsing the compressed encoding of the video to obtain keyframes and motion vectors of the keyframes in the video includes:

[0011] The compressed encoding of the video is parsed to obtain keyframes in the video, motion vectors of the forward frames corresponding to each keyframe, and motion vectors of the bidirectional frames corresponding to each forward frame.

[0012] Optionally, according to a video multimodal representation learning method provided by the present invention, obtaining action visual representation based on the entity visual representation and the motion vectors of the keyframe includes:

[0013] Based on the entity visual representation and the motion vector of the forward frame, a first action visual representation is obtained;

[0014] Based on the first action visual representation and the motion vector of the bidirectional frame, a second action visual representation is obtained;

[0015] Accordingly, fusing the visual representation of the action and the textual representation of the action to obtain a multimodal representation of the video action includes:

[0016] The first visual representation of the action, the second visual representation of the action, and the textual representation of the action are fused to obtain the multimodal representation of the video action.

[0017] Optionally, according to a video multimodal representation learning method provided by the present invention, fusing the first action visual representation, the second action visual representation, and the action text representation to obtain the video action multimodal representation includes:

[0018] Spatially downsample the first action visual representation to obtain the third action visual representation, and spatially downsample the second action visual representation to obtain the fourth action visual representation.

[0019] The third action visual representation, the fourth action visual representation, and the action text representation are fused to obtain the video action multimodal representation.

[0020] Optionally, according to a video multimodal representation learning method provided by the present invention, the step of extracting entity visual representations from the keyframes includes:

[0021] Based on a pre-built visual attention network, entity visual features are extracted from each of the keyframes;

[0022] Determine the cumulative similarity of entity visual features of adjacent keyframes, and construct a cumulative similarity curve based on the cumulative similarity;

[0023] The target keyframes corresponding to the first preset number of positions on the cumulative similarity curve with the largest slope change are determined, and the entity visual features extracted from the target keyframes are used as the entity visual representation.

[0024] Optionally, according to the video multimodal representation learning method provided by the present invention, the step of extracting action text representations and entity text representations corresponding to the action text representations from the text descriptions corresponding to the video includes:

[0025] Based on a pre-built text feature model, text representations are extracted from the text descriptions corresponding to the video.

[0026] Extract the action text representation, the subject text representation corresponding to the subject of the action corresponding to the action text representation, and the receptor text representation corresponding to the receptor of the action corresponding to the action text representation from the text representation, and use the subject text representation and the receptor text representation as the entity text representation.

[0027] Optionally, according to the video multimodal representation learning method provided by the present invention, the step of fusing the entity visual representation and the entity text representation to obtain the video entity multimodal representation includes:

[0028] The entity visual representation and the entity text representation are concatenated to obtain the entity representation;

[0029] The entity representation is input into a pre-constructed self-attention network to obtain the video entity multimodal representation output by the self-attention network.

[0030] Optionally, according to the video multimodal representation learning method provided by the present invention, the step of fusing the action visual representation and the action text representation to obtain a video action multimodal representation includes:

[0031] The visual representation of the action and the textual representation of the action are concatenated to obtain the action representation;

[0032] The action representation is input into a pre-constructed self-attention network to obtain the video action multimodal representation output by the self-attention network.

[0033] Optionally, according to a video multimodal representation learning method provided by the present invention, fusing the video entity multimodal representation and the video action multimodal representation to obtain the multimodal representation of the video includes:

[0034] Based on the video entity multimodal representation and the video action multimodal representation, an action-entity graph is constructed. Each node in the action-entity graph is composed of each entity corresponding to the video entity multimodal representation and each action corresponding to the video action multimodal representation. Each edge in the action-entity graph is composed of each action and the connecting line between each action and the entity corresponding to each action.

[0035] The action-entity graph, the video entity multimodal representation, and the video action multimodal representation are input into a pre-constructed graph self-attention network to obtain the multimodal representation of the video output by the graph self-attention network.

[0036] In a second aspect, the present invention also provides a video multimodal representation learning device, comprising:

[0037] The parsing module is used to parse the compressed encoding of the video to obtain the keyframes and motion vectors of the keyframes in the video;

[0038] The extraction module is used to extract entity visual representations from the keyframes, obtain action visual representations based on the entity visual representations and the motion vectors of the keyframes, and extract action text representations from the text descriptions corresponding to the video, as well as entity text representations corresponding to the action text representations.

[0039] The first fusion module is used to fuse the entity visual representation and the entity text representation to obtain a video entity multimodal representation, and to fuse the action visual representation and the action text representation to obtain a video action multimodal representation.

[0040] The second fusion module is used to fuse the video entity multimodal representation and the video action multimodal representation to obtain the multimodal representation of the video.

[0041] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the video multimodal representation learning method as described in the first aspect.

[0042] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video multimodal representation learning method as described in the first aspect.

[0043] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the video multimodal representation learning method as described in the first aspect.

[0044] The video multimodal representation learning method and apparatus provided by this invention first parses the compressed encoding of the video to obtain keyframes and motion vectors of the keyframes. Then, entity visual representations are extracted from the keyframes. Based on the entity visual representations and the motion vectors of the keyframes, action visual representations are obtained. That is, motion propagation is performed on the entity visual representations of the keyframes based on the motion vectors of the keyframes. Visual representations of the remaining frames in the video, excluding the keyframes, are obtained with minimal computational cost, significantly reducing the computational load of the visual representation extraction process. Then, entity visual representations and entity text representations are fused to obtain video entity multimodal representations. Action visual representations and action text representations are fused to obtain video action multimodal representations. Finally, video entity multimodal representations and video action multimodal representations are fused to obtain the video multimodal representation. In other words, the joint learning of spatiotemporal information in the video is decoupled into entity representation learning and action representation learning, reducing the complexity of spatiotemporal joint learning and reducing GPU utilization when understanding long videos. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0046] Figure 1 This is one of the flowcharts illustrating the video multimodal representation learning method provided by the present invention;

[0047] Figure 2 This is a schematic diagram of the video multimodal representation learning process in the video-text retrieval scenario provided by the present invention;

[0048] Figure 3 This is the second flowchart of the video multimodal representation learning method provided by the present invention;

[0049] Figure 4 This is a schematic diagram illustrating the principle of the graph self-attention network provided by the present invention;

[0050] Figure 5 This is a schematic diagram of the structure of the video multimodal representation learning device provided by the present invention;

[0051] Figure 6 This is a schematic diagram of the physical structure of the electronic device provided by the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0053] It should be noted that in the description of this invention, the terms "first," "second," etc., are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, the first object can be one or more.

[0054] The video multimodal representation learning method and apparatus provided by the present invention will be described exemplarily below with reference to the accompanying drawings.

[0055] Figure 1 This is one of the flowcharts illustrating the video multimodal representation learning method provided by this invention, such as... Figure 1 As shown, the method includes:

[0056] Step 100: Parse the compressed code of the video to obtain the keyframes in the video and the motion vectors of the keyframes;

[0057] Step 110: Extract entity visual representation from the keyframe, obtain action visual representation based on the entity visual representation and the motion vector of the keyframe, and extract action text representation and entity text representation corresponding to the action text representation from the text description corresponding to the video.

[0058] Step 120: Fuse the entity visual representation and the entity text representation to obtain a video entity multimodal representation, and fuse the action visual representation and the action text representation to obtain a video action multimodal representation.

[0059] Step 130: The video entity multimodal representation and the video action multimodal representation are fused to obtain the multimodal representation of the video.

[0060] It should be noted that the execution subject of the video multimodal representation learning method provided in this embodiment of the invention can be an electronic device, a component in an electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. For example, a mobile electronic device can be a mobile phone, tablet computer, laptop computer, PDA, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc., while a non-mobile electronic device can be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This embodiment of the invention does not specifically limit the specific implementation of these methods.

[0061] The following describes in detail the technical solution of the embodiments of the present invention, taking the computer execution of the video multimodal representation learning method provided by the present invention as an example.

[0062] Specifically, to overcome the high computational complexity of existing video multimodal representation learning methods for jointly modeling spatiotemporal information in videos, especially for long videos where GPU memory overflow can occur, hindering normal learning, this invention first parses the compressed encoding of the video to obtain keyframes and their motion vectors. Then, it extracts entity visual representations from the keyframes. Based on these entity visual representations and the motion vectors of the keyframes, it obtains action visual representations. In other words, it performs motion propagation on the entity visual representations of the keyframes based on their motion vectors, obtaining the action visual representations of the video with minimal computational cost. Visual representations of frames other than keyframes significantly reduce the computational cost of visual representation extraction. This allows for the fusion of entity visual representations and entity text representations to obtain video entity multimodal representations, and the fusion of action visual representations and action text representations to obtain video action multimodal representations. Finally, the fusion of video entity multimodal representations and video action multimodal representations yields the video's multimodal representation. In essence, the joint learning of spatiotemporal information in the video is decoupled into entity representation learning and action representation learning, reducing the complexity of spatiotemporal joint learning and decreasing GPU usage when understanding long videos.

[0063] Optionally, the compressed encoding of the video can be obtained first, and then the compressed encoding of the video can be parsed to obtain the keyframes and motion vectors of the keyframes in the video.

[0064] It should be noted that a keyframe is a subset of all frames in a video.

[0065] Optionally, after acquiring the keyframes, entity visual representations can be extracted from the keyframes.

[0066] It should be noted that, in the embodiments of the present invention, an entity refers to an object included in a frame image of a video, and the visual representation of an entity is some pixel features in the frame image that can represent an object.

[0067] Optionally, after extracting the entity visual representation, an action visual representation can be obtained based on the entity visual representation and the motion vectors of the keyframes. That is, the entity visual representation is propagated through motion based on the motion vectors of the keyframes to obtain the action visual representation.

[0068] Optionally, action text representations and entity text representations corresponding to the action text representations can be extracted from the text descriptions corresponding to the video.

[0069] Optionally, after extracting entity visual representation, motion visual representation, motion text representation and entity text representation, the entity visual representation and entity text representation can be fused to obtain video entity multimodal representation, and the motion visual representation and motion text representation can be fused to obtain video motion multimodal representation. Then, the video entity multimodal representation and video motion multimodal representation can be fused to obtain video multimodal representation.

[0070] It is understood that the embodiments of the present invention reduce the computational complexity of spatiotemporal information joint learning in training by transforming spatiotemporal joint learning in videos into spatial entity learning and temporal action learning. This can solve the problem of pre-training long sequence video frames under server memory constraints and achieve efficient training in large-scale video data scenarios.

[0071] Taking video-to-text retrieval as an example, Figure 2 This is a schematic diagram of the video multimodal representation learning process in the video-text retrieval scenario provided by the present invention, as shown below. Figure 2 As shown, entity features and action features are extracted from the video, and entity text features and action text features are extracted from the text. Then, the entity features extracted from the video and the entity text features extracted from the text are fused to obtain entity multimodal features, and the action features extracted from the video and the action text features extracted from the text are fused to obtain action multimodal features. Then, the entity multimodal features and action multimodal features are fused to obtain video-text multimodal features. The matching similarity of the video-text multimodal features can then be calculated for training.

[0072] The video multimodal representation learning method provided by this invention first parses the compressed encoding of the video to obtain keyframes and their motion vectors. Then, it extracts entity visual representations from the keyframes. Based on the entity visual representations and the motion vectors of the keyframes, it obtains action visual representations. That is, it propagates the motion of the entity visual representations of the keyframes based on their motion vectors, obtaining the visual representations of the remaining frames in the video (excluding keyframes) with minimal computational cost, significantly reducing the computational load of the visual representation extraction process. Then, it fuses the entity visual representations and entity text representations to obtain the video entity multimodal representation, and fuses the action visual representations and action text representations to obtain the video action multimodal representation. Finally, it fuses the video entity multimodal representations and the video action multimodal representations to obtain the video multimodal representation. In other words, it decouples the joint learning of spatiotemporal information in the video into entity representation learning and action representation learning, reducing the complexity of spatiotemporal joint learning and reducing GPU utilization when understanding long videos.

[0073] Optionally, parsing the compressed encoding of the video to obtain keyframes and motion vectors of the keyframes includes:

[0074] The compressed encoding of the video is parsed to obtain keyframes in the video, motion vectors of the forward frames corresponding to each keyframe, and motion vectors of the bidirectional frames corresponding to each forward frame.

[0075] It's important to note that video frames often contain a significant amount of redundant information. In video compression coding, frames are typically categorized as: keyframes (intra-coded pictures, I-frames), predictive coded pictures (P-frames), and bidirectionally predictive coded pictures (B-frames). Keyframes serve as reference images, essentially fixed images independent of other image types. Each group of images begins with an image of this type, and they are encoded independently, using only intra-frame coding techniques. Therefore, decoding does not reference other frames, similar to JPEG encoding. Predictive frames contain difference information from previous I or P frames. During encoding, motion compensation and motion estimation are used, employing forward estimation and referencing previous I or P frames to predict the P frame. Bidirectionally predictive frames contain difference information from previous and / or subsequent I or P frames. Encoding also uses motion compensation and motion estimation, employing forward, backward, or bidirectional estimation, primarily referencing preceding or following I or P frames. To reduce the encoding of redundant information, video encoding is usually divided into keyframes and motion vectors. During decoding, the keyframes are decoded first, and then the motion vectors are superimposed on the keyframes to obtain the forward frames. The motion vectors are superimposed on the forward frames to obtain the bidirectional frames.

[0076] Therefore, the compressed encoding of the video can be parsed to obtain the keyframes in the video, the motion vectors of the forward frames corresponding to each keyframe, and the motion vectors of the bidirectional frames corresponding to each forward frame.

[0077] Optionally, obtaining the action visual representation based on the entity visual representation and the motion vectors of the keyframe includes:

[0078] Based on the entity visual representation and the motion vector of the forward frame, a first action visual representation is obtained;

[0079] Based on the first action visual representation and the motion vector of the bidirectional frame, a second action visual representation is obtained;

[0080] Accordingly, fusing the visual representation of the action and the textual representation of the action to obtain a multimodal representation of the video action includes:

[0081] The first visual representation of the action, the second visual representation of the action, and the textual representation of the action are fused to obtain the multimodal representation of the video action.

[0082] Specifically, in this embodiment of the invention, in order to obtain an action visual representation based on the entity visual representation and the motion vector of the key frame, a first action visual representation can be obtained first based on the entity visual representation and the motion vector of the forward frame, and then a second action visual representation can be obtained based on the first action visual representation and the motion vector of the bidirectional frame.

[0083] It should be noted that, in the embodiments of the present invention, motion propagation is performed on the entity visual representation based on the motion vector of the forward frame to obtain the first action visual representation, and then motion propagation is performed on the first action visual representation based on the motion vector of the bidirectional frame to obtain the second action visual representation.

[0084] It is understood that the embodiments of the present invention propagate keyframe features through motion vectors to obtain the visual representation of the remaining frames of the video with a low computational cost.

[0085] Specifically, in this embodiment of the invention, after obtaining the first action visual representation, the second action visual representation, and the action text representation, the first action visual representation, the second action visual representation, and the action text representation can be fused to obtain a video action multimodal representation.

[0086] It should be noted that in video coding, only 20% of the frames are typically keyframes, while the rest are forward and bidirectional frames for keyframe motion propagation. Therefore, this embodiment of the invention reduces the visual representation extraction of 80% of the frames in the video by using feature propagation, thereby further significantly reducing the amount of computation.

[0087] Optionally, fusing the first visual representation of the action, the second visual representation of the action, and the textual representation of the action to obtain the multimodal representation of the video action includes:

[0088] Spatially downsample the first action visual representation to obtain the third action visual representation, and spatially downsample the second action visual representation to obtain the fourth action visual representation.

[0089] The third action visual representation, the fourth action visual representation, and the action text representation are fused to obtain the video action multimodal representation.

[0090] Specifically, in this embodiment of the invention, after obtaining the first action visual representation and the second action visual representation, the first action visual representation and the second action visual representation can be spatially downsampled to obtain the third action visual representation and the fourth action visual representation. Then, the third action visual representation, the fourth action visual representation and the previously obtained action text representation are fused to obtain the video action multimodal representation.

[0091] Optionally, in this embodiment of the invention, the sampling rate for spatial downsampling of the first action visual representation and the second action visual representation can be adaptively set based on the actual application. For example, the sampling rate can be 2 Hz, etc. This embodiment of the invention does not specifically limit this.

[0092] It is understood that, in the embodiments of the present invention, for the multimodal representation learning of video actions, the visual features of key frames are first propagated using motion vectors to obtain the visual features of forward and bidirectional frames in the video with minimal computation, which greatly reduces the computational load of the visual representation extraction process. Secondly, the extracted action visual representations are subjected to spatial downsampling processing, which further reduces the computational complexity of action information learning.

[0093] Optionally, extracting entity visual representations from the keyframes includes:

[0094] Based on a pre-built visual attention network, entity visual features are extracted from each of the keyframes;

[0095] Determine the cumulative similarity of entity visual features of adjacent keyframes, and construct a cumulative similarity curve based on the cumulative similarity;

[0096] The target keyframes corresponding to the first preset number of positions on the cumulative similarity curve with the largest slope change are determined, and the entity visual features extracted from the target keyframes are used as the entity visual representation.

[0097] Specifically, in this embodiment of the invention, in order to extract entity visual representations from keyframes, entity visual features can first be extracted from each keyframe based on a pre-constructed visual attention network. Then, the cumulative similarity of entity visual features of adjacent keyframes is determined, and a cumulative similarity curve is constructed based on the cumulative similarity. Then, the target keyframes corresponding to the first preset number of positions on the cumulative similarity curve with the largest slope change are determined, and the entity visual features extracted from the target keyframes are used as entity visual representations extracted from all keyframes.

[0098] It should be noted that the visual attention network in the embodiments of the present invention can be any kind of visual attention network in the related technologies, and the embodiments of the present invention do not specifically limit it.

[0099] Alternatively, the cumulative similarity of entity visual features in adjacent keyframes can be determined by calculating the cumulative Euclidean distance.

[0100] Optionally, the preset number in the embodiments of the present invention can be adaptively set based on actual applications, for example, it can be 3, 4, or 5, etc., and the embodiments of the present invention do not specifically limit it in this regard.

[0101] For example, based on the cumulative similarity curve, the four target keyframes corresponding to the four positions with the largest slope changes on the cumulative similarity curve can be determined, and the entity visual features extracted from these four target keyframes can be used as entity visual representations for subsequent learning.

[0102] It is understood that, for video entity multimodal information learning, the embodiments of the present invention only extract the visual representation of key frames in video compression coding, and use the cumulative similarity curve of visual representations between key frames to further remove redundant information, thereby further reducing the computational complexity of visual information learning.

[0103] Optionally, the step of extracting action text representations and entity text representations corresponding to the action text representations from the text descriptions corresponding to the video includes:

[0104] Based on a pre-built text feature model, text representations are extracted from the text descriptions corresponding to the video.

[0105] Extract the action text representation, the subject text representation corresponding to the subject of the action corresponding to the action text representation, and the receptor text representation corresponding to the receptor of the action corresponding to the action text representation from the text representation, and use the subject text representation and the receptor text representation as the entity text representation.

[0106] Specifically, in this embodiment of the invention, in order to extract action text representations and entity text representations corresponding to the action text representations from the text descriptions corresponding to the video, text representations can first be extracted from the text descriptions corresponding to the video based on a pre-built text feature model. Then, action text representations, subject text representations corresponding to the subject of the action text representations, and receptor text representations corresponding to the receptors of the action text representations can be extracted from the text representations. The subject text representations and receptor text representations are used as entity text representations extracted from the text descriptions.

[0107] Optionally, the text feature model can be constructed from any feature extraction network in related technologies that can extract text features, and the embodiments of the present invention do not specifically limit this.

[0108] Optionally, fusing the entity visual representation and the entity text representation to obtain a video entity multimodal representation includes:

[0109] The entity visual representation and the entity text representation are concatenated to obtain the entity representation;

[0110] The entity representation is input into a pre-constructed self-attention network to obtain the video entity multimodal representation output by the self-attention network.

[0111] Specifically, in this embodiment of the invention, in order to fuse entity visual representation and entity text representation to obtain video entity multimodal representation, entity visual representation and entity text representation can be first spliced ​​together to obtain entity representation, and then the entity representation can be input into a pre-constructed self-attention network to obtain the video entity multimodal representation output by the self-attention network.

[0112] Optionally, fusing the visual representation of the action and the textual representation of the action to obtain a multimodal representation of the video action includes:

[0113] The visual representation of the action and the textual representation of the action are concatenated to obtain the action representation;

[0114] The action representation is input into a pre-constructed self-attention network to obtain the video action multimodal representation output by the self-attention network.

[0115] Specifically, in this embodiment of the invention, in order to fuse the visual representation of action and the textual representation of action to obtain a multimodal representation of video action, the visual representation of action and the textual representation of action can be spliced ​​together to obtain the action representation. Then, the action representation is input into a pre-constructed self-attention network to obtain the multimodal representation of video action output by the self-attention network.

[0116] Optionally, fusing the video entity multimodal representation and the video action multimodal representation to obtain the multimodal representation of the video includes:

[0117] Based on the video entity multimodal representation and the video action multimodal representation, an action-entity graph is constructed. Each node in the action-entity graph is composed of each entity corresponding to the video entity multimodal representation and each action corresponding to the video action multimodal representation. Each edge in the action-entity graph is composed of each action and the connecting line between each action and the entity corresponding to each action.

[0118] The action-entity graph, the video entity multimodal representation, and the video action multimodal representation are input into a pre-constructed graph self-attention network to obtain the multimodal representation of the video output by the graph self-attention network.

[0119] Specifically, in this embodiment of the invention, in order to fuse video entity multimodal representation and video action multimodal representation to obtain a multimodal representation of the video, an action-entity graph can first be constructed based on the video entity multimodal representation and the video action multimodal representation. Then, the action-entity graph, the video entity multimodal representation, and the video action multimodal representation are input into a pre-constructed graph self-attention network to obtain the multimodal representation of the video output by the graph self-attention network. In this graph, each node is composed of each entity corresponding to the video entity multimodal representation and each action corresponding to the video action multimodal representation. Each edge in the action-entity graph is composed of each action and the connecting line between each action and the entity corresponding to each action.

[0120] Understandably, by inputting the constructed action-entity graph into a pre-constructed graph attention network, the graph attention network can perform structured learning on each action and entity in the video, thereby extracting multimodal representations of the video.

[0121] Figure 3 This is the second flowchart of the video multimodal representation learning method provided by the present invention, as shown below. Figure 3As shown, the entire method comprises three branches: entity branch, action branch, and text branch. For the entity branch, keyframes from the input video encoding are processed through a self-attention network and a feature sparsity module to obtain visual representations (entity visual representations) focusing on objects in the video. The action branch propagates and spatially downsamples the visual features of the keyframes to obtain the video's action visual representation. The text branch extracts entity text representations and verb text representations from the input text. Then, through multimodal interaction, the text entity representations and entity visual representations are fused to obtain the video entity multimodal representation, and the text action representations and action visual representations are fused to obtain the video action multimodal representation. Finally, through an action-entity graph attention network, actions and entities interact to obtain the video's multimodal representation. Average pooling and loss function calculations can then be performed on the video's multimodal representation for training.

[0122] The video multimodal representation learning method provided by the present invention will be exemplarily described below through a specific embodiment.

[0123] Step 1: Parse the compressed encoding of the input video v to obtain the keyframes I = {I0, I1, ..., I...} in the compressed encoding. N Each keyframe corresponds to the motion vector of the preceding frame. and the motion vectors of the forward frame corresponding to the bidirectional frame

[0124] Step 2: Extract the visual representation of entities from the video from the keyframes. The specific process is as follows:

[0125] First, based on a pre-built visual attention network Extracting visual representations F = {f0, f1, ..., f1} from keyframes. T The expression is as follows:

[0126]

[0127] Secondly, based on the obtained visual representations, the cumulative Euclidean distance between the visual representations of adjacent frames is calculated, resulting in the cumulative Euclidean distance {D0,…,D...}. T The cumulative Euclidean distance calculation expression is as follows:

[0128]

[0129] The visual features of the four frames with the largest changes in slope on the cumulative similarity curve can be retained for entity visual representation learning. This step removes redundant keyframes, reducing the computational load of subsequent entity representation learning branches.

[0130] Step 3: Perform motion propagation on the entity visual representations of the keyframes to obtain the motion representation (action visual representation) of the video. The specific process is as follows:

[0131] First, use motion vectors For keyframe I t ∈{I0,I1,...,I N The entity visual representation is propagated to obtain I. t Visual representation of the forward frame in:

[0132]

[0133] Similarly, using motion vectors The visual representation of the forward frame is propagated to obtain Visual representation of corresponding bidirectional frames in:

[0134]

[0135] Step 4: Extract textual representations of actions and corresponding entities from the video. The process is as follows:

[0136] First, based on a pre-built text feature extraction model, text representation E is extracted from the text description corresponding to the input video.

[0137] Secondly, extract the text representations of actions in the text A = {α1, α2, ... α} from the text representation E. i}, and the text representation M = {m1, m2, ... m} of the subject corresponding to the action. i The textual representation of the receptor R = {r1, r2, ... r} i}, where <α1,m1,r1> is a triple of <action, subject, receptor>.

[0138] Step 5: Fuse the visual representation of the video entity with the textual representation of the entity. The process is as follows:

[0139] First, the entity text representations M and R extracted in step 4 are compared with the sparse keyframe entity visual representations obtained in step 2. By concatenating these components, we obtain the video entity multimodal representation EN, where:

[0140]

[0141] Secondly, the pre-built self-attention network As a multimodal feature learning module for entities, it is as follows:

[0142]

[0143] It should be noted that, in order to reduce the amount of computation, Only the spatial attention matrix is ​​calculated, that is, only the attention matrix within the same frame is calculated, and the interaction between frames is not calculated.

[0144] Step 6: Fuse the visual representation of the video action with the textual representation of the action. The process is as follows:

[0145] First, the visual representations of the forward frame and the bidirectional frame obtained in step 3 are downsampled by a factor of 2, and then concatenated with the action text obtained in step 4 to obtain the video action multimodal representation (ACT).

[0146]

[0147] Secondly, the pre-built self-attention network As a multimodal action feature learning module, it is as follows:

[0148]

[0149] It should be noted that, in order to reduce the amount of computation, Only the temporal dimension attention matrix is ​​calculated, that is, only the attention matrix between different frames is calculated, and the spatial attention matrix within the same frame is not calculated.

[0150] Step 7: Fuse the video entity multimodal representation and the video action multimodal representation to obtain the video multimodal representation. The process is as follows:

[0151] First, construct an action-entity graph G, with actions and entities as nodes and the connecting lines between actions and entities as edges.

[0152]

[0153] Understandably, graph G only connects actions with their corresponding entities, without creating connections between other nodes.

[0154] Secondly, construct a graph self-attention network. Learn the action-entity graph G to extract the structured representation FV of the video:

[0155]

[0156] Figure 4 This is a schematic diagram illustrating the principle of the graph self-attention network provided by the present invention, as shown below. Figure 4 As shown, in this embodiment of the invention, the weights of the edges in the graph are used to weight the attention matrix attn calculated at each layer of the graph self-attention network. The weighting method is as follows:

[0157] attn = sofmax(q*k)*v;

[0158]

[0159] Where q represents the query vector, k represents the key vector, and v represents the value vector.

[0160] Step 8: Calculate the loss function for the obtained video multimodal representation and perform training. The process is as follows:

[0161] First, the video multimodal representation obtained in step 7 is subjected to average pooling, as follows:

[0162]

[0163] Where Avg is the average pooling layer, and its output is... It is a one-dimensional vector.

[0164] Secondly, Input to a binary classifier for classification, output The probability of matching video and text.

[0165] Finally, the cross-entropy loss function is used to analyze the probability. Calculate the loss function and train the entire model.

[0166] It should be noted that, given the same input video sequence length, the method provided in this embodiment of the invention can reduce the computational cost of existing video multimodal representation learning methods by more than 20 times, and complete the training and inference of the large-scale public video understanding dataset MSRVTT on a single V100 GPU (32GB VRAM). With the same computational cost, due to its low computational complexity, the method provided in this embodiment of the invention can extract representations from longer video sequences, significantly outperforming existing methods in long video understanding.

[0167] The video multimodal representation learning method provided by this invention first parses the compressed encoding of the video to obtain keyframes and their motion vectors. Then, it extracts entity visual representations from the keyframes. Based on the entity visual representations and the motion vectors of the keyframes, it obtains action visual representations. That is, it propagates the motion of the entity visual representations of the keyframes based on their motion vectors, obtaining the visual representations of the remaining frames in the video (excluding keyframes) with minimal computational cost, significantly reducing the computational load of the visual representation extraction process. Then, it fuses the entity visual representations and entity text representations to obtain the video entity multimodal representation, and fuses the action visual representations and action text representations to obtain the video action multimodal representation. Finally, it fuses the video entity multimodal representations and the video action multimodal representations to obtain the video multimodal representation. In other words, it decouples the joint learning of spatiotemporal information in the video into entity representation learning and action representation learning, reducing the complexity of spatiotemporal joint learning and reducing GPU utilization when understanding long videos.

[0168] The video multimodal representation learning device provided by the present invention will be described below. The video multimodal representation learning device described below and the video multimodal representation learning method described above can be referred to in correspondence.

[0169] Figure 5 This is a schematic diagram of the structure of the video multimodal representation learning device provided by the present invention, as shown below. Figure 5 As shown, the device includes: a parsing module 510, an extraction module 520, a first fusion module 530, and a second fusion module 540; wherein:

[0170] The parsing module 510 is used to parse the compressed code of the video to obtain the key frames in the video and the motion vectors of the key frames;

[0171] The extraction module 520 is used to extract entity visual representations from the keyframes, obtain action visual representations based on the entity visual representations and the motion vectors of the keyframes, and extract action text representations from the text descriptions corresponding to the video, as well as entity text representations corresponding to the action text representations.

[0172] The first fusion module 530 is used to fuse the entity visual representation and the entity text representation to obtain a video entity multimodal representation, and to fuse the action visual representation and the action text representation to obtain a video action multimodal representation.

[0173] The second fusion module 540 is used to fuse the video entity multimodal representation and the video action multimodal representation to obtain the multimodal representation of the video.

[0174] The video multimodal representation learning device provided by this invention first parses the compressed encoding of the video to obtain keyframes and motion vectors of the keyframes. Then, it extracts entity visual representations from the keyframes. Based on the entity visual representations and the motion vectors of the keyframes, it obtains action visual representations. That is, it performs motion propagation on the entity visual representations of the keyframes based on the motion vectors of the keyframes, obtaining the visual representations of the remaining frames in the video except for the keyframes with minimal computational cost, significantly reducing the computational load of the visual representation extraction process. Then, it fuses the entity visual representations and entity text representations to obtain the video entity multimodal representation, and fuses the action visual representations and action text representations to obtain the video action multimodal representation. Finally, it fuses the video entity multimodal representations and the video action multimodal representations to obtain the video multimodal representation. In other words, it decouples the joint learning of spatiotemporal information in the video into entity representation learning and action representation learning, reducing the complexity of spatiotemporal joint learning and reducing GPU utilization when understanding long videos.

[0175] It should be noted that the video multimodal representation learning device provided in this embodiment of the invention can implement all the method steps implemented in the above-mentioned video multimodal representation learning method embodiment, and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0176] Figure 6 This is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute the video multimodal representation learning method provided by the above methods, which includes:

[0177] The compressed encoding of the video is parsed to obtain the keyframes in the video and the motion vectors of the keyframes;

[0178] Entity visual representations are extracted from the keyframes. Based on the entity visual representations and the motion vectors of the keyframes, action visual representations are obtained. Action text representations are extracted from the text descriptions corresponding to the video, as well as entity text representations corresponding to the action text representations.

[0179] The entity visual representation and the entity text representation are fused to obtain a video entity multimodal representation, and the action visual representation and the action text representation are fused to obtain a video action multimodal representation.

[0180] The multimodal representation of the video entity and the multimodal representation of the video action are fused to obtain the multimodal representation of the video.

[0181] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0182] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the video multimodal representation learning method provided by the above methods, the method comprising:

[0183] The compressed encoding of the video is parsed to obtain the keyframes in the video and the motion vectors of the keyframes;

[0184] Entity visual representations are extracted from the keyframes. Based on the entity visual representations and the motion vectors of the keyframes, action visual representations are obtained. Action text representations are extracted from the text descriptions corresponding to the video, as well as entity text representations corresponding to the action text representations.

[0185] The entity visual representation and the entity text representation are fused to obtain a video entity multimodal representation, and the action visual representation and the action text representation are fused to obtain a video action multimodal representation.

[0186] The multimodal representation of the video entity and the multimodal representation of the video action are fused to obtain the multimodal representation of the video.

[0187] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the video multimodal representation learning methods provided above, the method comprising:

[0188] The compressed encoding of the video is parsed to obtain the keyframes in the video and the motion vectors of the keyframes;

[0189] Entity visual representations are extracted from the keyframes. Based on the entity visual representations and the motion vectors of the keyframes, action visual representations are obtained. Action text representations are extracted from the text descriptions corresponding to the video, as well as entity text representations corresponding to the action text representations.

[0190] The entity visual representation and the entity text representation are fused to obtain a video entity multimodal representation, and the action visual representation and the action text representation are fused to obtain a video action multimodal representation.

[0191] The multimodal representation of the video entity and the multimodal representation of the video action are fused to obtain the multimodal representation of the video.

[0192] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0193] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0194] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video multimodal representation learning method, characterized in that, include: The compressed encoding of the video is parsed to obtain the keyframes in the video and the motion vectors of the keyframes; Entity visual representations are extracted from the keyframes. Based on the entity visual representations and the motion vectors of the keyframes, action visual representations are obtained. Action text representations are extracted from the text descriptions corresponding to the video, as well as entity text representations corresponding to the action text representations. The entity visual representation and the entity text representation are fused to obtain a video entity multimodal representation, and the action visual representation and the action text representation are fused to obtain a video action multimodal representation. The video entity multimodal representation and the video action multimodal representation are fused to obtain the multimodal representation of the video; The step of parsing the compressed encoding of the video to obtain keyframes and motion vectors of the keyframes includes: The compressed encoding of the video is parsed to obtain keyframes in the video, motion vectors of forward frames corresponding to each keyframe, and motion vectors of bidirectional frames corresponding to each forward frame. The process of obtaining an action visual representation based on the entity visual representation and the motion vectors of the keyframes includes: Based on the entity visual representation and the motion vector of the forward frame, a first action visual representation is obtained; Based on the first action visual representation and the motion vector of the bidirectional frame, a second action visual representation is obtained; Accordingly, fusing the visual representation of the action and the textual representation of the action to obtain a multimodal representation of the video action includes: The first visual representation of the action, the second visual representation of the action, and the textual representation of the action are fused to obtain the multimodal representation of the video action.

2. The video multimodal representation learning method according to claim 1, characterized in that, The step of fusing the first visual representation of the action, the second visual representation of the action, and the textual representation of the action to obtain the multimodal representation of the video action includes: Spatially downsample the first action visual representation to obtain the third action visual representation, and spatially downsample the second action visual representation to obtain the fourth action visual representation. The third action visual representation, the fourth action visual representation, and the action text representation are fused to obtain the video action multimodal representation.

3. The video multimodal representation learning method according to claim 1, characterized in that, The extraction of entity visual representations from the keyframes includes: Based on a pre-built visual attention network, entity visual features are extracted from each of the keyframes; Determine the cumulative similarity of entity visual features of adjacent keyframes, and construct a cumulative similarity curve based on the cumulative similarity; The target keyframes corresponding to the first preset number of positions on the cumulative similarity curve with the largest slope change are determined, and the entity visual features extracted from the target keyframes are used as the entity visual representation.

4. The video multimodal representation learning method according to claim 1, characterized in that, The step of extracting action text representations from the text description corresponding to the video, and entity text representations corresponding to the action text representations, includes: Based on a pre-built text feature model, text representations are extracted from the text descriptions corresponding to the video. Extract the action text representation, the subject text representation corresponding to the subject of the action corresponding to the action text representation, and the receptor text representation corresponding to the receptor of the action corresponding to the action text representation from the text representation, and use the subject text representation and the receptor text representation as the entity text representation.

5. The video multimodal representation learning method according to claim 1, characterized in that, The step of fusing the entity visual representation and the entity text representation to obtain a video entity multimodal representation includes: The entity visual representation and the entity text representation are concatenated to obtain the entity representation; The entity representation is input into a pre-constructed self-attention network to obtain the video entity multimodal representation output by the self-attention network.

6. The video multimodal representation learning method according to claim 1, characterized in that, The process of fusing the visual representation of the action and the textual representation of the action to obtain a multimodal representation of video action includes: The visual representation of the action and the textual representation of the action are concatenated to obtain the action representation; The action representation is input into a pre-constructed self-attention network to obtain the video action multimodal representation output by the self-attention network.

7. The video multimodal representation learning method according to claim 1, characterized in that, The step of fusing the video entity multimodal representation and the video action multimodal representation to obtain the multimodal representation of the video includes: Based on the video entity multimodal representation and the video action multimodal representation, an action-entity graph is constructed. Each node in the action-entity graph is composed of each entity corresponding to the video entity multimodal representation and each action corresponding to the video action multimodal representation. Each edge in the action-entity graph is composed of each action and the connecting line between each action and the entity corresponding to each action. The action-entity graph, the video entity multimodal representation, and the video action multimodal representation are input into a pre-constructed graph self-attention network to obtain the multimodal representation of the video output by the graph self-attention network.

8. A video multimodal representation learning device, characterized in that, include: The parsing module is used to parse the compressed encoding of the video to obtain the keyframes and motion vectors of the keyframes in the video; The extraction module is used to extract entity visual representations from the keyframes, obtain action visual representations based on the entity visual representations and the motion vectors of the keyframes, and extract action text representations from the text descriptions corresponding to the video, as well as entity text representations corresponding to the action text representations. The first fusion module is used to fuse the entity visual representation and the entity text representation to obtain a video entity multimodal representation, and to fuse the action visual representation and the action text representation to obtain a video action multimodal representation. The second fusion module is used to fuse the video entity multimodal representation and the video action multimodal representation to obtain the multimodal representation of the video. The parsing module is specifically used for: The compressed encoding of the video is parsed to obtain keyframes in the video, motion vectors of forward frames corresponding to each keyframe, and motion vectors of bidirectional frames corresponding to each forward frame. The extraction module is specifically used for: Based on the entity visual representation and the motion vector of the forward frame, a first action visual representation is obtained; Based on the first action visual representation and the motion vector of the bidirectional frame, a second action visual representation is obtained; Accordingly, fusing the visual representation of the action and the textual representation of the action to obtain a multimodal representation of the video action includes: The first visual representation of the action, the second visual representation of the action, and the textual representation of the action are fused to obtain the multimodal representation of the video action.

Images