Video representation method, video classification method, electronic device and storage medium

Abstract

The present invention discloses a video characterization method, a video classification method, an electronic device and a storage medium, wherein the video characterization method comprises: obtaining key frames and video clips of a video to be characterized, wherein the number of the key frames and the video clips are both one or more; generating corresponding frame vectors based on the key frames, and generating global feature information according to the frame vectors; generating corresponding video clip vectors based on the video clips, and generating local feature information according to the video clip vectors; constructing graph structure features of the frame vectors and the video clip vectors; fusing the global feature information, the local feature information and the graph structure features to obtain a video feature representation of the video to be characterized. The present invention can improve the accuracy of video characterization.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing based on deep learning, and in particular to a video characterization method, a video classification method, an electronic device and a storage medium. Background Art

[0002] With the rapid development of the Internet, a series of Internet technologies have had a significant impact on people's lives. In particular, with the improvement of Internet broadband and the popularization of 5G networks, video has gradually become an important carrier for users to spread information. At the same time, as new media such as short videos and live broadcasts become more and more active, a large amount of video information has also been generated. How to classify these videos is very necessary and challenging for the effective management of user videos.

[0003] In the task of video classification, how to accurately and effectively represent the video is the key to accurate video classification. In the prior art, there is a keyframe-based video classification method. As the name implies, it obtains an effective representation of the video from the keyframes of the video by sampling a continuous sequence of video frames to obtain several representative video frames, and then uses these video frames for classification. This video classification method has a fast processing speed, but the classification effect is average because it only needs to process part of the video frames. However, videos often carry complex content. If only part of the video frame information is considered, the video content cannot be accurately understood. Summary of the invention

[0004] The problem solved by the present invention is that the existing video representation method has the disadvantage of low representation accuracy.

[0005] The present invention proposes a video representation method, comprising:

[0006] Acquire key frames and video clips of the video to be represented, wherein the number of the key frames and the number of the video clips are both one or more;

[0007] Generate a corresponding frame vector based on the key frame, and generate global feature information according to the frame vector;

[0008] generating a corresponding video segment vector based on the video segment, and generating local feature information according to the video segment vector;

[0009] Constructing graph structure features of the frame vector and the video clip vector;

[0010] The global feature information, the local feature information and the graph structure feature are integrated to obtain a video feature representation of the video to be represented.

[0011] Optionally, the global feature information is generated by a Transformer network based on time-space separation, and the Transformer network based on time-space separation includes a first temporal Transformer module and a plurality of spatial Transformer modules; the corresponding frame vector is generated based on the key frame, and the global feature information is generated according to the frame vector.

[0012] Divide each of the key frames into a plurality of image sub-blocks, and then input the image sub-blocks of each of the key frames into the spatial Transformer module corresponding to the key frame to obtain a frame vector corresponding to each of the key frames;

[0013] All the frame vectors are input into the first temporal Transformer module to obtain the global feature information.

[0014] Optionally, the local feature information is generated by using a preset video extraction network, the preset video extraction network includes a second temporal Transformer module and a primary feature extraction module based on an optical flow network; the generating a corresponding video segment vector based on the video segment, and generating the local feature information according to the video segment vector includes:

[0015] Inputting each of the video clips into the primary feature extraction module to obtain a video clip vector corresponding to each of the video clips;

[0016] All the video clip vectors are input into the second temporal Transformer module to obtain the local feature information.

[0017] Optionally, obtaining key frames and video clips of the video to be characterized includes:

[0018] Obtaining key frames of the video to be represented;

[0019] The video segment is obtained from the video to be represented according to the key frame.

[0020] Optionally, the graph structure features for constructing the frame vector and the video clip vector include:

[0021] Each feature vector is regarded as a vertex, and the similarity between each feature vector is regarded as an edge, so as to construct a first fully connected graph; wherein the feature vector is one of the frame vector and the video segment vector;

[0022] According to each vertex in the first fully connected graph and all the vertices and edges connected to it, generate a fusion feature corresponding to each vertex;

[0023] The graph attention mechanism is used to obtain the weight of each vertex, and the weight of each vertex is multiplied and accumulated with the corresponding fusion feature to obtain the graph structure feature of the feature vector.

[0024] Optionally, the key frame and the video segment have a one-to-one correspondence; the graph structure features for constructing the frame vector and the video segment vector include:

[0025] Considering each of the frame vectors as a vertex, the similarity between each of the frame vectors as a first edge, and the similarity between each of the video clip vectors as a second edge, to construct a second fully connected graph;

[0026] According to each vertex in the second fully connected graph and all the vertices and edges connected to it, generate a fusion feature corresponding to each vertex;

[0027] The graph attention mechanism is used to obtain the weight of each vertex, and the weight of each vertex is multiplied and accumulated with the corresponding fusion feature to obtain the graph structure features of the frame vector and the video clip vector.

[0028] Optionally, the fusing the global feature information, the local feature information and the graph structure feature to obtain a video feature representation of the video to be represented includes:

[0029] The global feature information, the local feature information and the graph structure feature are concatenated, and then the attention mechanism is used to obtain the dynamic weights of the global feature information, the local feature information and the graph structure feature;

[0030] The global feature information, the local feature information, the graph structure feature, and the dynamic weight are respectively multiplied to obtain a video feature representation of the video to be represented.

[0031] The present invention also proposes a video classification method, comprising:

[0032] Get the video to be classified;

[0033] Characterizing the video to be classified using the video characterization method as described above, and generating a video feature representation of the video to be classified;

[0034] The video to be classified is classified based on the video feature representation of the video to be classified.

[0035] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is read and executed by a processor, the video characterization method or the video classification method described above is implemented.

[0036] The present invention further provides an electronic device, comprising a computer-readable storage medium storing a computer program and a processor, wherein when the computer program is read and executed by the processor, the video characterization method or the video classification method described above is implemented.

[0037] The present invention obtains key frames and video clips of the video to be represented, uses the key frames to model the global feature information of the video, uses local video clips to learn the fine-grained local action information of the video, and then based on the frame vectors corresponding to the key frames and the video clip vectors corresponding to the video clips, uses a graph structure to model the complex association between the frame vectors and the video clip vectors, and generates a graph-based graph structure feature. By combining the global feature information, the fine-grained local action information and the graph structure feature, a video feature representation is obtained, which can ensure a faster representation speed and a higher representation accuracy. Video classification based on the video feature representation can also take into account both classification accuracy and processing speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of a process of a video characterization method according to an embodiment of the present invention;

[0039] Figure 2 It is a structural schematic diagram of a classification network based on the fusion of spatiotemporal separation Transformer and optical flow network in the video representation method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0041] like Figure 1 In one embodiment of the present invention, the video characterization method includes:

[0042] Step S100: obtaining key frames and video clips of a video to be represented, wherein the number of the key frames and the number of the video clips are both one or more.

[0043] When extracting key frames of a video, existing key frame extraction methods can be used to extract key frames of the video to be represented. For example, firstly, a deep neural network is used to extract the deep features of each video frame, and K-means clustering is performed on it to obtain K clusters of video frames. In each video frame cluster, the clearest video frame is selected by definition screening as the extracted key frame.

[0044] The video segments of the video to be characterized can be obtained from the video to be characterized according to the key frames after obtaining the key frames of the video to be characterized. In one embodiment, the key frames are used as nodes to segment the video to be characterized, and multiple video segments are obtained as the video segments here. In another embodiment, the key frames are used as the middle position, and video segments of several lengths are selected on the left and right as the video segments here. The video segments thus obtained are used to extract local feature information, which can capture fine-grained actions while taking into account processing speed.

[0045] Step S200: Generate a corresponding frame vector based on the key frame, and generate global feature information according to the frame vector.

[0046] Global features refer to the overall properties of an image, such as color features, texture features, and shape features. Global feature information can be extracted from key frames. A video contains multiple frames of images, and a key frame is one or more of them. In an optional implementation, when a key frame is a single frame of image, the global features extracted from the key frame are used as the global feature information here. When a key frame is a multi-frame image, features are first extracted from each key frame, and then the features extracted from all key frames are fused to obtain global feature information.

[0047] Furthermore, a convolutional neural network algorithm may be used to extract global feature information from key frames, or a Transformer algorithm may be used to extract global feature information from key frames.

[0048] Step S300: Generate a corresponding video segment vector based on the video segment, and generate local feature information according to the video segment vector.

[0049] Local features refer to fine-grained features extracted from images, including edges, corners, lines, curves and other features. It is difficult to obtain fine-grained features from key frames, which limits the accuracy of video feature representation, so local feature information is extracted from video clips. The video clips to be characterized may be one or more. In an optional implementation, when there is one video clip, the local features extracted from the video clip are used as the local feature information here. When there are multiple video clips, features are first extracted from each video clip, and then the features extracted from all video clips are fused to obtain local feature information.

[0050] Furthermore, an optical flow-based method may be used to extract local feature information from a video clip. For example, FlowNet or SpyNet may be used to extract local feature information from a video clip.

[0051] Step S400: constructing graph structure features of the frame vector and the video clip vector.

[0052] Because the relationships between video contents are relatively complex, Transformer is not sufficient to capture these complex relationships. Therefore, the present invention adopts a graph structure to model the complex association relationships between frame vectors and video segment vectors, so as to more accurately represent the video.

[0053] Specifically, a graph attention network can be used to construct graph structure features between each frame vector and each video clip vector to obtain a feature vector that can capture complex video content relationships.

[0054] Step S500: The global feature information, the local feature information and the graph structure feature are integrated to obtain a video feature representation of the video to be represented.

[0055] Weights can be dynamically assigned to global feature information, local feature information and graph structure features to obtain dynamic coordinated video features as the final video feature representation.

[0056] The present invention obtains key frames and video clips of the video to be represented, uses the key frames to model the global feature information of the video, uses local video clips to learn the fine-grained local action information of the video, and then based on the frame vectors corresponding to the key frames and the video clip vectors corresponding to the video clips, uses a graph structure to model the complex association between the frame vectors and the video clip vectors, and generates a graph-based graph structure feature. By combining the global feature information, the fine-grained local action information and the graph structure feature, a video feature representation is obtained, which can ensure a faster representation speed and a higher representation accuracy. Video classification based on the video feature representation can also take into account both classification accuracy and processing speed.

[0057] Optionally, the global feature information is generated using a Transformer network based on time-space separation, such as Figure 2 The Transformer network based on time-space separation includes a first temporal Transformer module and a plurality of spatial Transformer modules, wherein the number of the spatial Transformer modules is a preset value.

[0058] Furthermore, the step S200 includes: dividing each of the key frames into a plurality of image sub-blocks, and then inputting the image sub-blocks of each key frame into the spatial Transformer module corresponding to the key frame to obtain a frame vector corresponding to each key frame; and inputting all the frame vectors into the first temporal Transformer module to obtain the global feature information.

[0059] Among them, the Transformer network model needs to take sequence data as input, so in order to use Transformer to learn the spatial information of key frames, each key frame is divided into several image sub-blocks, and the spatial Transformer module is used to learn these image sub-blocks, capture the spatial features of the key frames, and generate the frame vector corresponding to the key frames.

[0060] For the T key frames input, a spatial Transformer module without shared parameters is used to learn the spatial features of the key frames and generate frame vectors. The number of key frames extracted from the video to be represented is less than or equal to the number of spatial Transformer modules. When the number of key frames is equal to the number of spatial Transformer modules, the key frames correspond to the spatial Transformer modules one by one. When the number of key frames is less than the number of spatial Transformer modules, image frames are automatically generated to complete the key frames input to the spatial Transformer module so that the number of key frames is equal to the number of spatial Transformer modules. The automatically generated image frames can be all-zero images.

[0061] After obtaining the frame vectors corresponding to T key frames, in order to learn the temporal features between the key frames, all the obtained frame vectors are input into the first temporal Transformer module to learn the temporal features of the video to be represented.

[0062] By independently modeling the spatial and temporal dimensions based on key frames and training them using Transformer, better processing speed and classification accuracy can be achieved.

[0063] Optionally, the local feature information is generated using a preset video extraction network, such as Figure 2 , the preset video extraction network includes a second temporal Transformer module and a primary feature extraction module based on an optical flow network. Among them, the primary feature extraction module can be selected as a primary feature extraction module based on an optical flow network, and the optical flow network can use FlowNet or SpyNet, which is a video feature extraction method based on optical flow. By inputting the video clip into the primary feature extraction module, the corresponding video clip vector can be obtained. Here, a primary feature extraction module with shared parameters can be used to simultaneously learn all the input video clips to obtain the corresponding video clip vector to save memory usage. Here, a primary feature extraction module without shared parameters can also be used to simultaneously learn all the input video clips.

[0064] The step S300 includes: inputting each of the video clips into the primary feature extraction module to obtain a video clip vector corresponding to each of the video clips; and inputting all of the video clip vectors into the second temporal Transformer module to obtain the local feature information.

[0065] It is difficult to capture fine-grained action information in key frames. In order to capture fine-grained action information, a preset video extraction network is used to learn the fine-grained actions contained in video clips. The learning of fine-grained action information of a single video clip is carried out separately. The primary feature extraction module learns the local fine-grained action information of a single video clip. By inputting all video clip vectors into the second sequential Transformer module to learn the global fine-grained action information of these video clips, we finally obtain the video feature representation containing fine-grained action information, that is, the local feature information here.

[0066] Furthermore, if Figure 2 , a feature map attention module is used to generate graph structure features.

[0067] Optionally, the graph structural features of the frame vector and the video clip vector are generated separately, that is, the graph structural features of the frame vector and the video clip vector are relatively independent. Specifically, the step S400 includes: treating each feature vector as a vertex and the similarity between each feature vector as an edge to construct a first fully connected graph; wherein the feature vector is one of the frame vector and the video clip vector; generating a fusion feature corresponding to each vertex in the first fully connected graph and all vertices and edges connected to it; using the graph attention mechanism, obtaining the weight of each vertex, multiplying the weight of each vertex with the corresponding fusion feature and accumulating them to obtain the graph structural feature of the feature vector.

[0068] Among them, according to each vertex in the first fully connected graph and all the vertices and edges connected to it, a fusion feature corresponding to each vertex is generated, specifically including: for each vertex, first concatenate the vertex with the vertices connected to it, multiply it by the corresponding edge, and then sum them up to obtain the fusion feature corresponding to the vertex. Take the frame vector as an example for explanation. For example, there are four frame vectors A, B, C, and D. A, B, C, and D are used as vertices, and the similarities between the two vectors of A, B, C, and D are used as edges to construct the first fully connected graph. For vertex A, A and A, A and B, A and C, and A and D are respectively concatenated for vectors, and then the concatenated vector of A and A is multiplied by the AA edge (i.e., the similarity between A and A), the concatenated vector of A and B is multiplied by the AB edge (i.e., the similarity between A and B), the concatenated vector of A and C is multiplied by the AC edge (i.e., the similarity between A and C), and the concatenated vector of A and D is multiplied by the AD edge (i.e., the similarity between A and D). The results are then added together to obtain the fused feature corresponding to vertex A. The above operations are then repeated for B, C, and D to obtain the fused features corresponding to B, C, and D.

[0069] The similarity in the above step may refer to the cosine similarity of two vectors.

[0070] The graph attention mechanism is used to obtain the weight of each vertex, specifically including: using a fully connected layer to obtain the weight of each vertex. For example, the vertex fusion feature matrix A (that is, the fusion feature corresponding to A) is known, and the dimension is (3,100). We first use the vertex matrix A as input and use the fully connected layer to generate the weight matrix W. The dimension of the weight matrix W is (3,1), which represents the weight of each vertex. Then, the weight matrix W and the vertex matrix A are matrix multiplied to obtain the graph structure feature vector E, and the dimension of E is 100.

[0071] The graph structure features obtained in this way can capture the feature vectors of complex video content relationships and improve the accuracy of video feature representation.

[0072] Optionally, after obtaining the key frame of the video to be represented, a video segment can be obtained from the video to be represented according to the key frame. For example, the key frame is used as the middle position, and video segments of a certain length are selected on the left and right sides as the extracted video segments. The key frame and the video segment have a one-to-one correspondence. In this case, step S400 includes:

[0073] Each of the frame vectors is regarded as a vertex, the similarity between each of the frame vectors is used as the first edge, and the similarity between each of the video clip vectors is used as the second edge to construct a second fully connected graph; according to each vertex in the second fully connected graph and all the vertices and edges connected to it, a fusion feature corresponding to each vertex is generated; using the graph attention mechanism, the weight of each vertex is obtained, the weight of each vertex is multiplied by the corresponding fusion feature and accumulated, and the common graph structure feature of the frame vector and the video clip vector is obtained.

[0074] The similarity in the above step may refer to the cosine similarity of two vectors.

[0075] Among them, because the key frame and the video clip have a one-to-one correspondence, taking the frame vector as the vertex is equivalent to taking the video clip vector as the vertex. For example, frame vector A corresponds to video clip vector a, frame vector B corresponds to video clip vector b, and taking frame vectors A and B as vertices is equivalent to taking video clip vectors a and b as vertices. Between every two vertices, the similarity between the frame vectors is used as the first edge, and the similarity between the video clip vectors is used as the second edge.

[0076] According to each vertex in the second fully connected graph and all the vertices and edges connected to it, a fusion feature corresponding to each vertex is generated. Specifically, for each vertex in the second fully connected graph, the fusion feature corresponding to each vertex is calculated according to the frame vector, and the fusion feature corresponding to each vertex is calculated according to the video clip vector, and then the two are spliced ​​as the fusion feature corresponding to each vertex in the second fully connected graph. Among them, the fusion feature corresponding to each vertex is calculated according to the frame vector / the fusion feature corresponding to each vertex is calculated according to the video clip vector. First, the vertex is spliced ​​with each of the vertices connected to it, multiplied by the corresponding edges, and then summed to obtain the fusion feature corresponding to the vertex. The relevant content has been described in detail above and will not be repeated here.

[0077] Optionally, after obtaining the global feature information, local feature information and graph structure features, they are input into a dynamic collaboration module, which dynamically assigns weights to the three features to obtain dynamically collaborative video features for use as the final video feature representation. Specifically, step S500 includes: concatenating the global feature information, the local feature information and the graph structure features, and then using the attention mechanism to obtain the dynamic weights of the global feature information, the local feature information and the graph structure features; multiplying the global feature information, the local feature information and the graph structure features with the dynamic weights respectively to obtain the video feature representation of the video to be represented.

[0078] In order to fully integrate the global feature information, local feature information and graph structure features, after the three features are spliced, a fully connected layer is used for nonlinear transformation, and then the attention mechanism is used to obtain the dynamic weights of the three features.

[0079] The dynamic weights of the global feature information, local feature information and graph structure features refer to the weights of the three features. After the weights of the three features are obtained, the three features are multiplied by their respective weights to obtain the video feature representation of the video to be represented.

[0080] Take global feature information, local feature information and graph structure features as input, use the fully connected layer to generate the weight values ​​of these three features, and multiply the three features and the corresponding weights to get the final video feature representation. For example: the three features are represented by a matrix M1 with dimensions (3, E), 3 represents the three features, and E represents the dimensions of the three features. The fully connected layer takes M1 as input and outputs a weight matrix W1 with dimensions (1, 3). Finally, the features and weight matrix are multiplied to get the final video representation: W1*M1=M2, and the dimensions of M2 are (1, E).

[0081] Different representations of video features are obtained through a Transformer network based on spatiotemporal separation, a preset video extraction network based on an optical flow network, and a feature map attention module. Different application scenarios have different inclinations for the three features of global feature information, local feature information, and graph structure features, that is, different application scenarios may have different weight distributions. For example, scenario one requires more global feature information, and scenario two requires more local feature information. Therefore, by setting up a dynamic collaboration module, the entire network can dynamically obtain the impact of various feature representations on the final video classification, dynamically assign weights to different feature representations, and obtain dynamically collaborative video features, which are used as the final video feature representation, making the final video classification effect better.

[0082] Therefore, by inputting key frames and video clips into a classification network based on the fusion of space-time separation Transformer and optical flow network, wherein the classification network includes a space-time separation Transformer network, an optical flow network-based preset video extraction network and a feature map attention module, key frames are input based on the space-time separation Transformer network, and the space-time separation Transformer network learns the spatial features and temporal features of the key frames; video clips are input into a preset video extraction network based on the optical flow network, and the preset video extraction network based on the optical flow network learns the fine-grained action information contained in the video; and then the feature map attention module uses the attention mechanism to construct the graph structure features between each frame vector and the video clip vector; finally, the dynamic coordination module dynamically assigns weights to the outputs of the space-time separation Transformer network, the optical flow network-based preset video extraction network and the feature map attention module to obtain dynamic coordinated video features as the final video representation features.

[0083] The present invention uses the Transformer network structure and key frame information to extract the global information of the video, which can give full play to the advantages of the Transformer and achieve acceptable classification accuracy under the premise of faster processing speed. In addition, in order to further improve the classification effect and ensure the operation effect, the local action information of the video is extracted by combining the optical flow method, and the Transformer network structure is used to learn this local information. Finally, the two are integrated through the dynamic collaborative module to achieve the purpose of obtaining both high classification accuracy and good operation efficiency.

[0084] When training the classification network based on the fusion of the spatiotemporal separation transformer and the optical flow network, the video data is first labeled and preprocessed before being divided into a training set and a test set. The training set data is then divided into multiple image sub-blocks and sent to the constructed deep neural network for training. After the training is completed, the test set data is sent to the network to evaluate the results of the network training.

[0085] The present invention also proposes a video classification method, comprising:

[0086] Acquire a video to be classified; characterize the video to be classified using the video characterization method described above to generate a video feature representation of the video to be classified; and classify the video to be classified based on the video feature representation of the video to be classified.

[0087] In actual video classification applications, the video characterization method described above is used to obtain the video feature representation of the video to be classified. Specifically, in one embodiment, the video to be classified is first preprocessed to obtain a key frame set and a video segment set of the video to be classified, the key frame set is input into a Transformer network based on time-space separation to generate global feature information, the video segment set is input into a preset video extraction network to generate local feature information, and then a graph structure feature between each frame vector and each video segment vector is constructed, and the global feature information, local feature information and graph structure feature are fused to obtain the video feature representation of the video to be characterized.

[0088] After obtaining the video feature representation of the video to be classified, the video feature representation can be input into the classification function for video classification. The specific implementation scheme of the classification function is not limited here, and it can be a Softmax function, a cross entropy function, etc.

[0089] By adopting the video characterization method as described above to obtain the video feature representation of the video to be classified, and performing video classification based on the representation, both classification accuracy and classification speed can be taken into account.

[0090] In one embodiment of the present invention, an electronic device includes a computer-readable storage medium storing a computer program and a processor, and when the computer program is read and executed by the processor, the video characterization method or video classification method described above is implemented. The beneficial effects of the electronic device of the present invention relative to the prior art are consistent with the above-mentioned video characterization method or video classification method, and are not described in detail here.

[0091] In one embodiment of the present invention, a computer readable storage medium stores a computer program, and when the computer program is read and executed by a processor, the video characterization method or video classification method described above is implemented. The beneficial effects of the computer readable storage medium of the present invention relative to the prior art are consistent with the above-mentioned video characterization method or video classification method, and are not described in detail here.

[0092] The reader should understand that in the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0093] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A video characterization method, characterized in that: include: Acquire key frames and video clips of the video to be represented, wherein the number of the key frames and the number of the video clips are both one or more; Generate a corresponding frame vector based on the key frame, and generate global feature information according to the frame vector; generating a corresponding video segment vector based on the video segment, and generating local feature information according to the video segment vector; Constructing graph structural features of the frame vector and the video clip vector; specifically comprising: considering each feature vector as a vertex and the similarity between each feature vector as an edge, and constructing a first fully connected graph; wherein the feature vector is one of the frame vector and the video clip vector; According to each vertex in the first fully connected graph and all the vertices and edges connected to it, generate a fusion feature corresponding to each vertex; Using the graph attention mechanism, the weight of each vertex is obtained, and the weight of each vertex is multiplied and accumulated with the corresponding fusion feature to obtain the graph structure feature of the feature vector; The global feature information, the local feature information and the graph structure feature are integrated to obtain a video feature representation of the video to be represented.

2. The video characterization method according to claim 1, characterized in that: The global feature information is generated by using a time-space separation based Transformer network, and the time-space separation based Transformer network includes a first temporal Transformer module and multiple spatial Transformer modules; The generating a corresponding frame vector based on the key frame, and generating global feature information according to the frame vector comprises: Divide each of the key frames into a plurality of image sub-blocks, and then input the image sub-blocks of each of the key frames into the spatial Transformer module corresponding to the key frame to obtain a frame vector corresponding to each of the key frames; All the frame vectors are input into the first temporal Transformer module to obtain the global feature information.

3. The video characterization method according to claim 1, characterized in that: The local feature information is generated by using a preset video extraction network, and the preset video extraction network includes a second temporal Transformer module and a primary feature extraction module based on an optical flow network; The generating a corresponding video segment vector based on the video segment, and generating local feature information according to the video segment vector comprises: Inputting each of the video clips into the primary feature extraction module to obtain a video clip vector corresponding to each of the video clips; All the video clip vectors are input into the second temporal Transformer module to obtain the local feature information.

4. The video characterization method according to claim 1, characterized in that: The step of obtaining key frames and video clips of the video to be represented includes: Obtaining key frames of the video to be represented; The video segment is obtained from the video to be represented according to the key frame.

5. The video characterization method according to claim 1, characterized in that: The key frame and the video segment have a one-to-one correspondence; the graph structure features for constructing the frame vector and the video segment vector include: Considering each of the frame vectors as a vertex, the similarity between each of the frame vectors as a first edge, and the similarity between each of the video clip vectors as a second edge, to construct a second fully connected graph; According to each vertex in the second fully connected graph and all the vertices and edges connected to it, generate a fusion feature corresponding to each vertex; The graph attention mechanism is used to obtain the weight of each vertex, and the weight of each vertex is multiplied and accumulated with the corresponding fusion feature to obtain the graph structure features of the frame vector and the video clip vector.

6. The video characterization method according to claim 1, characterized in that: The fusing of the global feature information, the local feature information and the graph structure feature to obtain the video feature representation of the video to be represented comprises: The global feature information, the local feature information and the graph structure feature are concatenated, and then the attention mechanism is used to obtain the dynamic weights of the global feature information, the local feature information and the graph structure feature; The global feature information, the local feature information, the graph structure feature, and the dynamic weight are respectively multiplied to obtain a video feature representation of the video to be represented.

7. A video classification method, characterized in that: include: Get the video to be classified; Characterizing the video to be classified using the video characterization method according to any one of claims 1 to 6 to generate a video feature representation of the video to be classified; The video to be classified is classified based on the video feature representation of the video to be classified.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is read and executed by a processor, the video characterization method according to any one of claims 1 to 6 or the video classification method according to claim 7 is implemented.

9. An electronic device, characterized in that: The method comprises a computer-readable storage medium storing a computer program and a processor, wherein when the computer program is read and executed by the processor, the video characterization method according to any one of claims 1 to 6 or the video classification method according to claim 7 is implemented.

Images