Video understanding method and device, computer readable medium and electronic equipment

Abstract

The embodiment of the invention provides a video understanding method and device, a computer readable medium and electronic equipment. The method comprises the following steps: encoding a video frame into a video frame feature, extracting an audio feature of an audio segment of a video, and encoding text information of the video into a text modal feature; constructing a video frame relation graph according to the video frame features, and constructing an audio segment relation graph according to the audio features; for each node in each target relation graph, determining an attention weight between the node and a neighbor node according to the feature of the node and the feature of the neighbor node, and determining an output node feature according to the attention weight and the feature of the neighbor node; generating visual modal features according to the output node features corresponding to the video frame relational graph, and generating audio modal features according to the output node features corresponding to the audio segment relational graph; and fusing each modal feature, and classifying fusion results to obtain a video understanding result. According to the embodiment of the invention, resources can be saved, and the real-time performance and accuracy of video classification can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a video understanding method, apparatus, computer-readable medium, and electronic device. Background Technology

[0002] Video classification is an extremely challenging task in the field of computer vision because it requires processing large amounts of data and is highly complex.

[0003] Currently, existing video classification schemes, especially when classifying long videos, not only consume a lot of resources but also have low real-time performance in obtaining video classification results, making it difficult to meet the speed requirements of video classification in specific scenarios. In addition, existing video classification schemes also suffer from low accuracy. Summary of the Invention

[0004] The embodiments of this application provide a video understanding method, apparatus, computer-readable medium, and electronic device, which can at least to some extent save the resources required for video classification and improve the real-time performance and accuracy of video classification.

[0005] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0006] According to one aspect of the embodiments of this application, a video understanding method is provided, the method comprising: encoding multiple video frames of a video into video frame features, extracting audio features of each audio segment of the video, and encoding text information corresponding to the video into text modal features; constructing a video frame relationship graph including multiple nodes based on the video frame features corresponding to each video frame, and constructing an audio segment relationship graph including multiple nodes based on the audio features of each audio segment; for each node in each target relationship graph, determining the relationship between the node and each of its neighboring nodes based on the features corresponding to the node and the features corresponding to each of the neighboring nodes. Attention weights are assigned between neighboring nodes, and output node features corresponding to each node are determined based on these attention weights and the features corresponding to each neighboring node. The target relationship graph is either the video frame relationship graph or the audio segment relationship graph, and the features are either video frame features or audio features. Visual modal features are generated based on the output node features corresponding to each node in the video frame relationship graph, and audio modal features are generated based on the output node features corresponding to each node in the audio segment relationship graph. The visual modal features, audio modal features, and text modal features are fused, and the fusion result is classified to obtain the video understanding result.

[0007] According to one aspect of the embodiments of this application, a video understanding apparatus is provided, the apparatus comprising: an encoding unit, configured to encode multiple video frames of a video into video frame features, extract audio features of each audio segment of the video, and encode text information corresponding to the video into text modal features; a relationship graph construction unit, configured to construct a video frame relationship graph including multiple nodes based on the video frame features corresponding to each video frame, and construct an audio segment relationship graph including multiple nodes based on the audio features of each audio segment; and a node feature encoding unit, configured to, for each node in each target relationship graph, determine the node feature based on the features corresponding to the node and the features corresponding to each of the node's neighboring nodes. The system calculates attention weights between a point and its neighboring nodes, and determines the output node features corresponding to the point based on these attention weights and the features corresponding to each neighboring node. The target relationship graph is either the video frame relationship graph or the audio segment relationship graph, and the features are either video frame features or audio features. A feature generation unit generates visual modal features based on the output node features corresponding to each node in the video frame relationship graph, and generates audio modal features based on the output node features corresponding to each node in the audio segment relationship graph. A fusion and classification unit fuses the visual modal features, the audio modal features, and the text modal features, and classifies the fusion result to obtain the video understanding result.

[0008] In some embodiments of this application, based on the foregoing scheme, the relationship graph construction unit is configured as follows:

[0009] Using the video frame features corresponding to each video frame as a node, edges are established between two nodes in each pair of nodes that satisfy the first predetermined condition, resulting in a complete frame graph including multiple nodes and edges between different nodes; and / or

[0010] According to the first predetermined rule, several key video frame features are selected from the video frame features corresponding to each video frame. Each key video frame feature is used as a node, and an edge is established between the two nodes in each pair of nodes to obtain a key frame graph that includes multiple nodes and edges between different nodes.

[0011] In some embodiments of this application, based on the foregoing scheme, the first predetermined condition is one of the following: the time interval between video frames corresponding to two nodes is less than a predetermined time interval threshold; the feature similarity between two nodes is less than a first predetermined similarity threshold; the time interval between video frames corresponding to two nodes is less than a predetermined time interval threshold, or the feature similarity between two nodes is less than a first predetermined similarity threshold; the time interval between video frames corresponding to two nodes is less than a predetermined time interval threshold, and the feature similarity between two nodes is less than a first predetermined similarity threshold.

[0012] In some embodiments of this application, based on the aforementioned scheme, the relationship graph construction unit includes a full frame graph construction subunit and a key frame graph construction subunit; the key frame graph construction subunit is configured to: select the first key video frame feature from the video frame features corresponding to each video frame as the current benchmark feature; repeatedly execute the key video frame feature filtering step until the last key video frame feature is filtered out; the key video frame feature filtering step includes: selecting the next key video frame feature from the video frame features corresponding to the video frames after the benchmark video frame corresponding to the current benchmark feature according to the temporal sequence of each video frame, wherein the video frame corresponding to the next key video frame feature is the first video frame in the video frames after the benchmark video frame where the feature similarity between the corresponding video frame feature and the current benchmark feature is less than a second predetermined similarity threshold; and using the next key video frame feature as the new current benchmark feature.

[0013] In some embodiments of this application, based on the foregoing scheme, the video frame relationship graph includes the all-frame graph and the keyframe graph, and the feature generation unit is configured to: generate the overall feature of the all-frame graph based on the output node features corresponding to each node of the all-frame graph; generate the overall feature of the keyframe graph based on the output node features corresponding to each node of the keyframe graph; and perform a splicing operation on the overall feature of the all-frame graph and the overall feature of the keyframe graph to obtain visual modal features.

[0014] In some embodiments of this application, based on the foregoing scheme, the relationship graph construction unit is configured as follows:

[0015] Using the audio feature corresponding to each audio segment as a node, edges are established between two nodes in each pair of nodes that satisfy the second predetermined condition, resulting in a complete audio segment graph including multiple nodes and edges between different nodes; and / or

[0016] According to the second predetermined rule, several key audio features are selected from the audio features corresponding to each audio segment. Each key audio feature is used as a node, and an edge is established between the two nodes in each pair of nodes to obtain a key audio segment graph that includes multiple nodes and edges between different nodes.

[0017] In some embodiments of this application, based on the foregoing scheme, the fusion and classification unit is configured to: perform information mapping on the visual modal features, the audio modal features, and the text modal features respectively to obtain visual modal mapping features, audio modal mapping features, and text modal mapping features; and fuse the visual modal mapping features, the audio modal mapping features, and the text modal mapping features to obtain a fusion result.

[0018] In some embodiments of this application, based on the foregoing scheme, the fusion and classification unit is configured to: concatenate the visual modality mapping features, the audio modality mapping features, and the text modality mapping features to obtain a concatenation result; perform a first linear mapping on the concatenation result to obtain a first mapping result; perform a nonlinear mapping on the first mapping result to obtain a nonlinear mapping result; and perform a second linear mapping on the nonlinear mapping result to obtain a fusion result.

[0019] In some embodiments of this application, based on the foregoing scheme, the node feature encoding unit is configured to: normalize each attention weight to obtain normalized attention weights; for each neighbor node, determine the feature corresponding to the neighbor node, the normalized attention weights between the node and the neighbor node, and the product of the parameter matrix; determine the sum of the products determined for each neighbor node; and perform a nonlinear mapping on the sum of the products to obtain the output node features.

[0020] According to one aspect of the embodiments of this application, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the video understanding method as described in the above embodiments.

[0021] According to one aspect of the embodiments of this application, an electronic device is provided, including: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the video understanding method as described in the above embodiments.

[0022] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium, a processor of a computer device reading the computer instructions from the computer-readable storage medium, and the processor executing the computer instructions, causing the computer device to perform the video understanding method as described in the above embodiments.

[0023] In some embodiments of this application, after obtaining multiple video frames, multiple audio segments, and corresponding text information of a video, the multiple video frames are first encoded into video frame features, the audio features of each audio segment are extracted, and the corresponding text information is encoded into text modal features. Next, a video frame relationship graph with multiple nodes is constructed based on the video frame features corresponding to each video frame, and an audio segment relationship graph with multiple nodes is constructed based on the audio features of each audio segment. Then, for each node in each video frame relationship graph or audio segment relationship graph, the video frame features or audio features corresponding to that node and the corresponding text modal features are used to extract the audio features of each audio segment. The video frame features or audio features corresponding to neighboring nodes are used to determine the attention weights between the node and each neighboring node. Based on these attention weights and the video frame features or audio features corresponding to each neighboring node, the output node features corresponding to the node are determined, thus obtaining the output node features corresponding to each node in each relationship graph. Next, visual modal features are generated based on the output node features corresponding to each node in the video frame relationship graph, and audio modal features are generated based on the output node features corresponding to each node in the audio segment relationship graph. Finally, the visual modal features, audio modal features, and text modal features are fused, and the fusion results are classified to obtain the video understanding results.

[0024] Because the embodiments of this application use information from three modalities—image, audio, and text—to classify and understand videos, they can capture the complex structure of videos and extract video features that can more accurately describe the videos, thus improving the accuracy of video classification. Furthermore, since the video frame features corresponding to video frames and the audio features of audio segments are represented as relatively sparse graph structures, less memory and computational resources are required during model inference. This effectively saves various resources, improves inference speed, and ensures the real-time nature of video classification results, meeting the needs of various scenarios requiring efficient and real-time video classification.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0027] Figure 1A schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of this application can be applied is shown;

[0028] Figure 2 A flowchart of a video understanding method according to an embodiment of this application is shown;

[0029] Figure 3 A model architecture diagram according to an embodiment of this application is shown;

[0030] Figure 4 A flowchart is shown illustrating an embodiment of the present application of selecting several key video frame features from video frame features corresponding to each video frame according to a first predetermined rule.

[0031] Figure 5 A flowchart is shown illustrating an embodiment of this application, which determines the output node features of a node based on each attention weight and the features corresponding to each neighbor node.

[0032] Figure 6 A flowchart illustrating the generation of visual modal features based on the output node features corresponding to each node in a video frame relationship graph according to an embodiment of this application is shown.

[0033] Figure 7 An embodiment according to this application is shown. Figure 2 A flowchart detailing step 250 in the embodiment;

[0034] Figure 8 A flowchart illustrating the fusion of visual modality mapping features, audio modality mapping features, and text modality mapping features according to an embodiment of this application to obtain a fusion result;

[0035] Figure 9 A block diagram of a video understanding apparatus according to an embodiment of this application is shown;

[0036] Figure 10 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0037] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0038] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0039] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0040] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0041] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0042] The video classification schemes used in related technologies mainly utilize models such as convolutional neural networks, recurrent neural networks, and Transformer models to process the visual information in videos, which is a single modality. The main drawbacks of this approach include:

[0043] 1. This approach suffers from insufficient information, ignoring important information contained in other modalities of the video and lacking sufficient representation ability for the video. This not only results in poor performance of video classification but also leads to poor algorithm robustness, such as the inability to handle situations where visual and audio information do not match.

[0044] 2. This method typically processes all video frames. However, adjacent frames often have high similarity, leading to information redundancy and wasted computational resources.

[0045] Video classification is a highly challenging task in computer vision, its complexity stemming primarily from several factors. First, the sheer volume of video data demands algorithms capable of efficiently processing massive amounts of data. In real-time applications, such as video surveillance, speed is critical for video classification. Second, videos contain temporal and spatial information, requiring algorithms to capture and understand the dynamic changes between frames, increasing the analytical difficulty. Furthermore, the need to fuse visual, auditory, and textual information within videos, along with the requirement for fine-grained classification, further complicates the task. Therefore, current video classification schemes are insufficient to address these challenges.

[0046] To address this, this application first provides a video understanding method. The video understanding method provided by the embodiments of this application overcomes the aforementioned deficiencies in related technologies. This method proposes, for the first time, a graph-based representation learning framework. Specifically, firstly, single-segment features are extracted using CLIP, VGGish, and BERT models respectively; then, the video is represented as a graph structure based on the inter-frame context relationship, using both all frames and keyframes as methods. Through attention-based layer-by-layer information transfer, multi-frame information is jointly modeled to obtain single-modal features; finally, multiple modalities (visual, speech, and text) are jointly modeled to obtain the recognition result. Therefore, the solution of this application proposes a video multimodal feature fusion model based on graph networks. This model jointly models the visual, audio, and text modalities of the video, comprehensively utilizing information from these three modalities to effectively fuse the multimodal information of the video. It can capture the complex structure of the video and extract accurately descriptive video features. Simultaneously, this application models the video as a graph structure, selecting effective keyframes and effectively fusing multi-frame and multimodal information. The relatively sparse graph structure allows for remote context aggregation within a relatively small memory and computational footprint, ensuring the model does not encounter training bottlenecks. This is particularly effective for long videos. Experimental results show that the solution of this application demonstrates strong performance in video recognition tasks, achieving optimal results in both recognition accuracy and efficiency. Therefore, the solution based on this application can improve the efficiency and accuracy of video recognition.

[0047] In summary, the solution proposed in this application is essentially a video recognition model that combines visual, audio, and textual modal information for comprehensive information extraction. This allows for rapid identification of harmful content and the provision of review results. The solution presented in this application can be applied to short video platforms, long video platforms, live streaming platforms, media review, etc. Specifically, it can be applied in the following real-world scenarios:

[0048] 1. Video Review Scenarios. The solution in this application can understand multiple modalities in a video using a single model, while exhibiting good generalization and robustness, and lower development and maintenance costs. Furthermore, by utilizing sparse connections in a graph, unnecessary computation is reduced. Whether the video is long or short, the model can employ effective key information fusion strategies to ensure review efficiency. Therefore, compared to previous models, the solution in this application achieves higher review performance while maintaining effectiveness.

[0049] 2. Live Streaming Scenarios. The model structure of the solution in this application is lightweight, requires less computation, and has the advantages of real-time and rapid response. It can identify the audio, text, and video content of live video streams in real time, respond quickly, and control business risks.

[0050] 3. Video Summary Scenarios. The keyframe extraction algorithm proposed in this application can quickly convey the core content of a video in practical applications, helping users to quickly grasp the key points of the video.

[0051] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of this application can be applied is shown. For example... Figure 1 As shown, the system architecture 100 may include a user terminal 110 and a cloud 120. The cloud 120 specifically includes a video server 121 and an inference server cluster 122. The inference server cluster 122 specifically includes a first inference server 1221, a second inference server 1222, a third inference server 1223, and a fourth inference server 1224. The user terminal 110 establishes a communication connection with the cloud 120. The first inference server 1221, the second inference server 1222, and the third inference server 1223 in the inference server cluster 122 can communicate with the video server 121, and the fourth inference server 1224 can also communicate with the first inference server 1221, the second inference server 1222, and the third inference server 1223, respectively. Video server 121 stores complete video data, including video frames, audio data, and corresponding text information. A CLIP model is deployed on the first inference server 1221, a VGGish model on the second inference server 1222, a BERT model on the third inference server 1223, and the main body of the video classification model is deployed on the fourth inference server 1224. This main body, together with the CLIP, VGGish, and BERT models, constitutes the video classification model of the embodiment of this application. When the video understanding method provided in this embodiment is applied... Figure 1In the system architecture shown, a process can be as follows: First, the first inference server 1221 obtains multiple video frames from the video server 121, encodes each video frame into video frame features using the CLIP model, and then sends each video frame feature to the fourth inference server 1224; the second inference server 1222 obtains the audio data of the video from the video server 121, divides the audio data into multiple audio segments, extracts the audio features of each audio segment using the VGGish model, and then sends the audio features of each audio segment to the fourth inference server 1224; the third inference server... Server 1223 obtains the text information corresponding to the video from video server 121, encodes the text information into text modal features using the BERT model, and then sends the text modal features to the fourth inference server 1224. Then, the fourth inference server 1224 processes the obtained video frame features, audio features of each audio segment, and text modal features through the main part of the video classification model, and finally outputs the video understanding result. Finally, the fourth inference server 1224 pushes the video understanding result to user terminal 110, where the reviewer using user terminal 110 reviews the video understanding result.

[0052] In some embodiments of this application, the fourth inference server 1224 pushes the video understanding results to the user terminal 110 as an operation performed based on a request from the user terminal 110.

[0053] In some embodiments of this application, the fourth inference server 1224 pushes the video understanding result to the user terminal 110 when it determines that the video understanding result belongs to a harmful type.

[0054] In some embodiments of this application, the fourth inference server 1224 pushes an interface containing the video understanding result and the access entry corresponding to the video understanding result to the user terminal 110. The reviewer using the user terminal 110 can watch the video corresponding to the video understanding result by triggering the access entry, and then review the video understanding result.

[0055] In some embodiments of this application, video server 121 is a cache server of the video platform, and video server 121 is used to temporarily store the complete data of videos uploaded by users to the video platform.

[0056] In some embodiments of this application, the video platform further includes a main server and a storage server. The fourth inference server 1224 also sends the video understanding result to the main server of the video platform. After obtaining the video understanding result, the main server will determine whether the video understanding result belongs to a harmful type. If it is determined that the video understanding result does not belong to a harmful type, the main server will migrate the complete data of the video corresponding to the video understanding result from the video server 121 to the main server. Other users can access the complete data of the video stored on the main server through the client of the video platform.

[0057] In some embodiments of this application, if the master server determines that the video understanding result is of a harmful type, it will send the corresponding video deletion instruction to the video server 121. The video deletion instruction indicates that the complete data of the video should be deleted or that the complete data of the video should be deleted after a predetermined time period. The video server 121 will immediately delete the complete data of the video or delete the complete data of the video after the predetermined time period.

[0058] In some embodiments of this application, after the master server sends the corresponding video deletion instruction to the video server 121, it will send a notification message including the video understanding result and video deletion information to the user who uploaded the complete data of the corresponding video. The video deletion information is used to indicate that the complete data of the video has been deleted or that the complete data of the video will be deleted after a predetermined time period.

[0059] In some embodiments of this application, the process of reviewing the video understanding results by the reviewer is triggered by a request from a user who uploaded complete data of the corresponding video.

[0060] It should be understood that Figure 1 The number of user terminals and video and inference servers in the cloud is merely illustrative. Depending on implementation needs, there can be any number of user terminals, and the number of video and inference servers in the cloud can also be arbitrary. That is, there can be multiple user terminals, the video servers in the cloud can be a server cluster consisting of multiple servers, and the cloud can include more or fewer inference servers.

[0061] It should be noted that, Figure 1 The illustration shown is merely one embodiment of this application. Although in Figure 1 In the embodiments of this application, the user terminal is a desktop computer. However, in other embodiments of this application, the user terminal can also be various types of terminal devices such as smartphones, laptops, tablets, vehicle terminals, portable wearable devices, and workstations. Figure 1In the embodiments of this application, the video understanding method is mainly used in video review scenarios. However, in other embodiments of this application, the video understanding method can also be applied to scenarios such as live streaming and video summaries. Figure 1 In the embodiments of this application, the video server and each inference server are located in the cloud; however, in other embodiments of this application, the video server or at least one inference server may be located outside the cloud. Figure 1 In the embodiment, video frames, audio data, and corresponding text information are all stored on the same server. However, in other embodiments of this application, they may be stored on different servers. Figure 1 In the embodiment, video frames are encoded using the CLIP model, audio features of each audio segment are extracted using the VGGish model, and text information is encoded using the BERT model. However, in other embodiments of this application, each model can be replaced with another model that has the same function; although in Figure 1 In the embodiment, the video server 121 stores the audio data of the video. The division of the audio data into multiple audio segments is an operation performed by the inference server. However, in other embodiments of this application, the video server 121 may also directly store the multiple audio segments into which the audio data is divided, and the inference server may directly obtain the audio segments from the video server 121. This application does not limit the scope of protection in any way, nor should it restrict the scope of protection of this application.

[0062] As is readily understood, the video understanding method provided in this application embodiment is generally executed by a server, and correspondingly, the video understanding device is generally located within the server. However, in other embodiments of this application, the terminal device may also have similar functions to the server, thereby executing the video understanding scheme provided in this application embodiment.

[0063] Therefore, the embodiments of this application can be applied to terminals or servers. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, and this application does not impose any restrictions.

[0064] The implementation details of the technical solutions in the embodiments of this application are described in detail below:

[0065] Figure 2 A flowchart of a video understanding method according to an embodiment of this application is shown. This video understanding method can be executed by various devices with processing and computing capabilities. Specifically, it can be executed by a target device, such as a user terminal or a cloud server. User terminals include, but are not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, smartwatches, etc. Please refer to... Figure 2 As shown, this video understanding method includes at least the following steps:

[0066] In step 210, multiple video frames of the video are encoded into video frame features, audio features of each audio segment of the video are extracted, and the text information corresponding to the video is encoded into text modal features.

[0067] Before encoding multiple video frames into video frame features, it is necessary to first obtain the multiple video frames of the video.

[0068] Multiple video frames can be all the video frames or only a portion of the video frames.

[0069] In one embodiment of this application, before encoding multiple video frames of a video into video frame features respectively, the method includes: sampling all video frames of the video to obtain multiple video frames.

[0070] Specifically, a video file is a video sequence consisting of multiple video frames. This video sequence can include 30 frames per second, and can be sampled by sampling one frame per second. Of course, other sampling methods can also be used, such as sampling two or more frames per second, or sampling one frame every two seconds.

[0071] In this embodiment of the application, multiple video frames of a video are obtained by sampling all video frames of the video, which can reduce the amount of data required for video classification and thus save computing resources.

[0072] The method of sampling one frame per second in the above embodiments is actually a uniform sampling method. Non-uniform sampling can also be used to sample video frames. Specifically, for videos with corresponding bullet comments, the sampling frequency can be determined according to the number of bullet comments. The number of bullet comments is positively correlated with the sampling frequency. For example, if there are 10 bullet comments in a certain second, then one frame can be sampled from the video frame corresponding to that second. If there are 20 bullet comments in another second, then two frames can be sampled from the video frame corresponding to that second. By sampling video frames in this non-uniform sampling method, video frames containing more information can be sampled, thereby improving the performance of video understanding.

[0073] Figure 3A model architecture diagram according to an embodiment of this application is shown. Figure 3 The provided model is actually a video classification model from the embodiments of this application, which can be used to execute the video understanding method provided in the embodiments of this application. Please refer to [link to relevant documentation]. Figure 3 As shown, video images are input into the Contrastive Language-Image Pre-training (CLIP) model of the video classification model.

[0074] CLIP is a multimodal model whose primary purpose is to learn the relationships between images and text. The CLIP model consists of two main components: an image encoder and a text encoder. The image encoder is responsible for extracting features from the input image. It can be based on existing Convolutional Neural Networks (CNNs) architectures, such as the ResNet series, or more advanced architectures, such as the Vision Transformer (ViT). The output of the image encoder is a fixed-length vector that captures the main information of the image. The text encoder processes the text input, typically using a Transformer architecture, a very successful model type in natural language processing tasks. The text encoder also outputs a fixed-length vector representing the semantic information of the input text. During training, the CLIP model aims to maximize the similarity between the image and its corresponding text description while minimizing the similarity with other mismatched image-text pairs. This process is usually achieved through contrastive learning, where the model learns how to distinguish between positive samples (correct image-text pairs) and negative samples (incorrect image-text pairs).

[0075] Therefore, each video frame can be encoded into video frame features using an image encoder based on the CLIP model. Of course, in other embodiments of this application, other models with image encoding capabilities can also be used to encode video frames.

[0076] If a video frame is represented as v, then the video frame features obtained using CLIP encoding are represented as follows:

[0077] f(v) = CLIP(v),

[0078] in, This represents the video frame features obtained using CLIP encoding, which is a 1024-dimensional vector.

[0079] Before extracting audio features from each audio segment of a video, it is necessary to first obtain multiple audio segments from the video. Audio segments are obtained from the video's audio data. Multiple audio segments of a video can be all audio segments of the video, meaning all audio segments can form complete audio data; alternatively, multiple audio segments of a video can be only a subset of all audio segments of the video.

[0080] Audio data can be divided into multiple audio segments. For example, multiple audio segments can be obtained by extracting a segment of audio data every 0.96 seconds. Of course, the time interval for extracting audio segments can also be set to 1 second, that is, multiple audio segments can be obtained by extracting a segment of audio data every 1 second.

[0081] The VGGish model can be used to extract audio features from various audio segments. VGGish is a deep learning model for audio signal processing, primarily used for audio feature extraction, and widely applied in the field of audio analysis.

[0082] If an audio segment is represented as s, then the extracted audio features are represented as follows:

[0083] f(s) = VGGish(s),

[0084] in, This represents the audio features encoded by the VGGish model, typically multiple (e.g., n) 128-dimensional vectors.

[0085] Please continue reading Figure 3 As shown, the video audio is input into the VGGish model.

[0086] Before encoding the text information corresponding to the video into text modal features, it is necessary to first obtain the text information corresponding to the video.

[0087] The text information corresponding to the video may include at least one of the following: video title, video subtitle, and text obtained by optical character recognition (OCR) of each video frame.

[0088] When the text information corresponding to the video includes multiple types of text as mentioned above, different types of text can be concatenated and then encoded.

[0089] Specifically, to obtain text modal features, the text information corresponding to the video can be encoded using the Bidirectional Encoder Representations from Transformers (BERT) model. By pre-training on a large amount of text, BERT learns deep language representations that can capture rich semantic information of words and sentences, and performs excellently in multiple fields such as text classification and machine translation.

[0090] If we represent the text information corresponding to the video as t, then the text modal features encoded from the text information corresponding to the video are:

[0091] F(t) = BERT(t),

[0092] Where F(t) represents the text modal feature.

[0093] Please continue reading Figure 3 As shown, the video title will be input into the BERT model.

[0094] In step 220, a video frame relationship graph with multiple nodes is constructed based on the video frame features corresponding to each video frame, and an audio segment relationship graph with multiple nodes is constructed based on the audio features of each audio segment.

[0095] The inventors of this application have discovered that there is a natural temporal relationship between continuously extracted frames and continuous audio segments, and that graphs are the data structure that best expresses contextual relationships.

[0096] In one embodiment of this application, a video frame relationship graph including multiple nodes is constructed based on the video frame features corresponding to each video frame, including:

[0097] Using the video frame features corresponding to each video frame as a node, edges are established between two nodes in each pair of nodes that satisfy the first predetermined condition, resulting in a complete frame graph including multiple nodes and edges between different nodes; and / or

[0098] According to the first predetermined rule, several key video frame features are selected from the video frame features corresponding to each video frame. Each key video frame feature is used as a node, and an edge is established between the two nodes in each pair of nodes to obtain a key frame graph that includes multiple nodes and edges between different nodes.

[0099] Therefore, a video frame relationship graph can include all frame graphs and / or keyframe graphs. Of course, a video frame relationship graph can include both all frame graphs and keyframe graphs. The advantage of doing so is that it can more fully represent the video, thereby improving video understanding performance.

[0100] All frame graphs include nodes corresponding to all the extracted video frames. Therefore, it is called an all-frame graph. When constructing an all-frame graph, for any pair of nodes, if the pair of nodes meets the first predetermined condition, an edge is established between the pair of nodes.

[0101] The all-frame graph graph_v_all can be represented as G v_all =(V v_all , E v_all ), where V v_all represents the nodes corresponding to all video frames, and E v_all represents the set of edges corresponding to these nodes.

[0102] In an embodiment of the present application, the first predetermined condition is one of the following: the time interval between the video frames corresponding to two nodes is less than a predetermined time interval threshold; the feature similarity between two nodes is less than a first predetermined similarity threshold; the time interval between the video frames corresponding to two nodes is less than a predetermined time interval threshold, or the feature similarity between two nodes is less than a first predetermined similarity threshold; the time interval between the video frames corresponding to two nodes is less than a predetermined time interval threshold, and the feature similarity between two nodes is less than a first predetermined similarity threshold.

[0103] It is easy to understand that each video frame in the video has a corresponding time. The predetermined time interval threshold can be t, the first predetermined similarity threshold can be s, and the feature similarity can be calculated by cosine similarity. If an edge (i, j) is to be established between node i and node j, it can be determined that the time interval between video frame i and video frame j is less than t, or it can be determined that the feature similarity between video frame i and video frame j is less than s, that is, the cosine similarity cos(f(v i ), f(v j )) < s, where v i represents video frame i, v j represents video frame j, f(v i ) represents node i, and f(v j ) represents node j.

[0104] By establishing edges between the two nodes in each pair of nodes that meet the first predetermined condition, this ensures the connection of relevant nodes required for information transmission, and at the same time can reduce memory consumption and computational overhead.

[0105] Each key video frame feature in the key frame graph is a node.

[0106] The key frame graph only includes nodes corresponding to key video frames and key video frame features. Therefore, it is called a key frame graph. When constructing a key frame graph, for any pair of nodes, an edge is established between the pair of nodes.

[0107] The keyframe graph graph_v_key can be represented as G v_key =(V v_key E v_key ), where V v_key E represents the node corresponding to the key video frame features. v_key This represents the set of edges corresponding to these nodes.

[0108] To obtain key video frame features, one can first select key video frames from each video frame and use the video frame features corresponding to the key video frames as key video frame features; alternatively, one can directly select key video frame features from the video frame features corresponding to each video frame.

[0109] Figure 4 A flowchart illustrating the selection of several key video frame features from video frame features corresponding to each video frame according to a first predetermined rule, based on an embodiment of this application, is provided. Please refer to [link to flowchart documentation]. Figure 4 As shown, selecting several key video frame features from the video frame features corresponding to each video frame according to a first predetermined rule may specifically include the following steps:

[0110] In step 410, the first key video frame feature is selected from the video frame features corresponding to each video frame and used as the current baseline feature.

[0111] The first key video frame feature can be the video frame feature corresponding to the first video frame, or it can be the video frame feature corresponding to a video frame that is earlier in the time sequence.

[0112] Next, the key video frame feature filtering step needs to be repeated until the last key video frame feature is selected. The key video frame feature filtering step may include steps 420-440, as follows:

[0113] In step 420, according to the temporal sequence of each video frame, the next key video frame feature is selected from the video frame features corresponding to the video frames following the current reference feature. The video frame corresponding to the next key video frame feature is the first video frame in the video frames following the reference video frame where the feature similarity between the corresponding video frame feature and the current reference feature is less than a second predetermined similarity threshold.

[0114] The second predetermined similarity threshold c can be the same as or different from the first predetermined similarity threshold.

[0115] Specifically, starting from the benchmark video frame corresponding to the current benchmark feature, each video frame after the benchmark video frame is traversed according to the temporal order of each video frame. The feature similarity between the video frame feature corresponding to each video frame and the current benchmark feature is determined. Video frames whose feature similarity between the corresponding video frame feature and the current benchmark feature reaches a second predetermined similarity threshold are removed until the next video frame with a certain difference (the feature similarity between the corresponding video frame feature and the current benchmark feature is less than the second predetermined similarity threshold) is found.

[0116] In step 430, the features of the next key video frame are used as the new current baseline features.

[0117] In step 440, is the feature of the last key video frame selected?

[0118] Determine whether the last key video frame feature has been selected. If the last key video frame feature has been selected, the key video frame feature selection step ends. Otherwise, continue the key video frame feature selection step, that is, execute steps 420-440 again based on the new current benchmark feature.

[0119] In other words, repeat the above steps until all video frames have been traversed and processed.

[0120] Although the embodiments of this application select key video frame features based on feature similarity between features, other embodiments of this application may also employ other methods to select key video frame features. For example, key video frames may be selected based on the number of bullet comments corresponding to each video frame. Specifically, video frames with a bullet comment count greater than a predetermined bullet comment count threshold may be designated as key video frames. After obtaining the key video frames, the video frame features corresponding to the key video frames may be used as the key video frame features.

[0121] Similarly, an audio segment relationship diagram can be constructed in a similar manner.

[0122] In one embodiment of this application, constructing an audio segment relationship graph including multiple nodes based on the audio features of each audio segment may include the following steps:

[0123] Using the audio feature corresponding to each audio segment as a node, edges are established between two nodes in each pair of nodes that satisfy the second predetermined condition, resulting in a complete audio segment graph including multiple nodes and edges between different nodes; and / or

[0124] According to the second predetermined rule, several key audio features are selected from the audio features corresponding to each audio segment. Each key audio feature is used as a node, and an edge is established between the two nodes in each pair of nodes to obtain a key audio segment graph that includes multiple nodes and edges between different nodes.

[0125] Similarly, an audio segment relationship graph can include a full audio segment graph and / or a key audio segment graph. Of course, an audio segment relationship graph can include both full and key audio segment graphs simultaneously. The advantage of doing so is that it can more fully represent the audio of the video, thereby improving video understanding performance.

[0126] The second predetermined condition may be the same as or different from the first predetermined condition. The second predetermined condition may also include the time interval between the audio segments corresponding to the two nodes being less than a preset time interval threshold, and / or the feature similarity between the two nodes being less than a third predetermined similarity threshold.

[0127] The preset time interval threshold can be the same as or different from the aforementioned predetermined time interval threshold. Similarly, the third predetermined similarity threshold can be the same as or different from the aforementioned first predetermined similarity threshold.

[0128] Therefore, through the above process, we can obtain the complete audio segment graph graph_a_all and the key audio segment graph graph_a_key.

[0129] Please refer to Figure 3. Based on the output of the CLIP model, the complete frame graph and key frame graph can be constructed. Based on the output of the VGGish model, the corresponding complete audio segment graph graph_a_all and key audio segment graph graph_a_key can also be constructed.

[0130] In step 230, for each node in each target relationship graph, the attention weight between the node and each neighbor node is determined according to the features corresponding to the node and the features corresponding to each neighbor node. The output node features corresponding to the node are determined according to the attention weights and the features corresponding to each neighbor node. The target relationship graph is a video frame relationship graph or an audio segment relationship graph, and the features are video frame features or audio features.

[0131] After the graph is constructed, the node features of the graph attention network are updated layer by layer. The graph attention network is a network composed of stacked graph attention layers. Through the graph attention mechanism, each frame continuously aggregates information from neighboring frames. At the same time, during the information transmission process, self-attention can adaptively adjust the weights of different neighboring nodes.

[0132] A node's neighbor nodes are nodes in the graph that have edges with that node.

[0133] For each node in the video frame relationship graph, the attention weights between the node and its neighboring nodes are determined based on the video frame features corresponding to the node and the video frame features corresponding to each neighboring node. The output node features corresponding to the node are then determined based on the attention weights and the video frame features corresponding to each neighboring node.

[0134] For each node in the audio segment relationship graph, the attention weights between the node and its neighbors are determined based on the audio features corresponding to the node and the audio features corresponding to each neighbor node. The output node features of the node are then determined based on the attention weights and the audio features corresponding to each neighbor node.

[0135] The attention weight between two nodes is the same as the attention weight on the edge between the corresponding nodes.

[0136] Specifically, it is necessary to first perform a linear transformation on the features of the input nodes, and then calculate the pairwise attention weights based on the linearly transformed features between the nodes. That is, the attention weights between node i and its neighbor node j can be calculated using the following formula:

[0137] s ij =a(W h h i W h h j ),

[0138] Among them, the two Ws h Let h represent the first parameter matrix corresponding to node i and the second parameter matrix corresponding to node j, respectively. The first parameter matrix and the second parameter matrix can be different parameter matrices, i.e., model parameters. i h represents the feature corresponding to node i. j Let be the feature corresponding to node j, a be the attention function, and s be the feature. ij Let be the attention weight between node i and node j, representing the importance of node j's features to node i.

[0139] This application uses an additive attention mechanism, which is defined as follows:

[0140] a(W h h i W h h j )=σ(v T (W h h i +W h h j )),

[0141] Where v is the third parameter matrix, i.e., the model parameters, and σ is the LeakyReLU activation function.

[0142] Figure 5 A flowchart illustrating the determination of output node features based on attention weights and features corresponding to neighboring nodes, according to an embodiment of this application, is provided. Please refer to [link to flowchart documentation]. Figure 5 As shown, the output node features corresponding to a node are determined based on the attention weights and the features corresponding to each neighbor node. Specifically, this may include the following steps:

[0143] In step 510, each attention weight is normalized to obtain the normalized attention weight.

[0144] To maintain the scale of features at each node, attention weights need to be normalized based on neighboring nodes.

[0145] Specifically, the attention weights between node i and its neighbor node j can be normalized using the following formula:

[0146]

[0147] Where exp(·) represents the exponential function, s ij Let s be the attention weight between node i and its neighbor node j. ik Let k be the attention weight between node i and its neighbor node k. Let i be the set of neighboring nodes of node i.

[0148] In step 520, for each neighboring node, the features corresponding to the neighboring node, the normalized attention weights between the node and its neighbors, and the product of the parameter matrices are determined.

[0149] After obtaining the normalized attention weights, the output node features can be obtained by weighted linear summation of the neighbor node features based on these weights.

[0150] In step 530, the sum of the products determined for each neighboring node is determined.

[0151] In step 540, the sum of the products is nonlinearly mapped to obtain the output node features.

[0152] Specifically, the output node features can be obtained based on the normalized attention weights using the following formula:

[0153]

[0154] Among them, a ij W represents the normalized attention weight between node i and its neighbor node j. h This represents the parameter matrix in step 520, which can be the fourth parameter matrix and may differ from the aforementioned parameter matrices.j The feature corresponding to neighbor node j, Let h be the set of neighboring nodes of node i, σ be the Leaky ReLU activation function, which can be used for nonlinear mapping, and h be the set of neighboring nodes of node i. ‘ i Let i be the output node feature corresponding to node i.

[0155] In step 240, visual modal features are generated based on the output node features corresponding to each node in the video frame relationship graph, and audio modal features are generated based on the output node features corresponding to each node in the audio segment relationship graph.

[0156] When the video frame relationship graph includes both the full frame graph and the key frame graph, it is necessary to generate visual modal features based on the output node features corresponding to each node in both the full frame graph and the key frame graph.

[0157] Figure 6 A flowchart illustrating the generation of visual modal features based on the output node features corresponding to each node in a video frame relationship graph, according to one embodiment of this application, is provided. Please refer to [link to flowchart documentation]. Figure 6 As shown, when the video frame relationship graph includes all frame graphs and keyframe graphs, visual modal features are generated based on the output node features corresponding to each node in the video frame relationship graph. Specifically, this may include the following steps:

[0158] In step 610, the overall features of all frame images are generated based on the output node features corresponding to each node of all frame images.

[0159] Specifically, in this embodiment of the application, after obtaining the output node features, they will be processed by other parts of the graph attention network to finally obtain the node features output by the output layer of the graph attention network.

[0160] The overall features of all frames can be obtained in the following way:

[0161]

[0162] Among them, o i For the node features of the output layer corresponding to node i in all frames, avgpool represents the average pooling operation, F all (v) represents the overall features of all frames.

[0163] In step 620, the overall features of the keyframe map are generated based on the output node features corresponding to each node of the keyframe map.

[0164] Similarly, the overall features of the keyframe can be obtained in the following way:

[0165]

[0166] Among them, o i For node i in the keyframe graph, avgpool represents the output layer node features, and F represents the average pooling operation. key (v) represents the overall features of the keyframe graph.

[0167] In step 630, the overall features of all frames and the overall features of key frames are spliced ​​together to obtain visual modal features.

[0168] Visual modal features can be obtained in the following ways:

[0169] F(s)=[F all (v),F key (v)],

[0170] Where [] represents the splicing operation, specifically splicing in a way that connects the beginning and end, and F(s) is the visual modal feature.

[0171] Similarly, the audio modal features F(v) can also be obtained. This can be achieved using methods similar to... Figure 6 In the embodiment, the same method of generating visual modal features is used to generate audio modal features F(v) based on the output node features corresponding to each node in the audio segment relationship graph.

[0172] Please continue reading Figure 3 As shown, after constructing all frame maps, key frame maps, all audio segment maps, and key audio segment maps, visual modal features can be generated using all frame maps and key frame maps, and audio modal features can be generated using all audio segment maps and key audio segment maps.

[0173] In step 250, the visual modal features, audio modal features, and text modal features are fused, and the fusion result is classified to obtain the video understanding result.

[0174] Figure 7 An embodiment according to this application is shown. Figure 2 A flowchart detailing step 250 in the embodiment is provided. Please refer to [link / reference]. Figure 7 As shown, fusing visual modal features, audio modal features, and text modal features can specifically include the following steps:

[0175] In step 251, information mapping is performed on the visual modal features, audio modal features, and text modal features respectively to obtain visual modal mapping features, audio modal mapping features, and text modal mapping features.

[0176] Information mapping for visual modal features, audio modal features, and text modal features can be performed using three feedforward layers, which are also known as feedforward neural network layers.

[0177] The specific calculation method for information mapping is as follows:

[0178] F′(v)=(W*F(v)+b),

[0179] F′(s)=(W*F(s)+b),

[0180] F′(t)=(W*F(t)+b),

[0181] Wherein, F(v) is the visual modal feature, F(s) is the audio modal feature, F(t) is the text modal feature, F′(v) is the visual modal mapping feature, F′(s) is the audio modal mapping feature, and F′(t) is the text modal mapping feature. W and b corresponding to F(v), F(s), and F(t) represent the parameters of the three feedforward layers used to map information of the visual modal feature, audio modal feature, and text modal feature, respectively. The parameters of these three feedforward layers are learned, and the parameters of these three feedforward layers can be different.

[0182] Please continue reading Figure 3 As shown, information mapping was performed on the three modal features respectively.

[0183] In step 252, the visual modality mapping features, audio modality mapping features, and text modality mapping features are fused to obtain the fusion result.

[0184] Figure 8 A flowchart illustrating the fusion of visual modality mapping features, audio modality mapping features, and text modality mapping features according to an embodiment of this application, is shown to obtain the fusion result. Please refer to... Figure 8 As shown, the specific steps may include the following:

[0185] In step 810, the visual modality mapping features, audio modality mapping features, and text modality mapping features are concatenated to obtain the concatenation result.

[0186] The visual modality mapping features, audio modality mapping features, and text modality mapping features are concatenated in a way that connects the beginning and end of each feature.

[0187] In step 820, the splicing result is subjected to a first linear mapping to obtain a first mapping result.

[0188] A first linear mapping can be performed on the concatenation result through a forward layer, thereby fusing features from different modalities into the same feature space.

[0189] Please continue reading Figure 3 As shown, the splicing result is linearly mapped through the forward layer.

[0190] In step 830, the first mapping result is subjected to nonlinear mapping to obtain a nonlinear mapping result.

[0191] The representational power of the model can be enhanced by performing a nonlinear mapping on the first mapping result through an activation function layer.

[0192] Please continue reading Figure 3 As shown, the splicing result is nonlinearly mapped through the activation layer.

[0193] In step 840, a second linear mapping is performed on the nonlinear mapping result to obtain the fusion result.

[0194] A second linear mapping can be performed on the nonlinear mapping result through another feedforward layer.

[0195] Specifically, the fusion result can be obtained in the following way:

[0196] P=W2*tanh(W1*[F′(v),F′(s),F′(t)]+b1)+b2,

[0197] In this context, [] represents the splicing operation, specifically splicing in a way that connects the beginning and end. W1 and b1 are the parameters of the feedforward layer used in step 820, W2 and b2 are the parameters of the feedforward layer used in step 840, W1, b1, W2, and b2 are learnable parameters, tanh is the activation function, and P is the fusion result.

[0198] After the fusion result P is processed by the softmax activation function, the probability value corresponding to each category is predicted. The category with the highest probability value can be selected as the predicted label based on the maximum a posteriori criterion, i.e., the video understanding result or the video classification result.

[0199] Please continue reading Figure 3 As shown, the nonlinear mapping result is linearly mapped through the last feedforward layer to obtain the fusion result, and the predicted category can be obtained based on the fusion result.

[0200] Figure 3 The two forward layers and the activation layer between the two forward layers actually form a classification layer. The first forward layer and the activation layer form a single layer structure of the classification layer. In order to improve the performance of the model, the classification layer can include a three-layer structure and a forward layer after the three-layer structure. Each layer structure can include a forward layer and an activation layer after that forward layer.

[0201] Predicted tags are also known as predicted categories. In video review scenarios, predicted tags can be one of the harmful categories such as vulgarity, horror, or black market activities. If a video is classified as a harmful category, then it belongs to the harmful type.

[0202] During training, this embodiment uses cross-entropy as the loss function: L = -ylog(P), and learns by minimizing the cross-entropy criterion. Here, y is the true label, and P is the fusion result, i.e., the final predicted score. In actual training, CLIP and VGGish, used to extract initial node features, are only used as feature extractors, and their parameters can remain fixed.

[0203] To verify the effectiveness of this application, the inventors conducted video recognition experiments on four types of illegal video data, evaluating both accuracy and speed. The results show that this application achieves superior performance compared to previous representative methods. The baseline model settings for comparison are as follows: The ST model has a two-stage architecture, first extracting basic features and then encoding them using a single Transformer encoder. The NT model replaces the single Transformer encoder of the ST model with three independent Transformer encoders that encode information for different modalities respectively. The DT model replaces the three-layer structure in the classification layer of the preferred embodiment of this application with a single-layer structure. The experimental results are shown in Table 1:

[0204] Model Violation 1 Violation 2 3 violations 4 violations ST model 76.2 / 176.4 65.0 / 173.9 85.3 / 174.6 79.5 / 175.1 NT model 77.9 / 169.8 68.1 / 168.3 86.5 / 166.4 81.8 / 166.0 DT model 78.4 / 141.0 70.8 / 139.9 89.6 / 142.1 85.3 / 140.3 This application 81.2 / 146.4 73.2 / 143.2 92.9 / 145.2 88.0 / 143.8

[0205] Table 1

[0206] Please refer to Table 1. Violation content 1-4 represent four types of illegal video data, which can include vulgarity, horror, and black market activities. Video recognition experiments were conducted on each model for each type of illegal video data, yielding corresponding experimental data. In Table 1, the data before " / " represents the classification accuracy (%), and the data after " / " represents the average inference speed (ms). For example, for violation content 1 type of illegal video data, the ST model achieved a classification accuracy of 76.2% and an average inference speed of 176.4ms. These results show that compared to other video recognition models, this application achieves significant improvements in both accuracy and inference performance, demonstrating its effectiveness. The ST model uses a single Transformer encoder to model multimodal information, lacking attention to modality specificity, leading to a severe performance degradation. Furthermore, since each input element in the Transformer needs to interact with all other input elements to calculate the attention score, the computational complexity is high, resulting in a long inference time. The NT model uses the same Transformer encoder to encode information from different modalities, similarly failing to consider the differences between modalities, and its inference speed is also affected. The DT model is actually an embodiment of this application. Its difference from the preferred embodiment lies only in the classification layer; it uses a single-layer feedforward neural network, which reduces the model's expressive power and consequently affects recognition accuracy. However, its model structure is simpler than the preferred embodiment, resulting in the shortest inference time. In summary, the preferred embodiment of this application offers the best overall performance in terms of both accuracy and efficiency.

[0207] In summary, with the development of the internet, especially the mobile internet, the number of both short and long videos is increasing significantly, making the rapid identification of harmful content in massive amounts of video data crucial. Videos integrate information from multiple modalities, requiring algorithms to simultaneously capture visual features (such as color, texture, and motion), audio features (such as frequency, volume, and speech), and textual features (such as keywords, themes, and sentiment) to comprehensively understand the video content. Content understanding of long videos places higher demands on the processing efficiency of algorithms. This application proposes an efficient video moderation model based on graph neural networks, which jointly models multiple modalities of information in videos, extracts key information from video streams, improves content moderation efficiency, and promotes the development of content moderation services.

[0208] The following describes an embodiment of the apparatus described in this application, which can be used to execute the video understanding method described above. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the video understanding method described above.

[0209] Figure 9A block diagram of a video understanding apparatus according to an embodiment of this application is shown. Please refer to... Figure 9 As shown, a video understanding device 900 according to an embodiment of this application includes: an encoding unit 910, a relationship graph construction unit 920, a node feature encoding unit 930, a feature generation unit 940, and a fusion and classification unit 950. The encoding unit 910 is used to encode multiple video frames of a video into video frame features, extract audio features from each audio segment of the video, and encode the text information corresponding to the video into text modal features. The relationship graph construction unit 920 is used to construct a video frame relationship graph including multiple nodes based on the video frame features corresponding to each video frame, and to construct an audio segment relationship graph including multiple nodes based on the audio features of each audio segment. The node feature encoding unit 930 is used to determine the attention relationship between each node and each of its neighboring nodes for each node in each target relationship graph, based on the features corresponding to the node and the features corresponding to each of the node's neighboring nodes. The attention weights are used to determine the output node features corresponding to each node, and the target relationship graph is either the video frame relationship graph or the audio segment relationship graph. The features are either video frame features or audio features. The feature generation unit 940 is used to generate visual modal features based on the output node features corresponding to each node in the video frame relationship graph, and to generate audio modal features based on the output node features corresponding to each node in the audio segment relationship graph. The fusion and classification unit 950 is used to fuse the visual modal features, the audio modal features, and the text modal features, and to classify the fusion result to obtain the video understanding result.

[0210] In some embodiments of this application, based on the foregoing scheme, the relational graph construction unit 920 is configured as follows:

[0211] Using the video frame features corresponding to each video frame as a node, edges are established between two nodes in each pair of nodes that satisfy the first predetermined condition, resulting in a complete frame graph including multiple nodes and edges between different nodes; and / or

[0212] According to the first predetermined rule, several key video frame features are selected from the video frame features corresponding to each video frame. Each key video frame feature is used as a node, and an edge is established between the two nodes in each pair of nodes to obtain a key frame graph that includes multiple nodes and edges between different nodes.

[0213] In some embodiments of this application, based on the foregoing scheme, the first predetermined condition is one of the following: the time interval between video frames corresponding to two nodes is less than a predetermined time interval threshold; the feature similarity between two nodes is less than a first predetermined similarity threshold; the time interval between video frames corresponding to two nodes is less than a predetermined time interval threshold, or the feature similarity between two nodes is less than a first predetermined similarity threshold; the time interval between video frames corresponding to two nodes is less than a predetermined time interval threshold, and the feature similarity between two nodes is less than a first predetermined similarity threshold.

[0214] In some embodiments of this application, based on the aforementioned scheme, the relationship graph construction unit 920 includes a full frame graph construction subunit and a key frame graph construction subunit; the key frame graph construction subunit is configured to: select the first key video frame feature from the video frame features corresponding to each video frame as the current benchmark feature; repeatedly execute the key video frame feature filtering step until the last key video frame feature is filtered out; the key video frame feature filtering step includes: selecting the next key video frame feature from the video frame features corresponding to the video frames after the benchmark video frame corresponding to the current benchmark feature according to the temporal sequence of each video frame, wherein the video frame corresponding to the next key video frame feature is the first video frame in the video frames after the benchmark video frame where the feature similarity between the corresponding video frame feature and the current benchmark feature is less than a second predetermined similarity threshold; and using the next key video frame feature as the new current benchmark feature.

[0215] In some embodiments of this application, based on the foregoing scheme, the video frame relationship graph includes the all-frame graph and the keyframe graph. The feature generation unit 940 is configured to: generate the overall feature of the all-frame graph based on the output node features corresponding to each node of the all-frame graph; generate the overall feature of the keyframe graph based on the output node features corresponding to each node of the keyframe graph; and perform a splicing operation on the overall feature of the all-frame graph and the overall feature of the keyframe graph to obtain the visual modal feature.

[0216] In some embodiments of this application, based on the foregoing scheme, the relational graph construction unit 920 is configured as follows:

[0217] Using the audio feature corresponding to each audio segment as a node, edges are established between two nodes in each pair of nodes that satisfy the second predetermined condition, resulting in a complete audio segment graph including multiple nodes and edges between different nodes; and / or

[0218] According to the second predetermined rule, several key audio features are selected from the audio features corresponding to each audio segment. Each key audio feature is used as a node, and an edge is established between the two nodes in each pair of nodes to obtain a key audio segment graph that includes multiple nodes and edges between different nodes.

[0219] In some embodiments of this application, based on the foregoing scheme, the fusion and classification unit 950 is configured to: perform information mapping on the visual modal features, the audio modal features and the text modal features respectively to obtain visual modal mapping features, audio modal mapping features and text modal mapping features; and fuse the visual modal mapping features, the audio modal mapping features and the text modal mapping features to obtain a fusion result.

[0220] In some embodiments of this application, based on the foregoing scheme, the fusion and classification unit 950 is configured to: concatenate the visual modality mapping features, the audio modality mapping features, and the text modality mapping features to obtain a concatenation result; perform a first linear mapping on the concatenation result to obtain a first mapping result; perform a nonlinear mapping on the first mapping result to obtain a nonlinear mapping result; and perform a second linear mapping on the nonlinear mapping result to obtain a fusion result.

[0221] In some embodiments of this application, based on the aforementioned scheme, the node feature encoding unit 930 is configured to: normalize each of the attention weights to obtain normalized attention weights; for each neighbor node, determine the feature corresponding to the neighbor node, the normalized attention weights between the node and the neighbor node, and the product of the parameter matrix; determine the sum of the products determined for each neighbor node; and perform a nonlinear mapping on the sum of the products to obtain the output node features.

[0222] Figure 10 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.

[0223] It should be noted that, Figure 10 The computer system 1000 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0224] like Figure 10As shown, the computer system 1000 includes a Central Processing Unit (CPU) 1001, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 1002 or programs loaded from storage portion 1008 into Random Access Memory (RAM) 1003, such as performing the methods described in the above embodiments. Various programs and data required for system operation are also stored in RAM 1003. The CPU 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. An Input / Output (I / O) interface 1005 is also connected to bus 1004.

[0225] The following components are connected to I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to I / O interface 1005 as needed. Removable media 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1010 as needed so that computer programs read from them can be installed into storage section 1008 as needed.

[0226] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1009, and / or installed from removable medium 1011. When the computer program is executed by central processing unit (CPU) 1001, it performs various functions defined in the system of this application.

[0227] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0228] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0229] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0230] In one aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0231] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0232] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0233] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0234] The data collection and processing plan outlined in this application must be implemented in strict accordance with the requirements of relevant national laws and regulations, obtaining the informed consent or separate consent of the data subject (or having a legal basis as stipulated by the relevant national laws and regulations), and conducting subsequent data use and processing within the scope authorized by laws and regulations and the data subject.

[0235] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A video understanding method, characterized in that, The method includes: The video frames are encoded into video frame features, the audio features of each audio segment of the video are extracted, and the text information corresponding to the video is encoded into text modal features. A video frame relationship graph with multiple nodes is constructed based on the video frame features corresponding to each video frame, and an audio segment relationship graph with multiple nodes is constructed based on the audio features of each audio segment. For each node in each target relationship graph, attention weights between the node and each of its neighboring nodes are determined based on the features corresponding to the node and the features corresponding to each of the node's neighboring nodes. Output node features corresponding to the node are then determined based on the attention weights and the features corresponding to each of the neighboring nodes. The target relationship graph is either the video frame relationship graph or the audio segment relationship graph, and the features are either video frame features or audio features. Visual modal features are generated based on the output node features corresponding to each node in the video frame relationship graph, and audio modal features are generated based on the output node features corresponding to each node in the audio segment relationship graph. The visual modal features, audio modal features, and text modal features are fused, and the fusion result is classified to obtain the video understanding result.

2. The video understanding method according to claim 1, characterized in that, The step of constructing a video frame relationship graph including multiple nodes based on the video frame features corresponding to each video frame includes: Using the video frame features corresponding to each video frame as a node, edges are established between two nodes in each pair of nodes that satisfy the first predetermined condition, resulting in a complete frame graph including multiple nodes and edges between different nodes; and / or According to the first predetermined rule, several key video frame features are selected from the video frame features corresponding to each video frame. Each key video frame feature is used as a node, and an edge is established between the two nodes in each pair of nodes to obtain a key frame graph that includes multiple nodes and edges between different nodes.

3. The video understanding method according to claim 2, characterized in that, The first predetermined condition is one of the following: The time interval between the video frames corresponding to the two nodes is less than a predetermined time interval threshold; The feature similarity between two nodes is less than a first predetermined similarity threshold; The time interval between the video frames corresponding to the two nodes is less than a predetermined time interval threshold, or the feature similarity between the two nodes is less than a first predetermined similarity threshold. The time interval between the video frames corresponding to the two nodes is less than a predetermined time interval threshold, and the feature similarity between the two nodes is less than a first predetermined similarity threshold.

4. The video understanding method according to claim 2, characterized in that, The step of selecting several key video frame features from the video frame features corresponding to each video frame according to a first predetermined rule includes: Select the first key video frame feature from the video frame features corresponding to each video frame, and use it as the current baseline feature; Repeat the key video frame feature filtering step until the last key video frame feature is filtered out; the key video frame feature filtering step includes: According to the time sequence of each video frame, the next key video frame feature is selected from the video frame features corresponding to the video frames after the current benchmark feature. The video frame corresponding to the next key video frame feature is the first video frame in the video frames after the benchmark video frame where the feature similarity between the corresponding video frame feature and the current benchmark feature is less than a second predetermined similarity threshold. The features of the next key video frame are used as the new current baseline features.

5. The video understanding method according to claim 2, characterized in that, The video frame relationship graph includes the all frame graphs and the key frame graphs. Generating visual modal features based on the output node features corresponding to each node in the video frame relationship graph includes: Generate the overall features of all frame graphs based on the output node features corresponding to each node of all frame graphs; Generate the overall features of the keyframe graph based on the output node features corresponding to each node of the keyframe graph; The overall features of all the frames and the overall features of the key frames are spliced ​​together to obtain the visual modal features.

6. The video understanding method according to claim 1, characterized in that, The construction of an audio segment relationship graph comprising multiple nodes based on the audio features of each audio segment includes: Using the audio feature corresponding to each audio segment as a node, edges are established between two nodes in each pair of nodes that satisfy the second predetermined condition, resulting in a complete audio segment graph including multiple nodes and edges between different nodes; and / or According to the second predetermined rule, several key audio features are selected from the audio features corresponding to each audio segment. Each key audio feature is used as a node, and an edge is established between the two nodes in each pair of nodes to obtain a key audio segment graph that includes multiple nodes and edges between different nodes.

7. The video understanding method according to claim 1, characterized in that, The fusion of the visual modal features, the audio modal features, and the text modal features includes: Information mapping is performed on the visual modal features, the audio modal features, and the text modal features respectively to obtain visual modal mapping features, audio modal mapping features, and text modal mapping features; The visual modality mapping features, the audio modality mapping features, and the text modality mapping features are fused to obtain a fusion result.

8. The video understanding method according to claim 7, characterized in that, The process of fusing the visual modality mapping features, the audio modality mapping features, and the text modality mapping features to obtain a fusion result includes: The visual modality mapping features, the audio modality mapping features, and the text modality mapping features are concatenated to obtain the concatenation result; Perform a first linear mapping on the splicing result to obtain a first mapping result; Perform a nonlinear mapping on the first mapping result to obtain a nonlinear mapping result; The nonlinear mapping result is then subjected to a second linear mapping to obtain the fusion result.

9. The video understanding method according to any one of claims 1-8, characterized in that, The step of determining the output node features corresponding to the node based on each attention weight and the features corresponding to each neighbor node includes: Each attention weight is normalized to obtain the normalized attention weights. For each neighboring node, determine the features corresponding to the neighboring node, the normalized attention weights between the node and the neighboring node, and the product of the parameter matrices. Determine the sum of the products determined for each of the neighboring nodes; The sum of the products is nonlinearly mapped to obtain the output node features.

10. A video understanding device, characterized in that, The device includes: The encoding unit is used to encode multiple video frames of the video into video frame features, extract audio features of each audio segment of the video, and encode the text information corresponding to the video into text modal features. The relationship graph construction unit is used to construct a video frame relationship graph including multiple nodes based on the video frame features corresponding to each video frame, and to construct an audio segment relationship graph including multiple nodes based on the audio features of each audio segment; A node feature encoding unit is used to determine the attention weight between the node and each of the neighboring nodes for each node in each target relationship graph, based on the features corresponding to the node and the features corresponding to each of the neighboring nodes, and to determine the output node feature corresponding to the node based on the attention weights and the features corresponding to each of the neighboring nodes, wherein the target relationship graph is the video frame relationship graph or the audio segment relationship graph, and the feature is the video frame feature or the audio feature; The feature generation unit is used to generate visual modal features based on the output node features corresponding to each node of the video frame relationship graph, and to generate audio modal features based on the output node features corresponding to each node of the audio segment relationship graph. The fusion and classification unit is used to fuse the visual modal features, the audio modal features, and the text modal features, and to classify the fusion result to obtain the video understanding result.

11. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the video understanding method as described in any one of claims 1 to 9.

12. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the video understanding method as described in any one of claims 1 to 9.

13. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium, a processor of a computer device reading the computer instructions from the computer-readable storage medium, and the processor executing the computer instructions to cause the computer device to perform the video understanding method as described in any one of claims 1 to 9.

Images