Video classification method, apparatus, device, and readable storage medium

Abstract

This application discloses a video classification method, apparatus, device, and readable storage medium, relating to the field of machine learning. The method includes: acquiring a target video; extracting features from the target video based on video frames to obtain video features corresponding to the target video; performing semantic analysis on the target video based on the video features to obtain semantic features corresponding to the target video; fusing the video features and semantic features to obtain fused features corresponding to the target video; and performing classification prediction on the fused features to obtain the video category to which the target video belongs. In this process, not only is the classification information contained in the video features fully considered, but also the semantic features corresponding to the video features are extracted and incorporated into the analysis. This expands the scope of video feature information acquisition, avoids incomplete information inclusion during video classification, and improves the accuracy of video classification.

Description

Technical Field

[0001] This application relates to the field of machine learning, and in particular to a video classification method, apparatus, device, and readable storage medium. Background Technology

[0002] Video classification is an important topic in the field of computer vision. Its main goal is to understand the content contained in a video and identify several key themes. Because video classification is of great significance in applications such as video retrieval and content analysis, effectively classifying videos has become a crucial problem that urgently needs to be solved.

[0003] In related technologies, video classification mainly focuses on behavior recognition and event detection. This typically involves extracting video features and then using machine learning to determine the video's category. This usually requires a large number of training samples to train the video classifier to distinguish video categories.

[0004] However, in many video classification tasks, there are very few training samples available, and the extracted video features contain limited information, resulting in low accuracy in video classification. Summary of the Invention

[0005] This application provides a video classification method, apparatus, device, and readable storage medium, which can effectively improve the accuracy of video classification. The technical solution is as follows:

[0006] On the one hand, a video classification method is provided, the method comprising:

[0007] Acquire the target video, which is the video to be classified and predicted;

[0008] Based on the video frames in the target video, feature extraction is performed on the target video to obtain the video features corresponding to the target video. The video features are used to indicate the picture information expressed by the video frames of the target video.

[0009] Based on the video features, semantic analysis is performed on the target video to obtain the semantic features corresponding to the target video. The semantic features are used to indicate the entity association relationships expressed by the video frames of the target video.

[0010] The video features and the semantic features are fused to obtain the fused features corresponding to the target video;

[0011] The fused features are used to perform classification prediction to obtain the video category to which the target video belongs.

[0012] On the other hand, a video classification device is provided, the device comprising:

[0013] The acquisition module is used to acquire the target video, which is the video to be classified and predicted;

[0014] The extraction module is used to extract features from the target video based on the video frames in the target video to obtain video features corresponding to the target video. The video features are used to indicate the picture information expressed by the video frames of the target video.

[0015] The analysis module is used to perform semantic analysis on the target video based on the video features to obtain the semantic features corresponding to the target video. The semantic features are used to indicate the entity association relationships expressed by the video frames of the target video.

[0016] The fusion module is used to fuse the video features and the semantic features to obtain the fused features corresponding to the target video;

[0017] The prediction module is used to perform classification prediction on the fused features to obtain the video category to which the target video belongs.

[0018] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement any of the video classification methods described in the embodiments of this application above.

[0019] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored therein, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the video classification method as described in any of the embodiments of this application above.

[0020] On the other hand, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the video classification methods described in the above embodiments.

[0021] The beneficial effects of the technical solutions provided in this application include at least the following:

[0022] After extracting features from the target video to be detected, video features can be obtained. Then, semantic features are extracted from the obtained video features. The semantic information is then fused with the video features to obtain fused features. At this time, the fused features not only contain visual information but also semantic information, which expands the scope of video feature information acquisition, avoids the situation of incomplete information in the process of video classification, and improves the accuracy of video classification. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of an implementation environment provided by an exemplary embodiment of this application;

[0025] Figure 2 This is a schematic diagram illustrating the implementation process of a solution provided in an exemplary embodiment of this application;

[0026] Figure 3 This is a flowchart of a video classification method provided in an exemplary embodiment of this application;

[0027] Figure 4 This is a flowchart of a video classification method provided in another exemplary embodiment of this application;

[0028] Figure 5 This is a flowchart of a video classification method provided in another exemplary embodiment of this application;

[0029] Figure 6 This is a flowchart of a video classification method provided in another exemplary embodiment of this application;

[0030] Figure 7 This is a schematic diagram of a video method process provided in an exemplary embodiment of this application;

[0031] Figure 8 This is a schematic diagram illustrating the accuracy of video classification obtained using a video classification method, provided in an exemplary embodiment of this application.

[0032] Figure 9 This is a structural block diagram of a video classification apparatus provided in an exemplary embodiment of this application;

[0033] Figure 10 This is a structural block diagram of a video classification apparatus provided in another exemplary embodiment of this application;

[0034] Figure 11 This is a structural block diagram of a server provided in an exemplary embodiment of this application. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0036] First, a brief introduction to the terms used in the embodiments of this application will be given.

[0037] Prototype Networks: Prototype networks are a metric-based meta-learning method designed to mitigate overfitting caused by insufficient data. The basic idea is to create a prototype representation for each category. For a video to be classified, the distance between the prototype vector and the query sample is calculated. Illustratively, the query sample is projected onto the prototype space, where samples of the same class are closer and samples of different classes are farther apart. The query sample is then projected into the prototype network, and the category closest to the query sample is taken as the category of the query sample.

[0038] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instruction-based learning.

[0039] Computer Vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing, tracking, and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0040] In related technologies, video classification requires training a mature classification model. Training this model necessitates a large amount of labeled video data. The model classifies and identifies the sample images, and the differences between the labeled data and the identification results are used to train the model. However, when the training sample size is small, some deep learning methods fail to demonstrate their advantages in video recognition. Furthermore, a limited training sample size may lead to videos of different categories being grouped into the same category simply because of similar video features, ultimately resulting in low accuracy in video classification.

[0041] This application provides a video classification method that improves training efficiency and accuracy when the number of samples is small. The video classification method trained according to this application can be applied in at least one of the following scenarios.

[0042] First, in the scenario of video classification in the internet film and television zone, illustratively speaking, for a newly added target video in the film and television zone, it is necessary to classify it according to its category. In this case, the target video undergoes feature extraction to obtain video features. Then, these video features are processed by a semantic generation network to generate semantic features. The video features and semantic features are then fused to obtain fused features. These fused features are input into a graph neural network for relation propagation to obtain more robust features. The target node is compared with other feature nodes, and the behavior category of the feature node with the smallest distance to the target node is taken as the category to which the target video belongs.

[0043] Secondly, in short video push services, illustratively speaking, the categories of short videos a user most frequently watches on short video apps can roughly determine the categories of short videos a user likes. Short videos of relevant categories can then be pushed to the user based on their preferences. Therefore, effective classification of short videos is necessary. In the application, the short video that determines whether to push to the user is taken as the target video. Video features are extracted from the target video to generate semantic features. The video features and semantic features are fused to obtain fused features. After the fused features are input into a graph neural network for relation propagation, the target node is compared with other feature nodes. The feature node with the smallest distance to the target node is taken as the category corresponding to the short video, thereby determining whether to push the short video to the user.

[0044] Third, user-uploaded videos are categorized. For example, after a user uploads a video, it needs to be partitioned, meaning the video category needs to be determined. In the application process, after a user uploads a video, semantic features are generated through video feature extraction. These video features and semantic features are then fused to obtain fused features. These fused features are input into a graph neural network for relation propagation. The target node is then compared with other feature nodes, and the behavior category of the feature node with the smallest distance to the target node is taken as the category to which the target video belongs, thus achieving video classification.

[0045] The target nodes mentioned above are the nodes of the target video in the graph neural network. It is worth noting that the above application scenarios are merely illustrative examples; the video classification method provided in this application embodiment can also be applied to other scenarios, such as pushing videos of the same category based on user searches. This application embodiment does not limit this application to such applications.

[0046] Secondly, the implementation environment involved in the embodiments of this application will be described, for illustrative purposes only. Please refer to [the relevant documentation]. Figure 1 The implementation environment involves a terminal 110 and a server 120, which are connected via a communication network 130.

[0047] In some embodiments, terminal 110 is used to send videos to be classified to server 120. In some embodiments, terminal 110 is equipped with an application with classification function; illustratively, terminal 110 is equipped with an application with video feature extraction; or, terminal 110 is equipped with an application with semantic feature extraction.

[0048] Server 120 includes classification results predicted by a video classification model. After classifying the video to be classified according to the classification results, the classification results are output and then fed back to terminal 110 for display.

[0049] The video classification model is trained using this video classification method on sample videos from a sample video library. Feature extraction is performed on video frames from the sample videos in the library to obtain video features. Then, semantic analysis is performed on the sample videos based on these video features to obtain semantic features. Finally, the video features and semantic features are fused to obtain the fused features, and the video category is determined based on these fused features. This process illustrates a non-unique scenario in the training of a video classification model.

[0050] The aforementioned terminal can be various forms of terminal devices such as mobile phones, tablets, desktop computers, portable laptops, and smart TVs, and this application embodiment does not limit this to any particular type.

[0051] It is worth noting that the aforementioned servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0052] Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Based on the cloud computing business model, cloud technology encompasses network technology, information technology, integration technology, management platform technology, and application technology. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing.

[0053] In some embodiments, the server described above can also be implemented as a node in a blockchain system. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and cryptographic algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer.

[0054] Indicative, Figure 2 This is a schematic diagram of the overall implementation process of an exemplary embodiment of this application, taking the video classification process of an internet film and television platform as an example, such as... Figure 2 As shown, the method includes:

[0055] Step 210: Obtain the target video.

[0056] The methods for acquiring the target video include at least one of the following: First, randomly selecting a video from an existing video library and classifying the selected video as the target video; second, acquiring a self-recorded video as the target video and then classifying it; third, the server receiving a video uploaded by a terminal and classifying the uploaded video as the target video. The above methods are merely brief examples of target video acquisition methods and are not intended to limit the scope of this embodiment.

[0057] The target video is the video to be classified and predicted. This target video includes both longer videos such as movies and TV shows, as well as shorter short videos played or pushed through various media channels. Optionally, any video of a certain duration consisting of continuous image frames can be classified using the video classification method provided in this application embodiment. Illustratively, the target video may be a TV series video, a movie video, a music video, or a short video, etc., but this application embodiment does not limit this to any specific type.

[0058] Step 220: Extract features from the target video based on the video frames in the target video to obtain the video features corresponding to the target video.

[0059] Among them, video features are used to indicate the picture information expressed by the video frames of the target video.

[0060] A frame is the smallest unit of network transmission, and a relatively complete and independent segment of information is generally divided into one frame. According to the principle of visual persistence, if continuous image changes exceed 24 frames per second, the human eye will be unable to distinguish a single static image. However, if multiple static images change sequentially, the human eye perceives a smooth and continuous visual effect, and this continuous sequence of images is called video.

[0061] Video frames contain visual features of a video within a short period of time. Compared to feature extraction from the entire video, feature extraction from video frames is more accurate and less costly. Therefore, video features obtained from video frames have better feature extraction results and are more accurate than those obtained by directly extracting features from the entire video.

[0062] There are several methods for extracting video frames. For example, random extraction can be used: after acquiring a video segment, import it into video editing software, randomly select one or more points in the video, and then click the "Export Frames" button to achieve random frame extraction. Alternatively, video frames can be extracted at preset intervals: after acquiring a video segment, import it into video editing software, set a preset interval for frame extraction, and then click the "Export Frames" button to obtain video frames with equal time intervals. Another method is segmented extraction: divide the target video into n equal-length video segments, and then randomly sample from these n segments to obtain video frames, where n is a positive integer.

[0063] Furthermore, there are various methods for video feature extraction. One approach is to extract frames from the video object, obtaining one or more frame images. Then, various types of pooling are performed on each frame image to obtain its image features. Finally, the video features are determined by combining the image features from one or more frame images. Alternatively, three-dimensional convolution operations can be used to extract video features.

[0064] Step 230: Perform semantic analysis on the target video based on video features to obtain the semantic features corresponding to the target video.

[0065] Among them, semantic features are used to indicate the entity relationships expressed by the video frames of the target video.

[0066] Semantics refers to how to utilize information from images, especially high-level information, to provide a way of describing images for research. In the context of video, semantics includes the temporal and spatial relationships between important objects in the video, as well as the implied content behind them. Semantic analysis can reveal features in the video that are not directly apparent from visuals and sound. Therefore, effectively extracting semantic features can help analyze videos from multiple perspectives, obtaining various temporal and spatial characteristics. Processing these features can improve the accuracy of video classification.

[0067] Semantic feature extraction is based on the description of low-level features in videos. Videos contain rich information. Although low-level features are too straightforward in describing video objects and cannot reflect deeper information, they are often obtained directly from statistical calculations on video data and represent the information level closest to the video data. Therefore, effective extraction of semantic features can help improve the accuracy of video classification.

[0068] In this context, the entity association relationship expressed by a video frame refers to the relationship between different things that can express the meaning of entities. In this embodiment, the entity association relationship refers to the relationship between objects within the same video frame or between different video frames that can express the meaning of entities. Illustratively, the entity relationship is used to indicate the association between object A and object B in a video frame, such as: object A is located on top of object B.

[0069] Step 240: Fuse video features and semantic features to obtain fused features corresponding to the target video.

[0070] Among them, video features are used to indicate the picture information expressed by the video frames of the target video.

[0071] In this application example, both video features and semantic features are mapped to corresponding vectors on a prototype network. Therefore, the vectors corresponding to video features and semantic features are relatively definite. The prototype network is pre-trained and reflects the feature vectors obtained after feature extraction from the input target video.

[0072] Fusion involves adding elements of the vectors corresponding to the video features and semantic features in the target video. Fusion of video and semantic features aims to make the information conveyed by the video frames more comprehensive and complete. The fused video and semantic features of the target video result in a fused feature, which is represented as a target node in the prototype network.

[0073] Since the aforementioned fusion features include both video features and semantic features, and video features include both visual and non-visual features (such as the aforementioned behavioral features), the feature information is relatively rich, thus the prediction results obtained by using the fusion features are also relatively accurate.

[0074] Step 250: Perform classification prediction on the fused features to obtain the video category to which the target video belongs.

[0075] The fused features at this point include not only video features but also semantic features. Based on this, video classification prediction can be performed, allowing for better category determination based on the different video tag information pointed to by the features. Here, video tag information refers to the features that distinguish one video from another.

[0076] To illustrate, we first acquire sample videos labeled with category tags, and then extract the sample fusion features of the sample videos. The sample fusion features include the sample semantic features and sample video features of the sample videos. The method for generating the sample fusion features is similar to the method for processing the target videos.

[0077] Secondly, a graph neural network is constructed based on the requirements of intra-class consistency and inter-class differences. The graph neural network is obtained through training, and the nodes in the graph neural network are composed of sample fusion features and target fusion features.

[0078] Subsequently, the fused features are input into a graph neural network (Graph Neural Network) for relation propagation to obtain more robust fused features. The target node is compared with the remaining feature nodes in the Graph Neural Network, and the behavior category of the feature node with the smallest distance to the target node is taken as the category to which the target video belongs. In this embodiment, the nodes of the Graph Neural Network are the fused features, and relation propagation is used to update the nodes in the Graph Neural Network to obtain enhanced fused features. The target fused features and sample fused features are input into the Graph Neural Network to obtain enhanced fused features of the target fused features and enhanced fused features of the sample fused features, and then the two are compared. In addition, the Graph Neural Network classifies the similarity degree by node distance; therefore, it is necessary to perform distance analysis between the target node and the remaining nodes in the Graph Neural Network to obtain the feature distance between the fused features and the remaining nodes. Here, the target node is the fused feature corresponding to the target video.

[0079] Then, the analysis node with the smallest feature distance to the fused feature in the graph neural network is determined. Illustratively, in this embodiment, distance analysis is performed between the target node and the other nodes in the graph neural network. If the target node is close to the other nodes in the graph neural network, it indicates a high degree of similarity between the target video and the sample video corresponding to that node; conversely, if the target node is far from the other nodes in the graph neural network, it indicates a low degree of similarity between the target video and the sample video corresponding to that node.

[0080] In a schematic representation, clustering results can also be determined based on the distance between the fusion features and the cluster centers. In this approach, the input sample data in the neural network is divided into several parts based on differences in sample features. These parts represent different feature meanings and are each called a cluster. The average value of the features within each cluster is used to obtain the cluster center, which can more accurately describe the core features of a cluster. The clustering result is determined based on the distance between the fusion features and the cluster centers of each cluster.

[0081] It is worth noting that the above-described feature distance comparison method is merely an illustrative example, and the embodiments of this application do not limit it.

[0082] The category corresponding to the analysis node is used as the video category to which the target video belongs.

[0083] Since the fused feature is close to the other nodes in the graph neural network, it indicates a high degree of similarity between the target video and the sample video. Therefore, the classification corresponding to this analysis node is most similar to the category of the target video. Thus, the classification corresponding to the analysis node can be taken as the video category to which the target video belongs.

[0084] In summary, the video classification method provided in this embodiment, after acquiring the target video, extracts features from the video frames within the target video to obtain the corresponding video features. Then, it performs semantic analysis on the target video based on these video features to obtain the corresponding semantic features. Next, it fuses the video features and semantic features to obtain the fused features corresponding to the target video. Finally, it performs classification prediction on the fused features to determine the video category to which the target video belongs. In this process, fusing video features and semantic features expands the scope of video feature information acquisition, resulting in richer and more discriminative features, thus improving the accuracy of video classification.

[0085] This illustration illustrates the acquisition of video features in step 220 of Embodiment 2 of this application. Please refer to [link / reference]. Figure 3 The process of extracting features from the target video based on its video frames to obtain the corresponding video features includes the following steps:

[0086] Step 310: Extract at least two video frames from the target video.

[0087] Among them, at least two video frames have a temporal relationship.

[0088] The term "temporal sequence" refers to the chronological order of events. In this application, the temporal relationship means that when extracting video frames from a target video, not only should the chronological order be considered, but also whether the time intervals between the extracted video frames conform to a certain pattern, including but not limited to the equality of time intervals between the extracted video frames.

[0089] As an illustration, a method for extracting video frames may include the following process: segmenting the target video to obtain at least two video segments, and sampling video frames from the at least two video segments to obtain at least two video frames. For example, dividing the target video into 8 equal segments and randomly extracting 1 frame from each segment will yield 8 video frames. Through the above operations, at least two video frames with a temporal relationship can be obtained.

[0090] Step 320: Extract video features of the target video frame based on the temporal relationship and picture of at least two video frames.

[0091] Specifically, based on the temporal relationships and image features extracted from at least two video frames in the previous operations, the video frames are interpreted from two-dimensional and three-dimensional perspectives, respectively. By performing multi-dimensional analysis and extraction on the target video frames, more accurate video features can be extracted, which helps improve the accuracy of the final video classification results.

[0092] Step 330: Extract behavioral features between at least two video frames based on the temporal relationship between at least two video frames;

[0093] Among them, the temporal relationship between at least two video frames is extracted, focusing on the two-dimensional analysis of the video frames, and temporal extraction can be performed based on the temporal relationship.

[0094] Indicatively, given an input feature X∈R N×T×C×H×W Where N is the batch size, T is the number of video frames, C is the number of channels, and H and W are the height and width of the video frames. Using the original input features, we can obtain the corresponding adjacent video frames, X. t+1 and X t First, a 3×3 2D convolution is used to process the original feature X. t Therefore, the time-difference process can be described as:

[0095] H(t) = conv2(X) t+1 )-X t , 1≤t≤T-1

[0096] Here, H(t) represents the behavioral feature at time t, and conv2 represents a 3×3 2D convolution operation. Specifically, for the last time T, we set the behavioral feature to 0, i.e., H(T) = 0. Then, we concatenate all the behavioral features together, i.e., H = [H(1), ..., H(T)], where [·] represents the concatenation operation.

[0097] The features are obtained by extracting features from the original features using a pre-trained feature extraction network.

[0098] Step 340: Extract visual features of the target video frame based on at least two video frames.

[0099] This involves extracting images between at least two video frames, focusing on the three-dimensional angle analysis of the video frames, and performing spatial extraction based on the images.

[0100] In addition, different channels contain a wealth of visual information.

[0101] Here, a channel refers to the region where video frame information flows. Adjacent channels refer to the regions where video frame information flows adjacent to the video frame being analyzed. Therefore, the differences between adjacent channels can be used to enhance the visual modeling capability of the video classification method in this application.

[0102] Indicatively, the input visual features are divided into C feature subsets. Given two adjacent features, X c and X c+1 First, a 3×3 2D convolution is used to process feature X. c+1The channel differential process is defined as follows:

[0103] S(c) = conv1(X c+1 )-X c , 1≤c≤C-1

[0104] Here, S(c) represents the visual feature of channel c, and conv1 represents a 3×3 2D convolution operation. In particular, to ensure that the dimensions of the obtained features are consistent with those of the original features, we set the visual feature of the last channel to zero, i.e., S(C) = 0.

[0105] The spatial and temporal extraction methods mentioned above can be refined into a spatiotemporal difference module, which means that at least two video frames are processed through the spatiotemporal difference module to extract the video features of the target video frame.

[0106] Step 350: The behavioral features and visual features are fused to obtain the video features of the target video frame.

[0107] The process of fusing behavioral and visual features to obtain video features of the target video frame includes the following two steps:

[0108] In some embodiments, behavioral features and visual features are fused to obtain hybrid features.

[0109] In a demonstrative manner, the temporal relationship and visual characteristics between at least two video frames are analyzed using both temporal and spatial extraction methods. Ultimately, extracting the temporal relationship between at least two video frames yields behavioral features at different times; extracting the visual characteristics of at least two video frames yields visual features from different channels. Extracting features separately along both temporal and spatial dimensions avoids incomplete feature extraction due to extracting features from a single dimension, as well as inaccurate feature extraction due to extracting features from both dimensions simultaneously, effectively improving the accuracy of feature extraction.

[0110] Next, the behavioral features extracted using the temporal extraction method are fused with the visual features extracted using the spatial extraction method to obtain hybrid features. These hybrid features now include information from both the extracted behavioral and visual features, providing a more comprehensive and accurate coverage of the video frame information.

[0111] Visual features and mixed features are connected using residual connections to obtain video features.

[0112] The hybrid features include behavioral features extracted using temporal extraction methods and visual features extracted using spatial extraction methods. While the features are comprehensive, gradient vanishing inevitably occurs after feature extraction. To mitigate this, performing a residual connection operation between the input visual features and the hybrid features effectively reduces this phenomenon, resulting in enhanced visual features, i.e., video features.

[0113] Residual connections represent the output as a linear superposition of the input and a nonlinear transformation of the input. The purpose of residual connections is to address the gradient vanishing phenomenon that occurs as network depth increases during deep learning.

[0114] Gradient vanishing refers to the phenomenon in neural networks where, as the number of hidden layers increases, the learning rate of the preceding hidden layers becomes lower than that of the subsequent hidden layers, causing the classification accuracy to stop increasing and instead decrease.

[0115] In summary, the video classification method provided in this embodiment explains how to obtain corresponding video features from a target video. The initial visual features obtained after feature extraction from the target video contain various types of noise; therefore, a method should be fully considered to obtain enhanced visual features while reducing noise. Analysis of the target video reveals that extracting features from at least two video frames with a temporal relationship yields more accurate video features.

[0116] Because video frame analysis considers not only the temporal relationships between frames but also the visual aspects of the frame time. By fusing the behavioral features obtained from temporal relationship analysis with the visual features obtained from visual aspect analysis, and then performing a residual connection operation on the visual features, enhanced video visual features, i.e., video features, are obtained. This process reduces the inaccuracies in visual features obtained directly from the video due to noise interference, thus improving the accuracy of extracting corresponding video features from the target video.

[0117] This illustration is intended to explain the acquisition of semantic features in step 230 of Embodiment 2 of this application. Please refer to [link / reference]. Figure 4 The process of performing semantic analysis on the target video based on video features to obtain the corresponding semantic features of the target video includes the following steps 410 to 420.

[0118] Step 410: Input the video features into the semantic generation network.

[0119] The semantic generation network is a semantic feature generation network obtained after training. A semantic network refers to a network containing multiple concepts and instances used to connect the relationships between two concepts. Semantic generation refers to the process of analyzing the described visual information. A semantic generation network is a network capable of generating semantics through the relationships between multiple concepts.

[0120] In this application example, semantic generation refers to the process of analyzing the described video information; the semantic generation network refers to a network that expresses entity relationships through video frames.

[0121] The purpose of inputting video features into a semantic generation network is to use the trained semantic generation network to learn the semantic information corresponding to the video features from the video features.

[0122] Step 420: Perform semantic analysis on the video features using a semantic generation network to output the semantic features corresponding to the target video.

[0123] In some embodiments, prototype acquisition is performed on video features to obtain prototype features corresponding to the video features.

[0124] Here, prototype features are the feature mapping results of video features in the prototype space. The prototype space is a space obtained through multiple training iterations that fully reflects the relationship between the original video and the processed video. By inputting the video features of the target video into the prototype space, prototype features corresponding to the video features can be obtained after prototype extraction.

[0125] The semantic generator network performs semantic analysis on the prototype features and outputs the semantic features corresponding to the target video.

[0126] After extracting prototypes from the video features, prototype features corresponding to the video features are obtained. Semantic analysis is then performed on the prototype features by passing them through a previously trained semantic generation network. This trained semantic generation network can express the entity relationships between semantic features and generate semantic features corresponding to the target video.

[0127] In summary, the video classification method provided in this embodiment focuses on the acquisition of semantic features. Semantic features are extracted from video features, specifically through a semantic generation network. This semantic generation network is a trained feature generation network; video features are processed by this network to generate semantic features. The semantic feature extraction network is trained by constructing a training task, thus obtaining the semantic generation network. This approach reduces the likelihood of inaccurate semantic feature extraction from video features when the number of sample videos is small, thereby improving the prediction accuracy of the semantic generation network.

[0128] This is illustrative; please refer to it. Figure 5 In Embodiment 3 of this application, before step 310 involves inputting video features into the semantic generation network, a training process for the semantic generation network is also included. The training of the semantic generation network is performed synchronously with the training process of the video feature extraction network. The training process of the semantic generation network includes the following steps:

[0129] Step 510: Obtain sample videos, which are labeled with category tags.

[0130] The category label indicates the reference video category to which the sample video belongs. Classifying videos using label information helps to clearly define the category to which the sample video belongs.

[0131] Step 520: Extract features from the sample video to obtain the sample video features corresponding to the sample video.

[0132] Among them, video features are used to indicate the picture information expressed by the video frames of the target video.

[0133] Feature extraction of the sample video here is performed by extracting features from the video frames of the sample video. The specific operation is similar to the method of extracting video features from the target video.

[0134] The extraction of video features from the sample video involves obtaining the video features of the sample video by performing spatial extraction and temporal extraction through the aforementioned spatiotemporal difference module.

[0135] In a schematic manner, at least two video frames with a temporal relationship are extracted from the sample video. Based on the temporal relationship between the at least two video frames, behavioral features between the at least two video frames are extracted. Based on the visual features of the at least two video frames, visual features of the target video frame are extracted. The behavioral features and visual features are then fused to obtain hybrid features. Finally, the visual features and hybrid features are connected using a residual connection to obtain the sample video features.

[0136] Step 530: Determine the visual features of the samples corresponding to the category labels based on the features of the sample videos.

[0137] After extracting features from the sample video, we obtain the sample video features. Based on these features, we can obtain its label information. In this embodiment, the sample video features include sample visual features. Using the video features obtained above, the sample visual features can be determined according to the category label corresponding to the sample video.

[0138] Step 540: Use a semantic generation network to generate semantic features from the sample video features to obtain the sample semantic features corresponding to the sample video.

[0139] The semantic generation network is trained together with the entire network and can obtain corresponding semantic features based on the video features of the samples.

[0140] Step 550: Determine the semantic generation loss based on the sample semantic features and category labels.

[0141] The semantic features of the samples are obtained by processing the video features of the samples through a semantic generation network. Optionally, the information indicated by the category labels is obtained through a pre-trained network, such as the Word2Vec model. In this embodiment, the category labels are pseudo-labels annotated by the Word2Vec model.

[0142] At this point, taking into full account the existence of such semantic differences, the semantic generation loss of the sample video can be determined based on the semantic features of the sample video itself and the category label of the sample video.

[0143] To illustrate, determining the semantic generation loss based on sample semantic features and category labels involves the following operations:

[0144] In some embodiments, the category labels are vector-space mapped to obtain spatial mapping features.

[0145] The semantic generation loss is determined based on the feature distance between the sample semantic features and the spatial mapping features.

[0146] Indicatively, a semantic generation network is used to project a prototype network into the semantic generation space to generate semantic features:

[0147]

[0148] in This refers to the generated semantic features. g(·) is the semantic generation network, which is a non-linear neural network. This semantic generation network g(·) is guided by a semantic generation loss function, making the visual prototype more closely resemble the corresponding class semantic features, which can be represented as:

[0149]

[0150] Where f ψ (·) represents the Word2Vec model, l m This represents the tag information for the sample video.

[0151] Step 560: Train the semantic generation network using semantic generation loss.

[0152] The semantic generation loss can be determined based on the feature distance between sample semantic features and spatial mapping features. Specifically, the semantic generation loss is obtained by training with multiple sample videos. The semantic generation network is trained with the goal of reducing this loss, making the semantic information generated by the network as close as possible to the real semantic information, thereby improving the semantic generation accuracy of the network.

[0153] In summary, the video classification method provided in this embodiment describes the process of training the semantic generation network before inputting visual features. First, sample videos labeled with category tags are acquired. Then, features are extracted from the sample videos to obtain the corresponding sample video features. Next, based on the sample video features, the visual features corresponding to the category tags are determined. The semantic generation network then performs semantic generation on these sample video features to obtain the corresponding sample semantic features. A semantic generation loss is determined based on the sample semantic features and category tags, and the semantic generation network is trained using this loss. This process uses the semantic generation loss to improve the accuracy of the semantic generation network in generating semantics. A semantic generation network trained in this way can extract semantic features as accurately as possible when generating semantics from video features, which is beneficial for improving the accuracy of video classification when the semantic features and video features are fused together for subsequent video classification analysis.

[0154] This illustration demonstrates the video classification results in step 250 of Embodiment 2 of this application. Please refer to [link / reference]. Figure 6 To classify and predict the video category of the target video based on the fused features, the following steps are involved:

[0155] Step 610: Obtain sample videos, which are labeled with category tags.

[0156] The category label indicates the reference video category to which the sample video belongs. Classifying videos using label information helps to clearly define the category to which the sample video belongs.

[0157] Step 620: Extract the sample fusion features from the sample videos.

[0158] The sample fusion features include both semantic features and video features from the sample video. The generation method for sample fusion features is similar to the processing method for the target video. Sample fusion features are obtained by extracting semantic features and video features separately and then fusing them.

[0159] The semantic features in the sample video are generated by using the sample video features through a semantic generation network.

[0160] Among them, the semantic generation network is a semantic feature generation network obtained after training with semantic generation loss.

[0161] The extraction of video features from sample videos involves obtaining the video features of the sample video frames by performing spatial and temporal extraction through the aforementioned spatiotemporal difference module.

[0162] In a schematic manner, at least two video frames with a temporal relationship are extracted from the sample video. Based on the temporal relationship between the at least two video frames, behavioral features between the at least two video frames are extracted. Based on the visual features of the at least two video frames, visual features of the target video frame are extracted. The behavioral features and visual features are then fused to obtain hybrid features. Finally, the visual features and hybrid features are connected using a residual connection to obtain the sample video features.

[0163] Step 630: Construct a graph neural network based on intra-class consistency requirements and inter-class difference requirements.

[0164] In this graph neural network, the nodes are composed of sample fusion features and target fusion features.

[0165] Graph neural networks combine graph data with neural networks to perform end-to-end computations on graph data. The entire computation process follows the structure of the graph, allowing for the learning of structural information while preserving the graph's structural information.

[0166] The requirement for intra-class consistency and inter-class diversity is based on the fact that distinguishing between consistency and diversity can better define the range of video categories. Graph neural networks built upon this principle achieve higher accuracy and are better able to effectively classify videos; this operation can be summarized as relation propagation.

[0167] In this embodiment, relation propagation is used to update relation nodes to obtain discriminative fused features. The target fused feature and the sample fused feature are input into a graph neural network to obtain fused features enhanced by the target fused feature and fused features enhanced by the sample fused feature, and then the two are compared.

[0168] Schematic, we denote the relation node matrices as R and R'. m This refers to the m-th relation node. Given a video v m Its semantic features are The corresponding video prototype is The relation node R is obtained by adding the two elements together. m A graph convolutional network is used to update relation nodes to obtain discriminative relation features. The relation propagation rules are as follows:

[0169]

[0170] in It is a relational adjacency matrix with self-joins. The adjacency matrix A is constructed using the cosine similarity function. E is the identity matrix. Q and δ are the weight matrices. δ is the activation function. This allows us to obtain the enhanced features after relation propagation.

[0171] Step 640: Perform distance analysis between the target node and the other nodes in the graph neural network to obtain the feature distance between the target node and the sample node.

[0172] In this embodiment, the nodes of the graph neural network are fused features, obtained through training. Furthermore, the graph neural network classifies similarity based on node distance; therefore, it is necessary to compare the feature distances between the target node and the remaining nodes in the graph neural network, where the remaining nodes are sample fused features, referred to as sample nodes.

[0173] When performing distance comparison analysis between target nodes and sample nodes, the Euclidean distance function is used schematically. The smaller the Euclidean distance, the more similar the fused features are to the sample fused features in terms of content; conversely, the larger the distance, the greater the difference between them. Furthermore, using the cosine similarity distance function for distance comparison can yield better experimental results.

[0174] To illustrate, we compare the fusion features of the sample video with the fusion features of the target video. This process can be represented as follows:

[0175]

[0176] in This represents the similarity distance function. B q The target video is q. It analyzes the fusion features of videos, B n The fusion features of the sample videos, l a y is the category to which the sample video belongs, and y is the predicted category of the target video.

[0177] Step 650: Determine the node with the smallest feature distance to the target node in the graph neural network as the analysis node.

[0178] Graph neural networks (GNNs) determine similarity based on node distance; closer nodes indicate higher similarity, while wider nodes indicate lower similarity. The nodes in a GNN represent the fused features.

[0179] In this embodiment, a distance analysis is performed between the target node and the sample node. If the target node and the sample node are close, it indicates that the target video and the sample video are highly similar; conversely, if the target node and the sample node are far apart, it indicates that the target video and the sample video are not highly similar. Here, the target node is the fusion feature corresponding to the target video.

[0180] Step 660: The category corresponding to the analysis node is taken as the video category to which the target video belongs.

[0181] By comparing the target node with sample nodes, the closer the sample node is to the target node, the higher the similarity between the target video and the sample video corresponding to that sample node. Therefore, the sample node closest to the target node is selected, and the video category corresponding to this sample node is most similar to the category of the target video. Thus, the node closest to the target node can be used as the analysis node, and the category corresponding to this analysis node can be taken as the video category to which the target video belongs.

[0182] In summary, the video classification method provided in this embodiment explains how to predict the classification of a target video based on fused features. This process briefly describes the training model used in the data classification process. First, sample videos are obtained from already classified videos, and sample fusion features are extracted. Then, a graph neural network is constructed based on intra-class consistency and inter-class difference requirements. The nodes in this network include sample fusion features and target fusion features. The graph neural network has relation propagation capabilities, which can enhance the fused features to obtain more discriminative features. After training, the graph neural network is used to map the target video. Distance analysis is then performed between the target node and the sample node to obtain the difference between the target video and the video categories represented by the compared sample nodes. Finally, the node with the smallest distance between the target video and the graph neural network node is selected, and the category corresponding to that node is taken as the video category to which the target video belongs. After the above training method, a video classification model is obtained. After processing the target video, the video classification model can be used to compare and determine the video's category. This method takes into account that the video categories in the model trained with multiple sample videos are more comprehensive. If the target video is compared with this model, the video classification results will be more comprehensive and accurate.

[0183] Based on the above description of terms and application scenarios, the video classification method provided in this application will be explained. This method can be executed by a server or a terminal, or by both a server and a terminal. In this embodiment, the method is illustrated by being executed by a server. Figure 7 As shown, the process of this method is as follows:

[0184] First, M video categories are extracted from the video sample library 701, and K videos are extracted from each category to form the support set 702. Then, one video is extracted from the remaining videos in the M categories to form the query set 703. Next, each video is divided into 8 segments, and one frame is randomly extracted from each segment, resulting in a total of 8 video frames. These video frames are then input into the feature extraction network 704 to obtain video features. To improve the feature representation capability of the video features, a spatiotemporal difference module 705 is used to obtain improved video features. To obtain semantic information, a semantic generation network 706 is used to generate semantic information. To make the generated semantic information closer to real semantic information, a semantic generation loss function 707 is introduced to supervise the learning of the semantic generation network 706. Then, visual information and corresponding semantic information constitute visual-semantic pairs. Utilizing intra-class consistency and inter-class differences, a graph convolutional network 708 is used to learn the relationship between visual-semantic pairs. Finally, a small sample classifier 709 is used to estimate the similarity between videos in the support set and videos in the query set, and the class with the highest estimated score is taken as the video class 710 of the query set.

[0185] in, Figure 7 The system uses a variety of symbols, and the meanings of the symbols are shown in Symbol Meaning 711.

[0186] In summary, the video classification method provided in this embodiment extracts features from the video frames of the target video to obtain the corresponding video features. Then, semantic features obtained based on the video features are fused with the video features to obtain fused features. Finally, the video is classified according to its category based on these fused features to obtain the video classification result. This process not only fully considers the classification information contained in the video features but also extracts the corresponding semantic features from the video features and incorporates them into the analysis. This expands the scope of video feature information acquisition, avoids incomplete information inclusion during video classification, and improves the accuracy of video classification.

[0187] The classification process described above is merely a simple example of a video classification method, and the embodiments of this application do not limit it.

[0188] This illustration demonstrates the video classification method provided in this application, such as... Figure 8 The figure shows a comparison of the accuracy of our video classification method with other commonly used existing video classification methods, with the experiments conducted on publicly available datasets.

[0189] Indicatively, the selected video classification methods 810 include existing video classification methods BaseNet, CMN, TARN, Embodied Learning, ARN, AMeFu-Net, and the present video classification method SRPN, and the selected public dataset types 820 include MiniKinetics, UCF101, and HMDB51.

[0190] Among them, BaseNet is a backbone feature extraction network, a mature model that can directly extract video features from videos; CMN is an abbreviation for Compound Memory Networks, which uses the accumulation properties of memory structures to allow the network to retain the characteristics of sample videos, thereby helping to solve video classification problems; TARN is an abbreviation for Temporal Attention Relation Network, which is a meta-learning method used to solve action recognition problems with few samples and zero samples; Embodied Learning, which constructs an UnrealAction dataset and proposes a data augmentation method for video segment replacement for action recognition with few samples, which is then used for video classification; ARN is an abbreviation for Action Recognition Net, which is based on the encoder capturing short-range action patterns, aggregating the encoded blocks through permutation invariant pooling, and studying the relationship between the method and the dependence of changing action lengths and long-distance events for video classification; AMeFu-Net is an abbreviation for Adaptive Meta-Fusion Network, which first introduces depth information as additional visual information and proposes a temporal asynchronous augmentation mechanism to enhance the original video representation. In this application example, the video classification method specifically adopts SPRN, which stands for Semantic Guided Relationship Propagation Network, a small sample behavior recognition method. This is just one example of a video classification method and is not intended to limit the scope of the application.

[0191] Figure 8 This presents experimental results demonstrating the video classification accuracy of different video classification methods on various datasets. "Shot" represents the number of video samples used for training; for example, "1-shot" means only one video sample was used for training. The following sections... Figure 8 Analysis:

[0192] When comparing data longitudinally, it can be observed that when using the same video classification method, the accuracy increases with the number of training samples. The UCF101 dataset shows the best performance in video classification.

[0193] When comparing different video classification methods, it can be found that SRPN achieves the best video classification results. In UCF101, when the number of training sample videos is large, the accuracy can even approach the features of the video itself indefinitely.

[0194] The above results fully demonstrate that the SRPN method proposed in this application classifies videos with better accuracy than existing video classification methods, and can achieve a higher accuracy rate.

[0195] It is worth noting that in the above embodiments, the method described in this embodiment is used as an example to compare with other listed limited methods in a limited dataset. This classification method can also be compared and analyzed with other video classification methods in other datasets, and this application embodiment does not limit this.

[0196] Figure 9 This is a structural block diagram of video classification provided in an exemplary embodiment of this application, such as... Figure 9 As shown, a video classification device includes:

[0197] The acquisition module 910 is used to acquire the target video, which is the video to be classified and predicted.

[0198] The extraction module 920 is used to extract features from the target video based on the video frames in the target video to obtain the video features corresponding to the target video. The video features are used to indicate the picture information expressed by the video frames of the target video.

[0199] The analysis module 930 is used to perform semantic analysis on the target video based on video features to obtain the semantic features corresponding to the target video. The semantic features are used to indicate the entity association relationships expressed by the video frames of the target video.

[0200] The fusion module 940 is used to fuse video features and semantic features to obtain the fused features corresponding to the target video.

[0201] The prediction module 950 is used to classify and predict the fused features to obtain the video category to which the target video belongs.

[0202] In an optional embodiment, such as Figure 10 As shown, the analysis module 930 includes:

[0203] Input unit 931 is used to input the video features into the semantic generation network, wherein the semantic generation network is a trained semantic feature extraction network;

[0204] Analysis unit 932 is used to perform semantic analysis on the video features through the semantic generation network and output the semantic features corresponding to the target video.

[0205] In an optional embodiment, the analysis module 930 further includes:

[0206] The acquisition unit 933 is used to perform prototype acquisition on the video features to obtain the prototype features corresponding to the video features, wherein the prototype features are the feature mapping results of the video features in the prototype space.

[0207] The analysis unit 932 is also used to perform semantic analysis on the prototype features through the semantic generation network and output the semantic features corresponding to the target video.

[0208] In an optional embodiment, the extraction module 920 is further configured to extract at least two video frames from the target video, wherein there is a temporal relationship between the at least two video frames; and to extract video features of the target video frames based on the temporal relationship and the scene between the at least two video frames.

[0209] In an optional embodiment, the extraction module 920 is further configured to extract behavioral features between at least two video frames based on the temporal relationship between the at least two video frames; and to extract visual features of the target video frame based on the images of the at least two video frames.

[0210] The fusion module 940 is further configured to fuse the behavioral features and the visual features to obtain the video features of the target video frame.

[0211] In an optional embodiment, the fusion module 940 is further configured to fuse the behavioral features and the visual features to obtain a hybrid feature; and to connect the visual features and the hybrid feature through a residual connection to obtain the video features.

[0212] In an optional embodiment, the extraction module 920 is further configured to segment the target video to obtain at least two video segments; and to sample video frames from the at least two video segments to obtain the at least two video frames.

[0213] In an optional embodiment, the prediction module 950 is further configured to perform distance analysis between the fused feature and the nodes in the graph neural network to obtain the feature distance between the fused feature and the node; determine the target node in the graph neural network with the smallest feature distance to the fused feature; and use the classification corresponding to the target node as the video classification to which the target video belongs.

[0214] In an optional embodiment, the acquisition module 910 is further configured to acquire a sample video, the sample video being labeled with a category tag, the category tag being used to indicate the reference video category to which the sample video belongs;

[0215] The extraction module 920 is also used to extract features from the sample video to obtain the sample video features corresponding to the sample video;

[0216] The analysis module 930 is also used to determine the visual features of the sample corresponding to the category label based on the sample video features;

[0217] Extraction module 920 is also used to extract semantics from the visual features of the sample through the semantic generation network to obtain the sample semantic features corresponding to the sample video;

[0218] The device further includes:

[0219] The determination module 960 is used to determine the semantic extraction loss based on the sample semantic features and the category label;

[0220] Training module 970 is used to train the semantic generation network with the semantic extraction loss.

[0221] In an optional embodiment, the determining module 960 is further configured to perform vector space mapping on the category label to obtain spatial mapping features; and determine the semantic extraction loss based on the feature distance between the sample semantic features and the spatial mapping features.

[0222] In an optional embodiment, the acquisition module 910 is further configured to acquire a sample video, the sample video being labeled with a category tag, the category tag being used to indicate the reference video category to which the sample video belongs;

[0223] The extraction module 920 is also used to extract the sample fusion features of the sample video, wherein the sample fusion features include the sample semantic features and sample video features of the sample video;

[0224] The device further includes:

[0225] The construction module 980 is used to construct the graph neural network based on intra-class consistency requirements and inter-class differences requirements, wherein the nodes in the graph neural network are composed of the sample fusion features.

[0226] In summary, the video classification device provided in this embodiment first acquires the target video through an acquisition device, then extracts features from the target video through an extraction module to obtain the corresponding video features. Next, an analysis module performs semantic analysis on the target video to obtain the corresponding semantic features. Then, a fusion module fuses the video features and semantic features to obtain the fused features corresponding to the target video. Finally, a prediction module performs classification prediction on the fused features to obtain the video category to which the target video belongs. In this process, fusing video features and semantic features expands the scope of video feature information acquisition, avoids incomplete information inclusion during video classification, and improves the accuracy of video classification.

[0227] It should be noted that the video classification device provided in the above embodiments is only an example of the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the video classification device and the video classification method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0228] Figure 11 This illustration shows a schematic diagram of a server provided in an exemplary embodiment of this application. The server 1000 includes a Central Processing Unit (CPU) 1001, a system memory 1004 including Random Access Memory (RAM) 1002 and Read Only Memory (ROM) 1003, and a system bus 1005 connecting the system memory 1004 and the CPU 1001. The server 1000 also includes a mass storage device 1006 for storing an operating system 1013, application programs 1014, and other program modules 1015.

[0229] Mass storage device 1006 is connected to central processing unit 1001 via a mass storage controller (not shown) connected to system bus 1005. Mass storage device 1006 and its associated computer-readable media provide non-volatile storage for server 1000. That is, mass storage device 1006 may include computer-readable media (not shown) such as hard disk or compact disc read-only memory (CD-ROM) drive.

[0230] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The system memory 1004 and mass storage device 1006 described above can be collectively referred to as memory.

[0231] According to various embodiments of this application, server 1000 can also be connected to a remote computer on a network, such as the Internet. That is, server 1000 can be connected to network 1012 via network interface unit 1011 connected to system bus 1005, or network interface unit 911 can be used to connect to other types of networks or remote computer systems (not shown).

[0232] The aforementioned memory also includes one or more programs, which are stored in the memory and configured to be executed by the CPU.

[0233] Embodiments of this application also provide a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the video classification method provided in the above-described method embodiments.

[0234] Embodiments of this application also provide a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the video classification method provided in the above-described method embodiments.

[0235] Embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the video classification methods described in the above embodiments.

[0236] Optionally, the computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. The random access memory may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM). The sequence numbers of the embodiments in this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0237] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0238] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A video classification method, characterized in that, The method includes: Acquire the target video, which is the video to be classified and predicted; At least two video frames are extracted from the target video, and there is a temporal relationship between the at least two video frames; based on the temporal relationship and the picture, video features of the target video frames are extracted, and the video features are used to indicate the picture information expressed by the video frames of the target video; The video features are input into a semantic generation network, which is a semantic feature extraction network trained based on semantic extraction loss. The semantic extraction loss is the loss between the sample semantic features and the category label, and the category label is the category label of the sample video corresponding to the sample semantic features. Prototype acquisition is performed on the video features to obtain the prototype features corresponding to the video features. The prototype features are the feature mapping results of the video features in the prototype space. In the prototype space, samples of the same type are closer together and samples of different types are farther apart. The semantic generation network performs semantic analysis on the prototype features to output semantic features corresponding to the target video. The semantic features are used to indicate the entity association relationships expressed by the video frames of the target video. The video features and the semantic features are fused to obtain the fused features corresponding to the target video; In this process, the video features and semantic features are each mapped to corresponding vectors on the prototype network. The fusion is achieved by adding the elements of the vectors corresponding to the video features and the semantic features. The fused features are represented as target nodes in the prototype network. The fused features are then used for classification prediction to obtain the video category to which the target video belongs. The training of the semantic generation network is performed synchronously with the training of the video feature extraction network; the semantic generation network is trained in the following manner: A sample video is acquired, and the sample video is labeled with a category label, which indicates the reference video category to which the sample video belongs; feature extraction is performed on the sample video to obtain the sample video features corresponding to the sample video; based on the sample video features, the sample visual features corresponding to the category label are determined; semantic extraction is performed on the sample video features through the semantic generation network to obtain the sample semantic features corresponding to the sample video; vector space mapping is performed on the category label to obtain spatial mapping features; based on the feature distance between the sample semantic features and the spatial mapping features, the semantic extraction loss is determined; and the semantic generation network is trained with the semantic extraction loss.

2. The method according to claim 1, characterized in that, The step of extracting video features of the target video frame based on the temporal relationship and scene composition between the at least two video frames includes: Based on the temporal relationship between the at least two video frames, extract the behavioral features between the at least two video frames; Based on the images of the at least two video frames, extract the visual features of the target video frame; The behavioral features and the visual features are fused to obtain the video features of the target video frame.

3. The method according to claim 2, characterized in that, The process of fusing the behavioral features and the visual features to obtain the video features of the target video frame includes: The behavioral features and the visual features are fused to obtain hybrid features; The visual features and the hybrid features are connected using a residual connection method to obtain the video features.

4. The method according to claim 1, characterized in that, Extracting at least two video frames from the target video includes: The target video is segmented to obtain at least two video segments; Video frames are sampled from the at least two video segments to obtain the at least two video frames.

5. The method according to claim 1, characterized in that, The step of classifying and predicting the fused features to obtain the video category to which the target video belongs includes: The feature distance between the fused feature and the sample node in the graph neural network is obtained by performing distance analysis on the fused feature and the sample node. Determine the analysis node in the graph neural network that has the smallest feature distance to the fused feature; The category corresponding to the analysis node is taken as the video category to which the target video belongs.

6. The method according to claim 5, characterized in that, Before performing distance analysis between the fused features and sample nodes in the graph neural network, the method further includes: Obtain sample videos, each of which is labeled with a category tag. The category tag is used to indicate the reference video category to which the sample video belongs. Extract the sample fusion features of the sample video, wherein the sample fusion features include the sample semantic features and sample video features of the sample video; The graph neural network is constructed based on the requirements of intra-class consistency and inter-class differences. The nodes in the graph neural network are composed of sample nodes and target nodes. The sample nodes are the sample fusion features, and the target nodes are the fusion features.

7. A video classification device, characterized in that, The device includes: The acquisition module is used to acquire the target video, which is the video to be classified and predicted; An extraction module is configured to extract at least two video frames from the target video, wherein there is a temporal relationship between the at least two video frames; and based on the temporal relationship and the scene between the at least two video frames, extract video features of the target video frames, wherein the video features are used to indicate the scene information expressed by the video frames of the target video. The input module is used to input the video features into a semantic generation network, which is a semantic feature extraction network trained based on semantic extraction loss. The semantic extraction loss is the loss between the sample semantic features and the category label, where the category label is the category label of the sample video corresponding to the sample semantic features. The analysis module is used to perform prototype acquisition on the video features to obtain the prototype features corresponding to the video features. The prototype features are the feature mapping results of the video features in the prototype space. In the prototype space, samples of the same type are closer together, and samples of different types are farther apart. The semantic generation network performs semantic analysis on the prototype features to output semantic features corresponding to the target video. The semantic features are used to indicate the entity association relationships expressed by the video frames of the target video. The fusion module is used to fuse the video features and the semantic features to obtain the fused features corresponding to the target video; In this process, the video features and semantic features are each mapped to corresponding vectors on the prototype network. The fusion is an operation of adding the elements of the vectors corresponding to the video features and the semantic features. The fused features are represented as target nodes in the prototype network. The prediction module is used to perform classification prediction on the fused features to obtain the video category to which the target video belongs; The training of the semantic generation network is performed synchronously with the training of the video feature extraction network; the semantic generation network is trained in the following manner: The acquisition module is further configured to acquire sample videos, each sample video being labeled with a category tag, the category tag indicating the reference video category to which the sample video belongs; the extraction module is further configured to extract features from the sample videos to obtain sample video features corresponding to the sample videos; determine sample visual features corresponding to the category tags based on the sample video features; perform semantic extraction on the sample video features through the semantic generation network to obtain sample semantic features corresponding to the sample videos; the analysis module is further configured to perform vector space mapping on the category tags to obtain spatial mapping features; determine the semantic extraction loss based on the feature distance between the sample semantic features and the spatial mapping features; and train the semantic generation network using the semantic extraction loss.

8. The apparatus according to claim 7, characterized in that, The extraction module is further configured to extract behavioral features between at least two video frames based on the temporal relationship between the at least two video frames; and to extract visual features of the target video frame based on the images of the at least two video frames. The fusion module is further configured to fuse the behavioral features and the visual features to obtain the video features of the target video frame.

9. The apparatus according to claim 8, characterized in that, The fusion module is further configured to fuse the behavioral features and the visual features to obtain hybrid features; and to connect the visual features and the hybrid features through a residual connection to obtain the video features.

10. The apparatus according to claim 7, characterized in that, The extraction module is further configured to segment the target video to obtain at least two video segments; and to sample video frames from the at least two video segments to obtain the at least two video frames.

11. The apparatus according to claim 7, characterized in that, The prediction module is further configured to perform distance analysis between the fused feature and the nodes in the graph neural network to obtain the feature distance between the fused feature and the node; determine the target node in the graph neural network with the smallest feature distance to the fused feature; and use the classification corresponding to the target node as the video classification to which the target video belongs.

12. The apparatus according to claim 11, characterized in that, The acquisition module is also used to acquire sample videos, which are labeled with category tags, and the category tags are used to indicate the reference video category to which the sample video belongs; The extraction module is further configured to extract sample fusion features of the sample video, wherein the sample fusion features include sample semantic features and sample video features of the sample video; The device further includes: A construction module is used to construct the graph neural network based on intra-class consistency requirements and inter-class differences requirements, wherein the nodes in the graph neural network are composed of the sample fusion features.

13. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one instruction, at least one program, a code set, or an instruction set, the at least one instruction, the at least one program, the code set, or the instruction set being loaded and executed by the processor to implement the video classification method as described in any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or instruction set is loaded and executed by a processor to implement the video classification method as described in any one of claims 1 to 6.

Images