Video classification method and device, computer device and storage medium

Abstract

The application relates to a video classification method and device, computer equipment and a storage medium, comprising the following steps: performing three-dimensional feature extraction on a plurality of video segments corresponding to a video to be classified, to obtain initial classification feature information corresponding to each video segment to be classified; assigning corresponding primary attention information to the initial classification feature information corresponding to each video segment to be classified; assigning corresponding secondary attention information to the initial classification feature information corresponding to each video segment to be classified based on the classification influence degree of each video segment to be classified, wherein the secondary attention information is used to improve the attention degree of the video segment to be classified with a non-key classification influence degree; fusing the initial classification feature information corresponding to each video segment to be classified based on the primary attention information and the secondary attention information, to obtain target classification video features; and determining a video classification result based on the target classification video features. The method can improve the accuracy of video classification.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video classification method, apparatus, computer equipment, and storage medium. Background Art

[0002] With the development and application of computer technology, various content platforms have emerged, allowing users to publish and browse a variety of short video content. Content platforms categorize videos for easier video management and user search.

[0003] In traditional techniques, video classification is mainly based on key frames in the video to determine the classification results. However, if the key frames in the video are disturbed, accurate video classification results cannot be obtained. Summary of the Invention

[0004] Based on this, it is necessary to provide a video classification method, apparatus, computer equipment and storage medium to address the above technical problems, which can improve the accuracy of video classification of videos with interfered key frames.

[0005] A video classification method, comprising:

[0006] Obtain multiple video clips to be classified corresponding to the video to be classified;

[0007] Perform three-dimensional feature extraction on each video clip to be classified, obtain the motion features of each video clip to be classified in the time dimension and the image features in the spatial dimension, and form the corresponding initial feature information to be classified;

[0008] Assigning corresponding primary attention information to the initial feature information to be classified corresponding to each video segment to be classified, and the primary attention information is used to represent the attention intensity corresponding to the video segment to be classified;

[0009] Based on the classification influence corresponding to each video segment to be classified, corresponding second-order attention information is allocated to the initial feature information to be classified corresponding to each video segment to be classified, and the second-order attention information is used to improve the attention intensity corresponding to the video segment to be classified with non-critical classification influence;

[0010] Based on the primary attention information and the second-order attention information, the initial feature information to be classified corresponding to each video clip to be classified is fused to obtain the target video feature to be classified;

[0011] Determine the video classification result corresponding to the target video to be classified based on the target video features.

[0012] In one embodiment, obtaining multiple video segments to be classified corresponding to the video to be classified includes:

[0013] Filter the head video frame and the tail video frame from the video to be classified to obtain a candidate video;

[0014] Extracting equally spaced video frames from each candidate video frame in the candidate video to obtain multiple intermediate video frames;

[0015] According to the time information corresponding to the video frames, the head video frame, the tail video frame and each intermediate video frame are sorted to obtain the target video frame sequence;

[0016] The target video frame sequence is segmented to obtain multiple video segments to be classified.

[0017] In one embodiment, three-dimensional feature extraction is performed on each video segment to be classified, and motion features in the time dimension and image features in the spatial dimension of each video segment to be classified are obtained to form corresponding initial feature information to be classified, including:

[0018] Based on each video segment to be classified, a corresponding three-dimensional video image block is formed, where the three dimensions of the three-dimensional video image block are respectively the length and width of the video frame and the number of video frames of the current video segment to be classified;

[0019] Through a three-dimensional convolutional network composed of a three-dimensional convolutional layer and a three-dimensional pooling layer, the motion information between temporally adjacent pixels in the three-dimensional video image block is extracted to obtain motion features, and the image information between spatially adjacent pixels in the three-dimensional video image block is extracted to obtain image features to form the corresponding initial feature information to be classified.

[0020] In one embodiment, the video classification result includes a motion tag. Determining the video classification result corresponding to the target video to be classified based on the features of the target video to be classified includes:

[0021] Performing object recognition on the target video to be classified based on the image features in the target video to be classified, and obtaining multiple target objects in the target video to be classified;

[0022] Perform global trajectory recognition on the target object based on the motion features in the target video features to be classified, and obtain the global motion trajectory corresponding to each target object;

[0023] Determine interaction information between target objects based on the global motion trajectories corresponding to the target objects;

[0024] The motion labels corresponding to the video to be classified are obtained based on the interaction information between each target object and the object type of the target object.

[0025] In one embodiment, determining interaction information between target objects based on the global motion trajectories corresponding to the target objects includes:

[0026] Based on the intersecting global motion trajectories and the motion directions of the global motion trajectories before and after the intersection, the interaction information between the corresponding target objects is generated.

[0027] A video classification device, comprising:

[0028] A video acquisition module is used to acquire multiple video clips to be classified corresponding to the video to be classified;

[0029] A feature extraction module is used to extract three-dimensional features from each video segment to be classified, obtain the motion features of each video segment to be classified in the time dimension and the image features in the spatial dimension, and form the corresponding initial feature information to be classified;

[0030] A primary attention allocation module is used to allocate corresponding primary attention information to the initial feature information to be classified corresponding to each video segment to be classified. The primary attention information is used to represent the attention intensity corresponding to the video segment to be classified;

[0031] A second-order attention allocation module is used to allocate corresponding second-order attention information to the initial feature information to be classified corresponding to each video segment to be classified based on the classification influence corresponding to each video segment to be classified. The second-order attention information is used to increase the attention intensity corresponding to the video segment to be classified with non-critical classification influence;

[0032] An information fusion module is used to fuse the initial feature information to be classified corresponding to each video segment to be classified based on the primary attention information and the second-order attention information to obtain the target video feature to be classified;

[0033] The video classification module is used to determine the video classification result corresponding to the video to be classified based on the target video features to be classified.

[0034] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0035] Obtain multiple video clips to be classified corresponding to the video to be classified;

[0036] Perform three-dimensional feature extraction on each video clip to be classified, obtain the motion features of each video clip to be classified in the time dimension and the image features in the spatial dimension, and form the corresponding initial feature information to be classified;

[0037] Assigning corresponding primary attention information to the initial feature information to be classified corresponding to each video segment to be classified, and the primary attention information is used to represent the attention intensity corresponding to the video segment to be classified;

[0038] Based on the classification influence corresponding to each video segment to be classified, corresponding second-order attention information is allocated to the initial feature information to be classified corresponding to each video segment to be classified, and the second-order attention information is used to improve the attention intensity corresponding to the video segment to be classified with non-critical classification influence;

[0039] Based on the primary attention information and the second-order attention information, the initial feature information to be classified corresponding to each video clip to be classified is fused to obtain the target video feature to be classified;

[0040] Determine the video classification result corresponding to the target video to be classified based on the target video features.

[0041] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0042] Obtain multiple video clips to be classified corresponding to the video to be classified;

[0043] Perform three-dimensional feature extraction on each video clip to be classified, obtain the motion features of each video clip to be classified in the time dimension and the image features in the spatial dimension, and form the corresponding initial feature information to be classified;

[0044] Assigning corresponding primary attention information to the initial feature information to be classified corresponding to each video segment to be classified, and the primary attention information is used to represent the attention intensity corresponding to the video segment to be classified;

[0045] Based on the classification influence corresponding to each video segment to be classified, corresponding second-order attention information is allocated to the initial feature information to be classified corresponding to each video segment to be classified, and the second-order attention information is used to improve the attention intensity corresponding to the video segment to be classified with non-critical classification influence;

[0046] Based on the primary attention information and the second-order attention information, the initial feature information to be classified corresponding to each video clip to be classified is fused to obtain the target video feature to be classified;

[0047] Determine the video classification result corresponding to the target video to be classified based on the target video features.

[0048] The above-mentioned video classification method, apparatus, computer device, and storage medium obtain multiple video segments to be classified corresponding to the video to be classified, perform three-dimensional feature extraction on each video segment to be classified, obtain motion features in the time dimension and image features in the spatial dimension of each video segment to be classified, and form corresponding initial feature information to be classified. The initial feature information to be classified corresponding to each video segment to be classified is assigned corresponding primary attention information. The primary attention information is used to characterize the attention intensity corresponding to the video segment to be classified. Based on the classification influence corresponding to each video segment to be classified, the initial feature information to be classified corresponding to each video segment to be classified is assigned corresponding second-order attention information. The second-order attention information is used to improve the attention intensity corresponding to the video segment to be classified with non-critical classification influence. Based on the primary attention information and the second-order attention information, the initial feature information to be classified corresponding to each video segment to be classified is fused to obtain the target video feature to be classified. The video classification result corresponding to the video to be classified is determined based on the target video feature to be classified. In this way, the second-order attention information can improve the attention intensity corresponding to the video segment to be classified with non-critical classification influence, that is, improve the attention intensity corresponding to the non-critical video segment. The feature information of the non-critical video segment can predict the feature information of the key frame video segment to a certain extent. Therefore, when classifying videos with key frames interfered with, the content of the video can also be identified based on the target video features to be classified that are fused with the feature information of non-key video clips. With the help of non-key video clips, the video classification accuracy of videos with key frames interfered with can be effectively improved.

[0049] A video classification method, comprising:

[0050] Obtain multiple training video clips and training labels corresponding to the training videos, and input each training video clip into the target video classification model to be trained;

[0051] Perform three-dimensional feature extraction on each training video clip to obtain the motion features of each training video clip in the time dimension and the image features in the spatial dimension, which constitute the corresponding initial training feature information;

[0052] Assigning corresponding primary attention information to the initial training feature information corresponding to each training video segment, where the primary attention information is used to characterize the attention intensity corresponding to the training video segment;

[0053] Based on the classification influence corresponding to each training video segment, corresponding second-order attention information is allocated to the initial training feature information corresponding to each training video segment, and the second-order attention information is used to improve the attention intensity corresponding to the training video segment with non-critical classification influence;

[0054] Based on the primary attention information and the second-order attention information, the initial training feature information corresponding to each training video clip is fused to obtain the target training video feature;

[0055] The target prediction label corresponding to the training video is determined based on the target training video features, and the model parameters of the target video classification model are adjusted based on the difference between the training label and the target prediction label until the convergence condition is met, thereby obtaining the trained target video classification model.

[0056] In one embodiment, three-dimensional feature extraction is performed on each training video segment to obtain motion features in the time dimension and image features in the spatial dimension of each training video segment, which form corresponding initial training feature information, including:

[0057] Through the three-dimensional convolutional network in the target video classification model, three-dimensional features are extracted from each training video clip to obtain the initial training feature information corresponding to each training video clip; the network parameters of the three-dimensional convolutional network are obtained by pre-training based on multiple training video clips corresponding to the training videos and the training labels corresponding to the training videos.

[0058] In one embodiment, pre-training of a 3D convolutional network includes the following steps:

[0059] Inputting each training video clip into an initial video classification model to be trained, the initial video classification model including a 3D convolutional network to be trained;

[0060] Perform three-dimensional feature extraction on each training video clip to obtain intermediate training feature information corresponding to each training video clip;

[0061] Based on each intermediate training feature information, the initial prediction label corresponding to the training video is obtained;

[0062] Based on the label difference between the initial predicted label and the training label, the model parameters of the initial video classification model are adjusted until the convergence condition is met, thereby obtaining a trained initial video classification model; the trained initial video classification model includes a trained three-dimensional convolutional network.

[0063] A video classification device, comprising:

[0064] A video acquisition module is used to obtain multiple training video clips and training labels corresponding to the training video, and input each training video clip into the target video classification model to be trained;

[0065] A feature extraction module is used to extract three-dimensional features from each training video segment, obtain the motion features of each training video segment in the time dimension and the image features in the spatial dimension, and form the corresponding initial training feature information;

[0066] A primary attention allocation module is used to allocate corresponding primary attention information to the initial training feature information corresponding to each training video segment, and the primary attention information is used to represent the attention intensity corresponding to the training video segment;

[0067] A second-order attention allocation module is used to allocate corresponding second-order attention information to the initial training feature information corresponding to each training video segment based on the classification influence corresponding to each training video segment. The second-order attention information is used to increase the attention intensity corresponding to the training video segment with non-critical classification influence;

[0068] An information fusion module is used to fuse the initial training feature information corresponding to each training video clip based on the primary attention information and the second-order attention information to obtain the target training video features;

[0069] The parameter adjustment module is used to determine the target prediction label corresponding to the training video based on the target training video features, and adjust the model parameters of the target video classification model based on the difference between the training label and the target prediction label until the convergence condition is met, thereby obtaining the trained target video classification model.

[0070] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0071] Obtain multiple training video clips and training labels corresponding to the training videos, and input each training video clip into the target video classification model to be trained;

[0072] Perform three-dimensional feature extraction on each training video clip to obtain the motion features of each training video clip in the time dimension and the image features in the spatial dimension, which constitute the corresponding initial training feature information;

[0073] Assigning corresponding primary attention information to the initial training feature information corresponding to each training video segment, where the primary attention information is used to characterize the attention intensity corresponding to the training video segment;

[0074] Based on the classification influence corresponding to each training video segment, corresponding second-order attention information is allocated to the initial training feature information corresponding to each training video segment, and the second-order attention information is used to improve the attention intensity corresponding to the training video segment with non-critical classification influence;

[0075] Based on the primary attention information and the second-order attention information, the initial training feature information corresponding to each training video clip is fused to obtain the target training video feature;

[0076] The target prediction label corresponding to the training video is determined based on the target training video features, and the model parameters of the target video classification model are adjusted based on the difference between the training label and the target prediction label until the convergence condition is met, thereby obtaining the trained target video classification model.

[0077] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0078] Obtain multiple training video clips and training labels corresponding to the training videos, and input each training video clip into the target video classification model to be trained;

[0079] Perform three-dimensional feature extraction on each training video clip to obtain the motion features of each training video clip in the time dimension and the image features in the spatial dimension, which constitute the corresponding initial training feature information;

[0080] Assigning corresponding primary attention information to the initial training feature information corresponding to each training video segment, where the primary attention information is used to characterize the attention intensity corresponding to the training video segment;

[0081] Based on the classification influence corresponding to each training video segment, corresponding second-order attention information is allocated to the initial training feature information corresponding to each training video segment, and the second-order attention information is used to improve the attention intensity corresponding to the training video segment with non-critical classification influence;

[0082] Based on the primary attention information and the second-order attention information, the initial training feature information corresponding to each training video clip is fused to obtain the target training video feature;

[0083] The target prediction label corresponding to the training video is determined based on the target training video features, and the model parameters of the target video classification model are adjusted based on the difference between the training label and the target prediction label until the convergence condition is met, thereby obtaining the trained target video classification model.

[0084] The above-mentioned video classification method, device, computer equipment and storage medium obtain multiple training video clips corresponding to the training video and training labels corresponding to the training video, input each training video clip into the target video classification model to be trained, perform three-dimensional feature extraction on each training video clip, obtain the motion features of each training video clip in the time dimension and the image features of each training video clip in the spatial dimension, and form the corresponding initial training feature information, and assign corresponding primary attention information to the initial training feature information corresponding to each training video clip. The primary attention information is used to characterize the attention intensity corresponding to the training video clip, and based on the classification influence corresponding to each training video clip, the initial training feature information corresponding to each training video clip is assigned corresponding second-order attention information. The second-order attention information is used to improve the attention intensity corresponding to the training video clip with non-critical classification influence. Based on the primary attention information and the second-order attention information, the initial training feature information corresponding to each training video clip is fused to obtain the target training video feature, and the target prediction label corresponding to the training video is determined based on the target training video feature. The model parameters of the target video classification model are adjusted based on the difference between the training label and the target prediction label until the convergence condition is met, thereby obtaining the trained target video classification model. In this way, second-order attention information can improve the attention level corresponding to the video segments to be classified that are not critical to the classification. This means that the feature information of non-critical video segments can, to a certain extent, predict the feature information of keyframe video segments. Therefore, applying second-order attention information to model training can produce a target video classification model that can accurately classify both regular videos and videos with disturbed keyframes. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 A diagram illustrating an application environment of a video classification method according to an embodiment;

[0086] Figure 2 1 is a flow chart of a video classification method according to an embodiment;

[0087] Figure 3 is a schematic diagram of a video tag in one embodiment;

[0088] Figure 4A Schematic diagram of a three-dimensional convolutional network in one embodiment;

[0089] Figure 4B A schematic diagram of data processing performed by a convolution block in one embodiment;

[0090] Figure 5A is a flow chart of a video classification method according to another embodiment;

[0091] Figure 5Bis a schematic diagram of a multilayer perceptron in one embodiment;

[0092] Figure 5C A schematic diagram of data processing performed by neurons in one embodiment;

[0093] Figure 6 is a schematic diagram of various attention networks in one embodiment;

[0094] Figure 7 1 is a flow chart of a video classification method according to another embodiment;

[0095] Figure 8 is a schematic diagram of an initial video classification model in one embodiment;

[0096] Figure 9 is a flow chart of a video classification method according to another embodiment;

[0097] Figure 10 is a structural block diagram of a video classification device in one embodiment;

[0098] Figure 11 is a structural block diagram of a video classification device in another embodiment;

[0099] Figure 12 is a structural block diagram of a video classification device in yet another embodiment;

[0100] Figure 13 is a diagram of the internal structure of a computer device in one embodiment;

[0101] Figure 14 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0102] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0103] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0104] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0105] Computer vision (CV) is the science of making machines "see." Specifically, it refers to using cameras and computers to replace the human eye in identifying and measuring objects, and then further processing them to produce images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0106] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0107] The solutions provided in the embodiments of this application involve artificial intelligence computer vision technology, machine learning and other technologies, which are specifically illustrated by the following embodiments:

[0108] The video classification method provided in this application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 via a network. The terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server 104 can be implemented as an independent server or a server cluster consisting of multiple servers or a cloud server.

[0109] Both the terminal 102 and the server 104 can be used independently to execute the video classification method provided in the embodiments of the present application.

[0110] For example, the server can obtain multiple video clips to be classified corresponding to the video to be classified, perform three-dimensional feature extraction on each video clip to be classified, obtain the motion features of each video clip to be classified in the time dimension and the image features in the space dimension, and form the corresponding initial feature information to be classified. The server can assign corresponding primary attention information to the initial feature information to be classified corresponding to each video clip to be classified. The primary attention information is used to characterize the attention intensity corresponding to the video clip to be classified. Based on the classification influence corresponding to each video clip to be classified, the server assigns corresponding second-order attention information to the initial feature information to be classified corresponding to each video clip to be classified. The second-order attention information is used to improve the attention intensity corresponding to the video clip to be classified with non-critical classification influence. The server can fuse the initial feature information to be classified corresponding to each video clip to be classified based on the primary attention information and the second-order attention information to obtain the target video feature to be classified, and determine the video classification result corresponding to the video to be classified based on the target video feature to be classified.

[0111] The terminal 102 and the server 104 may also be used in conjunction to execute the video classification method provided in the embodiment of the present application.

[0112] For example, the server can obtain multiple video clips to be classified corresponding to the video to be classified from the terminal. The server performs three-dimensional feature extraction on each video clip to be classified, obtains the motion features of each video clip to be classified in the time dimension and the image features of each video clip to be classified in the space dimension, and constitutes the corresponding initial feature information to be classified. The server can assign corresponding primary attention information to the initial feature information to be classified corresponding to each video clip to be classified. The primary attention information is used to characterize the attention intensity corresponding to the video clip to be classified. Based on the classification influence corresponding to each video clip to be classified, the server assigns corresponding second-order attention information to the initial feature information to be classified corresponding to each video clip to be classified. The second-order attention information is used to improve the attention intensity corresponding to the video clip to be classified with non-critical classification influence. The server can fuse the initial feature information to be classified corresponding to each video clip to be classified based on the primary attention information and the second-order attention information to obtain the target video feature to be classified, and determine the video classification result corresponding to the video to be classified based on the target video feature to be classified. The server can send the video classification result to the terminal for display on the terminal.

[0113] In one embodiment, Figure 2 As shown, a video classification method is provided, which is applied to Figure 1 The computer device in the example is used to illustrate, and the computer device can be the above Figure 1The terminal 102 or the server 104 in FIG. Figure 2 ,The video classification method includes the following steps:

[0114] Step S202: Acquire multiple video segments to be classified corresponding to the video to be classified.

[0115] The video to be classified refers to the video used to determine the video classification result. The video can be a video posted on various content platforms, such as a video posted on a social platform, a video playback platform, a video posted on a gaming platform, etc. A video clip includes at least one video frame in the video. The video frames in a video clip can be continuous video frames or spaced video frames. Different video clips can include the same video frames or different video frames, that is, different video clips can overlap or not overlap. The video clip to be classified is a video clip extracted from the video to be classified.

[0116] Specifically, the computer device can obtain the video to be classified locally or from another terminal or server, extract the video segments to be classified from the video to be classified, and obtain multiple video segments to be classified. The computer device can also directly obtain multiple video segments to be classified corresponding to the video to be classified. It is understood that after receiving a video uploaded by a user, the content platform can use the video as the video to be classified, classify the video to be classified based on the multiple video segments to be classified corresponding to the video to be classified, and obtain a video classification result corresponding to the video to be classified. The video classification result can be used for video recommendation or video search on the content platform.

[0117] Step S204 , performing three-dimensional feature extraction on each video segment to be classified, obtaining motion features in the time dimension and image features in the space dimension of each video segment to be classified, and forming corresponding initial feature information to be classified.

[0118] Three-dimensional feature extraction involves extracting features from both the two-dimensional pixel space and the third dimension of time. Extracting features from the temporal dimension yields motion features, which reflect the action and visual changes within the video. Extracting features from the spatial dimension yields image features, which reflect the visual information at a specific moment in the video. The initial feature information to be classified includes both motion and temporal features.

[0119] Specifically, the computer device can perform three-dimensional feature extraction on each video segment to be classified, obtaining motion features in the temporal dimension and image features in the spatial dimension of the video segment to be classified. The motion features and image features constitute initial feature information corresponding to the video segment to be classified. The computer device can perform three-dimensional feature extraction on each video segment to be classified, obtaining initial feature information corresponding to each video segment to be classified.

[0120] Step S206 : assigning corresponding primary attention information to the initial feature information to be classified corresponding to each video segment to be classified. The primary attention information is used to represent the attention intensity corresponding to the video segment to be classified.

[0121] Attention information can be used to reflect the importance of a video segment and its impact on the final classification result. Based on this attention information, a computer device can focus on the features that are more critical to the video classification result among the numerous initial features to be classified, reduce attention to other features, and even filter out irrelevant information. Attention information includes a quantifiable attention intensity. The higher the attention intensity corresponding to a video segment, the more important the video segment is. When determining the video classification result, the focus is primarily on video segments with high attention intensity. Primary attention information can reflect the general importance of a video segment. Primary attention information is used to characterize the attention intensity corresponding to the video segment to be classified. The attention intensity can indicate the degree of attention paid to the video segment to be classified, that is, the degree of influence of the video segment to be classified on the final classification result. Attention intensity is quantifiable data. Generally speaking, the richer the feature information in a video segment and the more important the video segment, the higher the attention intensity in the corresponding primary attention information. Based on this primary attention information, a computer device can classify standard videos and obtain accurate video classification results.

[0122] Specifically, the computer device may assign corresponding preliminary attention information to the initial feature information to be classified corresponding to each video segment to be classified. Specifically, the computer device may assign corresponding preliminary attention information to the initial feature information to be classified corresponding to each video segment to be classified based on the classification influence corresponding to each video segment to be classified. The attention intensity in the preliminary attention information corresponding to a video segment to be classified with a higher classification influence is greater than the attention intensity in the preliminary attention information corresponding to a video segment to be classified with a lower classification influence.

[0123] In one embodiment, the primary attention information includes at least one of soft attention information and hard attention information. Soft attention information is allocated to the initial feature information to be classified corresponding to the video segments to be classified based on a soft attention mechanism. Soft attention information can evenly reflect the importance of each video segment. The higher the classification influence of a video segment, the higher the attention intensity in the corresponding soft attention information. Hard attention information is allocated to the initial feature information to be classified corresponding to the video segments to be classified based on a hard attention mechanism. Hard attention information can specifically and distinctively reflect the importance of each video segment. Based on the classification influence, the video segments to be classified can be divided into a first category of video segments with a higher classification influence and a second category of video segments with a lower classification influence. The first category of video segments is uniformly allocated a higher attention intensity, while the second category of video segments is uniformly allocated a lower attention intensity.

[0124] Step S208, based on the classification influence corresponding to each video segment to be classified, corresponding second-order attention information is allocated to the initial feature information to be classified corresponding to each video segment to be classified, and the second-order attention information is used to improve the attention intensity corresponding to the video segment to be classified with non-critical classification influence.

[0125] Among them, the classification influence is the data obtained by quantifying the degree of influence of the video clip to be classified on the classification result. The value range corresponding to the classification influence is larger than the value range corresponding to the attention intensity. The computer device can determine the classification influence corresponding to the video clip to be classified based on the initial feature information to be classified corresponding to the video clip to be classified. The second-order attention information can reflect the special importance of the video clip. In the primary attention information, the attention intensity corresponding to the video clip to be classified with the key classification influence is higher than the attention intensity corresponding to the video clip to be classified with the non-key classification influence. Unlike the primary attention information, in the second-order attention information, the attention intensity corresponding to the video clip to be classified with the non-key classification influence can be higher than the attention intensity corresponding to the video clip to be classified with the key classification influence, or it can be equal to or lower than the attention intensity corresponding to the video clip to be classified with the key classification influence. The second-order attention information is used to improve the attention intensity corresponding to the video clip to be classified with the non-key classification influence. When the classification influence of the video clip to be classified is a key classification influence, it indicates that the video clip to be classified has a high decisive role in the classification result. When the classification influence of the video clip to be classified is a non-key classification influence, it indicates that the video clip to be classified has a general decisive role in the classification result. The computer device can distinguish between key classification influence and non-key classification influence based on a preset threshold, and regard the classification influence greater than or equal to the preset threshold as the key classification influence, and regard the classification influence less than the preset threshold as the non-key influence. The computer device can also sort the classification influence from large to small, and regard a preset number of classification influences at the top of the sort as key classification influences, and regard the other classification influences as non-key classification influences.

[0126] Specifically, the computer device can allocate corresponding second-order attention information to the initial feature information to be classified corresponding to each video segment to be classified based on the classification influence corresponding to each video segment to be classified. The second-order attention information is used to increase the attention intensity corresponding to the video segment to be classified of the non-critical classification influence. Based on the classification influence, the computer device can divide the video segments to be classified into a third category of video segments with higher classification influence and a fourth category of video segments with lower classification influence. In one embodiment, the computer device can uniformly allocate lower attention intensity to the third category of video segments. Furthermore, in the fourth category of video segments, the computer device re-allocates attention intensity to each video segment to be classified based on the classification influence. In this case, the higher the classification influence of the video segment, the higher the attention intensity in the corresponding second-order attention information. In one embodiment, the computer device can increase the classification influence of the fourth category of video segments while keeping the classification influence of the third category of video segments unchanged, and then normalize the classification influence of the third category of video segments and the corrected classification influence of the fourth category of video segments to obtain the second-order attention information corresponding to each initial feature information to be classified. When increasing the classification influence of the fourth-category video clips, the classification influence of all fourth-category video clips can be increased by a uniform preset value, or different preset values ​​can be added for different fourth-category video clips. Specifically, the preset value corresponding to the fourth-category video clip with a greater classification influence can be larger. In this way, based on the second-order attention information, the attention level corresponding to the video clips to be classified with non-key classification influence can be increased. Therefore, even if there is interference from the key frames in the video, accurate video classification results can be obtained by further using non-key frames.

[0127] For example, let's assume the video to be classified is a sports video. The corresponding video segments include video segment A, which consists of video frames before a shot, video segment B, which consists of video frames after the shot, and video segment C, which consists of video frames after the shot. Based on the initial feature information to be classified corresponding to each video segment, the computer device can determine the classification influence corresponding to each video segment. It is understandable that because video segment B contains richer motion information, the classification influence of video segment B is greater than that of video segments A and C. The computer device assigns corresponding preliminary attention information to the initial feature information to be classified corresponding to each video segment to be classified. In this preliminary attention information, the attention level of video segment B is greater than that of video segments A and C. However, if video segment B contains interfering information, such as a blurred or obscured shooting motion, the computer device will only focus on video segment B and ignore video segments A and C. Consequently, a relatively accurate video classification result cannot be obtained, and the shot cannot be clearly identified and labeled as a shot. Therefore, the computer device can further assign corresponding second-order attention information to the initial feature information corresponding to each video segment to be classified, using this second-order attention information to increase the attention level of video segments A and C. This way, when performing video classification, the computer device not only focuses on video segment B, but also on video segments A and C. Even if video segment B is interfered with, the computer device can still accurately classify the video by further combining video segments A and C, identifying the shot and assigning a shot label. By leveraging the contextual feature information of key frames, the computer device can achieve relatively accurate classification of videos with interfered key frames.

[0128] In step S210 , based on the primary attention information and the second-order attention information, the initial feature information to be classified corresponding to each video segment to be classified is fused to obtain the target video feature to be classified.

[0129] The target video features to be classified are feature information ultimately used to determine the video classification result.

[0130] Specifically, after obtaining the primary-order attention information and second-order attention information corresponding to each video segment to be classified, the computer device can fuse the initial feature information corresponding to each video segment to be classified based on the primary-order attention information and the second-order attention information to obtain the target video features to be classified. It can be understood that fusing the initial feature information corresponding to each video segment to be classified based on the primary-order attention information and the second-order attention information before determining the video classification result based on the feature information can effectively increase the proportion of important feature information and reduce the proportion of unimportant feature information. In this way, the computer device can quickly and accurately obtain the video classification result based on the target video features to be classified.

[0131] Step S212: determining a video classification result corresponding to the video to be classified based on the target video features to be classified.

[0132] Specifically, the computer device may determine a video classification result corresponding to the video to be classified based on the target video features to be classified. The computer device may classify the target video features to obtain a video classification result. For example, the computer device may input the target video features to be classified into a multi-layer perceptron, which may classify the target video features and output a video classification result.

[0133] In one embodiment, the video classification result is a video tag set, which includes hierarchical video tags from coarse-grained to fine-grained.

[0134] Specifically, the video classification result finally obtained by the computer device can be a single label or a video label set obtained by combining multiple video labels. The video labels in the video label set can be video labels of the same granularity or hierarchical video labels from coarse granularity to fine granularity. Coarse granularity video labels include scene labels, overall event / behavior labels, etc. Scene labels are used to describe the scene of the video, for example, Figure 3 The overall event / behavior tag is used to describe the overall and global event / behavior information of the video, for example, Figure 3 "Celebration" and "Frustration" under the overall branch in the video. Fine-grained video tags include close-up event / behavior tags. Close-up event / behavior tags are used to describe the event / behavior information of a part of the video, such as Figure 3 In this way, the computer equipment can analyze the video data, parse the semantic information of the video into multi-label classification, and output hierarchical video labels from coarse to fine granularity, completing the structured understanding of the video.

[0135] In one embodiment, the computer device can determine the classification probabilities of the video to be classified based on the features of the target video to be classified, for example, the classification probabilities of the video to be classified corresponding to the "entrance," "stands," "bench / sidelines," and "field" labels, and the classification probabilities of the video to be classified corresponding to the "shot," "pass," and "interception" labels. The computer device can then select candidate labels with classification probabilities greater than a preset threshold as the video labels corresponding to the video to be classified, thereby obtaining a video classification result for the video to be classified.

[0136] In one embodiment, the video classification results can be applied to video searches. For example, after a computer device receives a video search request carrying a search keyword, it can match the search keyword with the video tag of the video, generate a video search result based on at least one successfully matched video, and return the video search result to the sender of the video search request. The video classification results can also be applied to video recommendations. For example, after a computer device receives a user login notification, the computer device can analyze the user's historical video browsing history to obtain user preference information, where the user preference information includes the video tags of a category of videos that the user is interested in, match the user preference information with the video tags of the video to be recommended, generate a video recommendation result based on at least one successfully matched video to be recommended, and display the video recommendation result to the user. A successful match can be a complete string match or a semantic match.

[0137] In one embodiment, a computer device can perform video classification using a machine learning model. The computer device inputs multiple video clips corresponding to the video to be classified into a trained target video classification model. The machine learning model then performs three-dimensional feature extraction, allocation of primary attention information, allocation of secondary attention information, information fusion, and information classification, ultimately outputting a video classification result.

[0138] In the above-mentioned video classification method, multiple video clips to be classified corresponding to the video to be classified are obtained, and three-dimensional features are extracted for each video clip to be classified. The motion features in the time dimension and the image features in the spatial dimension of each video clip to be classified are obtained to form the corresponding initial feature information to be classified. The corresponding primary-order attention information is assigned to the initial feature information to be classified corresponding to each video clip to be classified. Based on the classification influence corresponding to each video clip to be classified, the corresponding second-order attention information is assigned to the initial feature information to be classified corresponding to each video clip to be classified. The second-order attention information is used to improve the attention intensity corresponding to the video clip to be classified with non-critical classification influence. Based on the primary-order attention information and the second-order attention information, the initial feature information to be classified corresponding to each video clip to be classified is fused to obtain the target video feature to be classified. The video classification result corresponding to the video to be classified is determined based on the target video feature to be classified. In this way, the second-order attention information can improve the attention intensity corresponding to the video clip to be classified with non-critical classification influence, that is, improve the attention intensity corresponding to the non-critical video clip. The feature information of the non-critical video clip can predict the feature information of the key frame video clip to a certain extent. Therefore, when classifying videos with key frames interfered with, the content of the video can also be identified based on the target video features to be classified that are fused with the feature information of non-key video segments. With the help of non-key video segments, the accuracy of video classification for videos with key frames interfered with can be improved.

[0139] In one embodiment, obtaining multiple video segments to be classified corresponding to the video to be classified includes:

[0140] The head video frame and the tail video frame are filtered from the video to be classified to obtain a candidate video; video frames are extracted at equal intervals from each candidate video frame in the candidate video to obtain multiple intermediate video frames; the head video frame, the tail video frame and each intermediate video frame are sorted according to the time information corresponding to the video frame to obtain a target video frame sequence; the target video frame sequence is segmented to obtain multiple video clips to be classified.

[0141] The head video frame may be the first frame in the video, or may include the first frame and at least one adjacent video frame of the first frame. The tail video frame may be the last frame in the video, or may include the last frame and at least one adjacent video frame of the last frame. It is understood that all video frames in the video carry time information.

[0142] Specifically, the computer device can segment the video to be classified at equal intervals to obtain multiple video segments to be classified. However, considering the vast number of short videos with varying lengths, to balance information validity and conciseness and reduce computational complexity, the computer device can extract frames at equal intervals from the video to be classified to obtain the video segments to be classified. Since the head and tail video frames in a video are relatively important, they need to be extracted separately. The computer device can filter the head and tail video frames from the video to be classified to obtain candidate videos. Within each candidate video frame, the computer device extracts frames at equal intervals to obtain multiple intermediate video frames. The rules for extracting frames at equal intervals can be set according to actual needs, for example, extracting one frame every ten frames. The computer device then sorts the extracted head, tail, and intermediate video frames in chronological order to obtain a target video frame sequence. Finally, the computer device segments the target video frame sequence to obtain multiple video segments to be classified. The segmentation rule can also be set according to actual needs. For example, every 16 video frames constitute a video segment to be classified.

[0143] In this embodiment, frames are extracted at equal intervals from the video to be classified, and the head video frame and the tail video frame are retained to obtain the video segments to be classified. Data analysis is performed based on such video segments to be classified, which can reduce the amount of calculation, reduce the calculation complexity, and improve the efficiency of video classification.

[0144] In one embodiment, three-dimensional feature extraction is performed on each video segment to be classified, and motion features in the time dimension and image features in the spatial dimension of each video segment to be classified are obtained to form corresponding initial feature information to be classified, including:

[0145] Based on each video clip to be classified, a corresponding three-dimensional video image block is formed. The three dimensions of the three-dimensional video image block are the length, width and number of video frames of the current video clip to be classified. Through a three-dimensional convolutional network composed of a three-dimensional convolutional layer and a three-dimensional pooling layer, the motion information between temporally adjacent pixels in the three-dimensional video image block is extracted to obtain motion features. The image information between spatially adjacent pixels in the three-dimensional video image block is extracted to obtain image features, which constitute the corresponding initial feature information to be classified.

[0146] The 3D convolutional network is a neural network used to extract 3D features from video clips to be classified. It consists of a 3D convolutional layer and a 3D pooling layer. The 3D convolutional layer performs convolution processing on the input data to extract feature information. The 3D pooling layer reduces the dimensionality of the input data, thereby removing redundant information, compressing features, and simplifying network complexity.

[0147] Specifically, the computer device can form three-dimensional video image blocks corresponding to each video segment to be classified based on each video segment to be classified. The three dimensions of the three-dimensional video image blocks are the length, width, and number of video frames in the current video segment to be classified. The computer device can use a three-dimensional convolutional network composed of a three-dimensional convolutional layer and a three-dimensional pooling layer to extract motion information between temporally adjacent pixels in the three-dimensional video image blocks to obtain motion features, and extract image information between spatially adjacent pixels in the three-dimensional video image blocks to obtain image features. The extracted motion features and image features form initial feature information to be classified corresponding to the video segment to be classified.

[0148] In one embodiment, the 3D convolutional network may further include a fully connected layer, which is used to further extract and classify features. Figure 4AThe 3D convolutional network consists of 8 convolutional layers, 5 max pooling layers, and 2 fully connected layers. The number of convolution kernels in the 8 convolutional layers is 64, 128, 256, 256, 512, 512, 512, and 512, respectively. The size of the convolution kernel in each convolutional layer is 3×3×3, and the stride is 1. In the 5 max pooling layers, except for the first max pooling layer with a pooling kernel of 1×2×2, the pooling kernels of the other max pooling layers are 2×2×2. The max pooling layer traverses the feature map according to the size of the pooling kernel and retains the feature with the maximum response of the corresponding block in the feature map. The computer device inputs the video clip to be classified into the three-dimensional convolutional network, extracts the motion information between temporally adjacent pixels in the three-dimensional video image block through the convolution layer in the three-dimensional convolutional network to obtain motion features, extracts the image information between spatially adjacent pixels in the three-dimensional video image block to obtain image features, reduces the dimensionality of the extracted motion features and image features through the maximum pooling layer in the three-dimensional convolutional network, and finally outputs the initial feature information to be classified corresponding to the video clip to be classified through the fully connected layer.

[0149] refer to Figure 4B , Figure 4B Figure 1. Schematic diagram of 3D feature extraction using a convolution kernel. x and y represent the length and width of a video frame, and z represents the number of frames. The convolution kernel extracts feature information by traversing all pixels in a 3D video image block.

[0150] In this embodiment, by performing three-dimensional feature extraction on the video clip to be classified through a three-dimensional convolutional network, initial feature information to be classified including motion features and image features can be obtained, so that a more accurate video classification result can be obtained based on the initial feature information to be classified.

[0151] In one embodiment, the preliminary attention information includes soft attention information, and the preliminary attention information is assigned to the initial feature information to be classified corresponding to each video segment to be classified, including:

[0152] The first classification influence corresponding to each initial feature information to be classified is calculated, and each first classification influence is normalized to obtain the soft attention information corresponding to each video clip to be classified.

[0153] Among them, the soft attention information is the attention information obtained by allocating attention to the initial feature information to be classified corresponding to the video clip to be classified based on the soft attention mechanism. The soft attention information can evenly reflect the importance of each video clip. The higher the classification influence of the video clip, the higher the attention intensity in the corresponding soft attention information.

[0154] Specifically, the computer device can calculate the first classification influence corresponding to each initial feature information to be classified based on each initial feature information to be classified, normalize each first classification influence to obtain the attention intensity corresponding to each initial feature information to be classified, and obtain the soft attention information corresponding to each video clip to be classified based on the attention intensity corresponding to each initial feature information to be classified.

[0155] For example, a computer device analyzes the initial feature information to be classified of each video segment to be classified, and can encode each initial feature information to be classified into a T-dimensional vector, where T represents the number of video segments to be classified, and the initial vector value of each vector represents the first classification influence of the corresponding video segment to be classified on the final classification result. The computer device can normalize all the initial vector values ​​to obtain a target vector value, and the target vector value can be used as the attention intensity in the soft attention information corresponding to the video segment to be classified. The normalization process can specifically be to convert all the initial vector values ​​to between 0 and 1, and the sum of all the target vector values ​​obtained by the normalization process is 1. Finally, a T-dimensional vector representing the soft attention information corresponding to each video segment to be classified is obtained. It can be understood that the larger the initial vector value, the larger the target vector value, that is, the greater the first classification influence, and the greater the attention intensity in the corresponding soft attention information.

[0156] In this embodiment, the first classification influence corresponding to each initial feature information to be classified is calculated and normalized to obtain soft attention information corresponding to each video segment to be classified. This allows for a balanced allocation of soft attention information to each video segment to be classified based on its classification influence. During video classification, the soft attention information can be used to focus on the feature information of important video segments, thereby improving the accuracy of video classification.

[0157] In one embodiment, the preliminary attention information includes hard attention information, and the preliminary attention information is assigned to the initial feature information to be classified corresponding to each video segment to be classified, including:

[0158] Calculate the second classification influence corresponding to each initial feature information to be classified; divide each video clip to be classified into a first category video clip and a second category video clip based on the second classification influence, and the second classification influence corresponding to the first category video clip is greater than the second classification influence corresponding to the second category video clip; determine the attention intensity corresponding to the first category video clip as a first preset value, and determine the attention intensity corresponding to the second category video clip as a second preset value, and the first preset value is greater than the second preset value; normalize the attention intensity corresponding to each video clip to be classified to obtain hard attention information corresponding to each video clip to be classified.

[0159] Among them, the hard attention information is the attention information obtained by allocating attention to the initial feature information to be classified corresponding to the video clip to be classified based on the hard attention mechanism. The hard attention information can specifically and distinctively reflect the importance of each video clip.

[0160] Specifically, the computer device can calculate the second classification influence corresponding to each initial feature information to be classified based on each initial feature information to be classified, and divide each video clip to be classified into a first category video clip and a second category video clip based on the second classification influence, wherein the second classification influence corresponding to the first category video clip is greater than the second classification influence corresponding to the second category video clip. Then, the computer device can determine the attention intensity corresponding to the first category video clip as a first preset value, and determine the attention intensity corresponding to the second category video clip as a second preset value, wherein the first preset value is greater than the second preset value. In this way, based on the first preset value and the second preset value, it is possible to clearly distinguish between video clips to be classified with high classification influence and video clips to be classified with low classification influence. Then, the computer device can normalize the attention intensity corresponding to each video clip to be classified to obtain hard attention information corresponding to each video clip to be classified.

[0161] For example, a computer device analyzes the initial feature information to be classified for each video segment to be classified, and can encode each initial feature information to be classified into a T-dimensional vector, where T represents the number of video segments to be classified, and the vector value of each vector represents the second-class influence of the corresponding video segment to be classified on the final classification result. The computer device can find the video segment to be classified with the largest vector value from the video segments to be classified as the first-class video segment, and the other video segments to be classified as the second-class video segments. The vector value corresponding to the first-class video segment is corrected to 1, and the vector value corresponding to the second-class video segment is corrected to 0. Ultimately, a T-dimensional vector representing the hard attention information corresponding to each video segment to be classified is obtained.

[0162] In this embodiment, hard attention information can be allocated to each video segment to be classified in a targeted manner according to the classification influence of each video segment to be classified, thereby improving the distinction of attention information between important video segments and unimportant video segments. When classifying videos, the hard attention information can be used to focus on the feature information of important video segments, thereby helping to improve the accuracy of video classification.

[0163] In one embodiment, based on the classification influence corresponding to each video segment to be classified, corresponding second-order attention information is allocated to the initial feature information to be classified corresponding to each video segment to be classified, including:

[0164] Calculate the third classification influence corresponding to each initial feature information to be classified; divide each video clip to be classified into a third category video clip and a fourth category video clip based on the third classification influence, the third classification influence corresponding to the third category video clip is greater than the third classification influence corresponding to the fourth category video clip, the third classification influence corresponding to the third category video clip is a key classification influence, and the third classification influence corresponding to the fourth category video clip is a non-key classification influence; determine the attention intensity corresponding to the third category video clip as a second preset value, normalize the third classification influence corresponding to the fourth category video clip, and obtain the attention intensity corresponding to the fourth category video clip; obtain the second-order attention information corresponding to each video clip to be classified based on the attention intensity corresponding to each video clip to be classified.

[0165] Specifically, the computer device may calculate the third-category influence corresponding to each initial feature information to be classified based on the initial feature information to be classified, and classify each video segment to be classified into a third category and a fourth category based on the third-category influence, wherein the third-category influence corresponding to the third category video segment is greater than the third-category influence corresponding to the fourth category video segment, the third-category influence corresponding to the third category video segment is a key category influence, and the third-category influence corresponding to the fourth category video segment is a non-key category influence. Next, the computer device may determine the attention intensity corresponding to the third category video segment as a second preset value, and normalize the third-category influence intensity corresponding to the fourth category video segment to obtain the attention intensity corresponding to the fourth category video segment. It will be appreciated that, compared to the first normalization result obtained by normalizing all third-category influence indices, the attention intensity corresponding to each fourth category video segment is improved in the second normalization result obtained by normalizing only the third-category influence intensity corresponding to the fourth category video segment. Then, based on the attention intensity corresponding to each video segment to be classified, the computer device obtains second-order attention information corresponding to each video segment to be classified.

[0166] For example, the computer device analyzes the initial feature information to be classified of each video clip to be classified, and can encode each initial feature information to be classified to form a T-dimensional vector, where T represents the number of video clips to be classified, and the vector value of each vector represents the third classification influence of the corresponding video clip to be classified on the final classification result. The computer device can sort the video clips to be classified from large to small based on the third classification influence, correct the vector values ​​corresponding to the top-K video clips to be classified in the sorting result to 0, and normalize the vector values ​​corresponding to the remaining video clips to be classified so that the sum of the vector values ​​corresponding to the remaining video clips to be classified is 1. Finally, a T-dimensional vector representing the second-order attention information corresponding to each video clip to be classified is obtained.

[0167] It can be understood that the first classification influence, second classification influence, and third classification influence corresponding to the same initial feature information to be classified can be the same or different. Even if they are different, the first classification influence, second classification influence, and third classification influence corresponding to the same initial feature information to be classified are also similar. The computer device can use a machine learning model to assign corresponding primary attention information and second-order attention information to the initial feature information to be classified corresponding to each video clip to be classified. The computer device can assign corresponding primary attention information to the initial feature information to be classified corresponding to each video clip to be classified through a primary attention network, and the primary attention network includes at least one of a soft attention network and a hard attention network. The computer device can assign corresponding second-order attention information to the initial feature information to be classified corresponding to each video clip to be classified based on the classification influence corresponding to each video clip to be classified through a semi-soft attention network. The current attention network is a primary-order attention network or a second-order attention network. After the computer device inputs the initial feature information to be classified corresponding to each video clip to be classified into the current attention network, the current attention network performs primary-order full-connection processing on each initial feature information to obtain the initial classification influence corresponding to each video clip to be classified, performs nonlinear processing on each initial classification influence to obtain the intermediate classification influence corresponding to each video clip to be classified, performs advanced full-connection processing on each intermediate classification influence to obtain the current classification influence corresponding to each video clip to be classified, and based on the current classification influence corresponding to each video clip to be classified, allocates corresponding current-order attention information to the initial feature information to be classified corresponding to each video clip to be classified.

[0168] In this embodiment, the attention intensity corresponding to the video segments to be classified with non-critical classification influence can be increased based on the classification influence of each video segment to be classified, thereby obtaining second-order attention information. When classifying videos, the second-order attention information can be used to increase attention to non-critical video segments. When important video segments are disturbed, the content of the important video segments can be predicted with the help of non-critical video segments to obtain a more accurate classification result, thereby improving the classification accuracy of videos with disturbed key frames.

[0169] In one embodiment, based on the primary attention information and the second-order attention information, the initial feature information to be classified corresponding to each video segment to be classified is fused to obtain the target video feature to be classified, including:

[0170] The primary attention information and the second-order attention information corresponding to the same initial feature information to be classified are fused to obtain the target attention information corresponding to each initial feature information to be classified; the initial feature information to be classified and the target attention information corresponding to the same video clip to be classified are weightedly summed to obtain the intermediate feature information to be classified corresponding to each video clip to be classified; and the target video feature to be classified is obtained based on each intermediate feature information to be classified.

[0171] Specifically, when fusing the initial feature information to be classified, the computer device may first fuse the primary-order attention information and the second-order attention information corresponding to the same initial feature information to be classified to obtain the target attention information corresponding to each initial feature information to be classified. For example, the average of the primary-order attention information and the second-order attention information corresponding to a piece of initial feature information to be classified may be used as the target attention information corresponding to the initial feature information to be classified. Alternatively, the maximum value of the primary-order attention information and the second-order attention information corresponding to a piece of initial feature information to be classified may be obtained as the target attention information corresponding to the initial feature information to be classified. Alternatively, the sum of the primary-order attention information and the second-order attention information corresponding to a piece of initial feature information to be classified may be used as the target attention information corresponding to the initial feature information to be classified. Next, the computer device performs a weighted sum of the initial feature information to be classified and the target attention information corresponding to the same video clip to be classified to obtain the intermediate feature information to be classified corresponding to each video clip to be classified. Finally, the computer device obtains the target video feature to be classified based on each piece of intermediate feature information to be classified. For example, the target video feature to be classified may be obtained by combining each piece of intermediate feature information to be classified.

[0172] For example, suppose there are three video clips, and the corresponding initial feature information to be classified is A, B, and C, all of which are 1024-dimensional vectors. Then, the initial video features to be classified are 3×1024-dimensional vectors. The soft attention information corresponding to the three video clips is 0.1, 0.2, and 0.7, respectively; the hard attention information corresponding to the three video clips is 0, 0, and 1, respectively; and the second-order attention information corresponding to the three video clips is 0.5, 0.5, and 0, respectively. The soft attention information, hard attention information, and second-order attention information corresponding to the same video clip are summed to obtain the target attention information corresponding to the three video clips. The target attention information for the three video clips is 0.6, 0.7, and 1.7, respectively. The matrix composed of the target attention information corresponding to the three video clips is dot-multiplied with the matrix composed of the initial feature information to be classified, to obtain the target video features to be classified.

[0173] In one embodiment, the video classification result includes a motion tag. Determining the video classification result corresponding to the target video to be classified based on the features of the target video to be classified includes:

[0174] Based on the image features in the target video features to be classified, object recognition is performed on the video to be classified to obtain multiple target objects in the video to be classified; based on the motion features in the target video features to be classified, global trajectory recognition is performed on the target object to obtain the global motion trajectory corresponding to each target object; based on the global motion trajectory corresponding to each target object, interaction information between each target object is determined; based on the interaction information between each target object and the object type of the target object, the motion label corresponding to the video to be classified is obtained.

[0175] The global motion trajectory refers to the motion trajectory of the target object as a whole. Interaction information refers to the information generated by the interaction between target objects. This information can include the time when the interaction occurs and the motion information before and after the interaction. Motion tags are used to describe motion information in the video.

[0176] Specifically, the computer device can identify motion behaviors in the video to be classified based on the target video features to be classified, and obtain motion labels corresponding to the video to be classified. First, the computer device can perform object recognition on the video to be classified based on the image features in the target video features to be classified, obtaining multiple target objects in the video to be classified. It will be appreciated that image features are derived based on image information in video frames at each moment, and the image information corresponding to different target objects is obviously different. For example, a player and a ball are distinct in color, shape, and texture, and thus clearly correspond to different image features in the video frames. Therefore, the computer device can pre-learn image features corresponding to various target objects. After obtaining the target video features corresponding to the video to be classified, the computer device can match the image features in the target video features with the pre-learned image features corresponding to known target objects. Based on the matching results, the computer device can determine which target objects appear in the video to be classified. The target objects in the video to be classified can be of the same type or different types. For example, a ball and a person in a video are different types of target objects.

[0177] The computer device can further perform global trajectory identification on the target object based on the motion features in the target video features to be classified, obtaining the global motion trajectory corresponding to each target object. It will be understood that the motion features are obtained based on image change information between video frames at different times. Therefore, after identifying the target object in the video to be classified, the computer device can perform trajectory identification on the target object based on the motion features in the target video features to obtain the global motion trajectory of the target object in the video. Different target objects can correspond to different global motion trajectories.

[0178] Next, the computer device can determine interaction information between the target objects based on the global motion trajectories corresponding to the target objects. For example, the computer device analyzes the direction of the global motion trajectories. If the global motion trajectories of different target objects intersect, it indicates that there is interaction between the target objects. Therefore, the computer device can generate corresponding interaction information between the target objects based on the intersecting global motion trajectories. If there is no interaction between the target objects, the corresponding interaction information may be none.

[0179] Finally, the computer device can obtain a motion label corresponding to the video to be classified based on the interaction information between the target objects and the object types of the target objects. It can be understood that the computer device can determine what interactions occurred between specific objects based on the object types and interaction information between the target objects, thereby determining which specific motion events / behaviors occurred in the video and obtaining the corresponding motion label.

[0180] For example, a computer device performs object recognition on a target video to be classified based on image features within the target video to be classified, determining the presence of a ball and a player in the target video to be classified. Based on the motion features within the target video to be classified, the computer device performs global trajectory recognition on the target object to obtain the corresponding global motion trajectories of the ball and the player. Furthermore, based on the corresponding global motion trajectories of the ball and the player, the computer device can determine the interaction information between the ball and the player. For example, if the global motion trajectory of the ball intersects with the global motion trajectory of the player, and after the trajectory intersection, the global motion trajectories of the ball and the player substantially overlap, then the computer device can determine the motion label "running with the ball" based on the interaction information between the ball and the player.

[0181] In one embodiment, a computer device can train a machine learning model and use it to classify features of a target video to be classified. The computer device can input the features of the target video to be classified into the trained machine learning model, which then processes data internally to perform object recognition, global trajectory recognition, and interaction information determination and analysis, outputting a motion label corresponding to the video to be classified.

[0182] In this embodiment, the motion label corresponding to the target video to be classified can be obtained by analyzing the image features and motion features in the target video to be classified.

[0183] In one embodiment, determining interaction information between target objects based on the global motion trajectories corresponding to the target objects includes:

[0184] Based on the intersecting global motion trajectories and the motion directions of the global motion trajectories before and after the intersection, the interaction information between the corresponding target objects is generated.

[0185] Specifically, the computer device can generate interaction information between corresponding target objects based on the presence of intersecting global motion trajectories and the motion directions of the global motion trajectories before and after the intersection. This interaction information can record relevant information about the intersection points between the global motion trajectories and the motion directions of the target objects before and after the intersection. Subsequently, based on the interaction information between the target objects and the object types of the target objects, the motion label corresponding to the video to be classified can be obtained. For example, when passing a ball, a specific interaction occurs between the ball and the player. Therefore, if the interaction information between two target objects can reflect that the global motion trajectories of the two target objects have a trajectory intersection, and after the trajectory intersection, the two target objects move away from each other, and the object types of the two target objects are respectively the ball and the player, then it can be determined that there is a high probability of a passing action in the video, and the passing label can be used as the motion label of the video. When cooking, specific interactions also occur between ingredients and kitchen utensils. When eating, specific interactions also occur between food and people.

[0186] In this embodiment, interaction information between corresponding target objects is generated based on the intersecting global motion trajectories and the motion directions of the global motion trajectories before and after the intersection. Subsequently, the motion label corresponding to the video can be quickly and accurately determined based on the interaction information.

[0187] In one embodiment, the video classification result includes an emotion label. Determining the video classification result corresponding to the target video to be classified based on the features of the target video to be classified includes:

[0188] Based on the image features in the target video features to be classified, object recognition is performed on the video to be classified to obtain multiple target objects in the video to be classified; based on the motion features in the target video features to be classified, key point trajectories of the target objects are identified to determine the local motion trajectories corresponding to the key points of each target object; based on the local motion trajectories, state change information of the corresponding target object is determined; based on the state change information of each target object and the object type of the target object, the emotion label corresponding to the video to be classified is obtained.

[0189] The local motion trajectory refers to the motion trajectory of the target object's key points when it is an individual. State change information refers to information generated by the target object's state changes. State change information can include the motion direction of each key point and the positional relationship between key points.

[0190] Specifically, the computer device can identify emotional events in the video to be classified based on the target video features to be classified, and obtain the emotional label corresponding to the video to be classified. First, the computer device can perform object recognition on the video to be classified based on the image features in the target video features to be classified, and obtain multiple target objects in the video to be classified. The computer device can pre-learn the image features corresponding to various target objects, so that after obtaining the target video features corresponding to the video to be classified, the image features in the target video features to be classified can be matched with the pre-learned image features corresponding to the known target objects, and based on the matching results, it can determine which target objects appear in the video to be classified. The target object includes at least one key point. For example, when the target object is a person, the key points of the target object can be the facial features and limbs of the person. After identifying the target object, the computer device can locate the key points on the target object.

[0191] The computer device can further perform trajectory identification on key points on the target object based on the motion features in the target video features to be classified, that is, perform key point trajectory identification on the target object to obtain local motion trajectories corresponding to the key points on each target object. It can be understood that motion features are obtained based on image change information between video frames at different times. Therefore, after identifying the target object in the video to be classified, the computer device can perform trajectory identification on key points on the target object based on the motion features in the target video features to obtain local motion trajectories corresponding to the key points of the target object. Different key points can correspond to different local motion trajectories.

[0192] Next, the computer device can determine the state change information of the corresponding target object based on the local motion trajectory. Specifically, the computer device can analyze the direction of the local motion trajectory of each key point on the same target object to obtain the target object's state change information. For example, the computer device can analyze the local motion trajectory of a person's facial features and obtain state change information such as an upward turn of the mouth corners. Analysis of the local motion trajectory of a person's limbs can obtain state change information such as raised arms or swinging arms.

[0193] Finally, the computer device can obtain the emotion label corresponding to the video to be classified based on the state change information of each target object and the object type of the target object. It can be understood that when the object type of the target object is a person, the computer device can determine the target object's emotional changes based on the target object's state change information, and then obtain the corresponding emotion label. Furthermore, emotion labels also have different granularities. Coarse-grained emotion labels are determined based on the state change information of the vast majority of target objects, while fine-grained emotion labels are determined based on the state change information of a single or local target object.

[0194] For example, a computer device performs object recognition on a target video based on image features within the target video to be classified, identifying the presence of an audience member in the target video. Based on the motion features within the target video, the computer device then performs key point trajectory recognition on the audience's facial features and limbs, obtaining local motion trajectories corresponding to the facial features and limbs. Furthermore, based on the local motion trajectories corresponding to the audience's facial features and limbs, the computer device can determine the audience's state change information. For example, if the audience member's mouth corners are raised, their arms are raised, or their arms are swinging, the computer device can determine the emotional label "celebration" based on the audience's state change information.

[0195] In one embodiment, a computer device can train a machine learning model and use it to classify features of a target video to be classified. The computer device can input the features of the target video to be classified into the trained machine learning model. The model then processes data internally to perform object recognition, key point trajectory recognition, and state change information determination and analysis, outputting an emotion label corresponding to the video to be classified.

[0196] In this embodiment, the emotion label corresponding to the target video to be classified can be obtained by analyzing the image features and motion features in the target video to be classified.

[0197] In one embodiment, Figure 5A As shown, the method further includes:

[0198] Step S502: Input each video clip to be classified into the trained target video classification model.

[0199] In step S504, three-dimensional features are extracted from each video segment to be classified through the three-dimensional convolutional network of the target video classification model to obtain motion features in the time dimension and image features in the spatial dimension of each video segment to be classified, forming corresponding initial feature information to be classified.

[0200] In step S506, corresponding primary attention information is assigned to the initial feature information to be classified corresponding to each video segment to be classified through the primary attention network of the target video classification model; the primary attention network includes at least one of a soft attention network and a hard attention network.

[0201] In step S508 , the semi-soft attention network of the target video classification model allocates corresponding second-order attention information to the initial feature information to be classified corresponding to each video segment to be classified based on the classification influence corresponding to each video segment to be classified.

[0202] In step S510 , the initial feature information to be classified corresponding to each video segment to be classified is fused based on the primary attention information and the second-order attention information to obtain the target video feature to be classified.

[0203] Step S512: Determine a video classification result corresponding to the video to be classified based on the target video features through the fully connected network of the target video classification model.

[0204] The 3D convolutional network is used to extract 3D features from video clips. The primary attention network is used to allocate primary attention information to the initial feature information to be classified. The primary attention network includes at least one of a soft attention network and a hard attention network. The soft attention network allocates soft attention information to the initial feature information to be classified, while the hard attention network allocates hard attention information to the initial feature information to be classified. The semi-soft attention network allocates second-order attention information to the initial feature information to be classified. The fully connected network is used for feature classification.

[0205] Specifically, a computer device can classify videos using a machine learning model. The target video classification model used for video classification includes a three-dimensional convolutional network, a preliminary attention network, a semi-soft attention network, and a fully connected network. After the computer device obtains multiple video segments corresponding to the video to be classified, it can input each of the video segments to be classified into the trained target video classification model. The three-dimensional convolutional network of the target video classification model then extracts three-dimensional features from each of the video segments to be classified, obtaining initial feature information corresponding to each of the video segments to be classified. Each of the initial feature information to be classified is then input into the preliminary attention network, where preliminary attention information is assigned to each of the initial feature information corresponding to the video segments to be classified. Each of the initial feature information to be classified is then input into the semi-soft attention network, where second-order attention information is assigned to each of the initial feature information corresponding to the video segments to be classified based on their corresponding classification influence. Based on the preliminary attention information and the second-order attention information, the initial feature information corresponding to each of the video segments to be classified is fused to obtain the target video features to be classified. Finally, the target video features to be classified are input into the fully connected network, the target video features to be classified are classified, and the video classification results corresponding to the video to be classified are obtained. The target video classification model outputs the video classification results.

[0206] It can be understood that the specific process of each network processing data can refer to the methods described in the aforementioned relevant embodiments, and will not be repeated here.

[0207] There are many types of fully connected networks, such as support vector machines (SVM), multi-layer perceptrons (MLP), fully connected layers (FC), random forests, and decision trees. Figure 5BThe computer device can select a multi-layer perceptron as a fully connected network. The computer device inputs the target video features to be classified into the multi-layer perceptron. The target video features to be classified are first input into the input layer of the multi-layer perceptron, and then the multi-layer perceptron operation is performed. The output layer outputs the video classification result. Figure 5B Each circle in the figure represents a neuron, and the operation of each neuron is as follows Figure 5C As shown in the figure, x1 and x2 are one-dimensional data in the target video feature to be classified, w1 and w2 are weights, b is the bias, and the output of the neuron is Y = f(w1×x1+w2×x2+b). When the target video feature to be classified is a 4096-dimensional feature vector, the input layer of the multilayer perceptron includes 4096 neurons, the hidden layer includes 201 perceptron neurons, and the output layer has 1 neuron.

[0208] In this embodiment, the target video classification model is used to perform video classification on the video to be classified, and a corresponding video classification result is obtained, which can improve the efficiency of video classification.

[0209] In one embodiment, the current attention network is a primary-order attention network or a second-order attention network, and the data processing of the input data by the current attention network includes the following steps:

[0210] Perform preliminary full-connection processing on each initial feature information to be classified to obtain the initial classification influence corresponding to each video clip to be classified; perform nonlinear processing on each initial classification influence to obtain the intermediate classification influence corresponding to each video clip to be classified; perform advanced full-connection processing on each intermediate classification influence to obtain the current classification influence corresponding to each video clip to be classified; based on the current classification influence corresponding to each video clip to be classified, allocate corresponding current-order attention information to the initial feature information to be classified corresponding to each video clip to be classified.

[0211] Specifically, the network structures of the primary and second-order attention networks are similar, and the data processing processes for input data are also similar. Therefore, taking the current attention network as an example for explanation, the current attention network is either a primary or a second-order attention network. After receiving each initial feature information to be classified, the current attention network first performs a primary fully connected process on the initial feature information corresponding to the video segment to be classified, obtaining the initial classification influence corresponding to each video segment to be classified. Next, each initial classification influence is subjected to nonlinear processing to obtain the intermediate classification influence corresponding to each video segment to be classified. Finally, each intermediate classification influence is subjected to an advanced fully connected process to obtain the current classification influence corresponding to each video segment to be classified. Finally, based on the current classification influence corresponding to each video segment to be classified, the corresponding current-order attention information is assigned to the initial feature information to be classified. The primary fully connected process refers to the first fully connected process, and the advanced fully connected process refers to the second fully connected process. The two fully connected processes can avoid network overfitting and underfitting, improve network performance, and compress multidimensional data into one-dimensional data. Non-linear processing can specifically be non-linear processing of data through an excitation function, such as the ReLU (Rectified Linear Unit) function. Non-linear processing can also prevent network overfitting. For example, the input data of the current attention network is a T×1024 vector. Through the first full connection processing, the vector can be compressed to T×256. Through non-linear processing, the vector is still T×256. Through the second full connection processing, the vector can finally be compressed to T×1, where T represents the number of video segments to be classified.

[0212] refer to Figure 6 The soft attention network includes a fully connected layer, a ReLU layer, a fully connected layer, and a normalization layer (softmax) connected in sequence. The computer device can perform preliminary fully connected processing on each initial feature information to be classified through the first fully connected layer in the soft attention network to obtain the initial classification influence corresponding to each video segment to be classified, perform nonlinear processing on each initial classification influence through the ReLU layer in the soft attention network to obtain the intermediate classification influence corresponding to each video segment to be classified, perform advanced fully connected processing on each intermediate classification influence through the second fully connected layer in the soft attention network to obtain the first classification influence corresponding to each video segment to be classified, and assign corresponding soft attention information to the initial feature information to be classified corresponding to each video segment to be classified based on the first classification influence corresponding to each video segment to be classified through the normalization layer in the soft attention network.

[0213] The hard attention network includes a fully connected layer, a ReLU layer, a fully connected layer, a first processing layer, and a normalization layer (softmax) connected in sequence. The computer device can perform preliminary fully connected processing on each initial feature information to be classified through the first fully connected layer in the hard attention network to obtain an initial classification influence corresponding to each video segment to be classified, perform nonlinear processing on each initial classification influence through the ReLU layer in the hard attention network to obtain an intermediate classification influence corresponding to each video segment to be classified, perform advanced fully connected processing on each intermediate classification influence through the second fully connected layer in the hard attention network to obtain a second classification influence corresponding to each video segment to be classified, and classify each video segment to be classified into a first category and a second category based on the second classification influence corresponding to each video segment to be classified through the first processing layer in the hard attention network, determine the attention intensity corresponding to the first category video segment as a first preset value, and determine the attention intensity corresponding to the second category video segment as a second preset value, and assign corresponding hard attention information to the initial feature information to be classified corresponding to each video segment to be classified through the normalization layer in the hard attention network. Among them, if in the first processing layer, the attention intensity corresponding to the video clip with the greatest influence in the second classification is set to 1, and the attention intensity corresponding to the video clips with other classification influences is set to 0, then the hard attention network does not need to input the data into the normalization layer for processing, and can directly output the set attention intensity as hard attention information.

[0214] The advanced attention network (also called semi-soft attention network) consists of a fully connected layer, a ReLU layer, a fully connected layer, a second processing layer, and a normalization layer (softmax) connected in sequence. The computer device can perform preliminary full-connection processing on each initial feature information to be classified through the first fully-connected layer in the semi-soft attention network to obtain the initial classification influence corresponding to each video clip to be classified, perform nonlinear processing on each initial classification influence through the ReLU layer in the semi-soft attention network to obtain the intermediate classification influence corresponding to each video clip to be classified, perform advanced full-connection processing on each intermediate classification influence through the second fully-connected layer in the semi-soft attention network to obtain the third classification influence corresponding to each video clip to be classified, divide each video clip to be classified into a third category video clip and a fourth category video clip based on the third classification influence corresponding to each video clip to be classified through the second processing layer in the semi-soft attention network, set the attention intensity corresponding to the third category video clip to a second preset value, perform normalization processing on the third classification influence corresponding to the fourth category video clip through the normalization layer in the semi-soft attention network to obtain the attention intensity corresponding to the fourth category video clip, and obtain second-order attention information (also called semi-soft attention information) based on the attention intensity corresponding to each video clip to be classified.

[0215] It can be understood that the network parameters of the fully connected layer and the ReLU layer in the soft attention network, the hard attention network, and the semi-soft attention network can be the same or different.

[0216] In one embodiment, Figure 7 As shown, a video classification method is provided, which is applied to Figure 1 The computer device in the example is used to illustrate, and the computer device can be the above Figure 1 The terminal 102 or the server 104 in FIG. Figure 7 ,The video classification method includes the following steps:

[0217] Step S702: Obtain multiple training video segments corresponding to the training video and training labels corresponding to the training video, and input each training video segment into the target video classification model to be trained.

[0218] Step S704 : performing three-dimensional feature extraction on each training video segment to obtain motion features in the time dimension and image features in the space dimension of each training video segment to form corresponding initial training feature information.

[0219] Step S706 : assigning corresponding primary attention information to the initial training feature information corresponding to each training video segment. The primary attention information is used to characterize the attention intensity corresponding to the training video segment.

[0220] Step S708, based on the classification influence corresponding to each training video clip, corresponding second-order attention information is allocated to the initial training feature information corresponding to each training video clip, and the second-order attention information is used to improve the attention intensity corresponding to the training video clip with non-critical classification influence.

[0221] In step S710 , the initial training feature information corresponding to each training video segment is fused based on the primary attention information and the second-order attention information to obtain the target training video feature.

[0222] Step S712, determine the target prediction label corresponding to the training video based on the target training video features, and adjust the model parameters of the target video classification model based on the difference between the training label and the target prediction label until the convergence condition is met, thereby obtaining the trained target video classification model.

[0223] Specifically, the computer device can obtain training samples to train a target video classification model for classifying videos. The computer device can obtain training videos and corresponding training labels locally, from other terminals, or from a server. The computer device can further extract training video segments from the training videos to obtain multiple training video segments. The computer device can also directly obtain multiple training video segments and corresponding training labels corresponding to the training videos.

[0224] Next, the computer device can input each training video clip into the target video classification model to be trained. The target video classification model can be used to extract three-dimensional features from each training video clip, obtaining motion features in the temporal dimension and image features in the spatial dimension for each training video clip, which are then used to form corresponding initial training feature information. Corresponding primary attention information is assigned to the initial training feature information corresponding to each training video clip. Based on the classification influence corresponding to each training video clip, corresponding second-order attention information is assigned to the initial training feature information corresponding to each training video clip. Based on the primary attention information and the second-order attention information, the initial training feature information corresponding to each training video clip is fused to obtain target training video features. A target prediction label corresponding to the training video is determined based on the target training video features. The computer device can adjust the model parameters of the target video classification model based on the difference between the training label and the target prediction label until a convergence condition is met, thereby obtaining a trained target video classification model. The convergence condition can be customized, such as when the number of iterations reaches an iteration threshold or when the difference between the training label and the target prediction label reaches a minimum. Adjusting the model parameters can specifically be calculating the difference between the training label and the target prediction label, adjusting the model parameters of the target video classification model through difference back propagation, and continuing training until the updated difference or the number of iterations meets the convergence condition. The training is completed and the trained target video classification model is obtained.

[0225] It can be understood that the specific process of the target video classification model processing the input data to obtain the target prediction label can refer to the methods described in the aforementioned relevant embodiments, and will not be repeated here.

[0226] In one embodiment, the target video classification model can output at least one target prediction label corresponding to a video. The computer device can set the target video classification model to obtain the classification probability of the video on multiple preset candidate labels when performing data processing. Furthermore, at least one target prediction label corresponding to the video can be determined from multiple preset candidate labels based on the classification probability. For example, all candidate labels with a classification probability greater than a preset threshold are used as target prediction labels, and a preset number of candidate labels with a classification probability greater than a preset threshold and a larger classification probability are used as target prediction labels. It can be understood that when training the target video classification model, the training label corresponding to the training video is at least one of the multiple candidate labels. For example, training video A includes training label 1 and training label 2, training video B includes training label 1 and training label 3, and training video C includes training label 4 and training label 5. The computer device can set the target video classification model to obtain the classification probability of the video on training labels 1 to training labels 5 when performing data processing.

[0227] The above-mentioned video classification method obtains multiple training video clips corresponding to the training video and training labels corresponding to the training video, inputs each training video clip into the target video classification model to be trained, performs three-dimensional feature extraction on each training video clip, obtains the motion features of each training video clip in the time dimension and the image features in the spatial dimension, constitutes the corresponding initial training feature information, assigns corresponding primary attention information to the initial training feature information corresponding to each training video clip, based on the classification influence corresponding to each training video clip, assigns corresponding second-order attention information to the initial training feature information corresponding to each training video clip, the second-order attention information is used to improve the attention intensity corresponding to the training video clip with non-critical classification influence, based on the primary attention information and the second-order attention information, the initial training feature information corresponding to each training video clip is fused to obtain the target training video feature, determines the target prediction label corresponding to the training video based on the target training video feature, adjusts the model parameters of the target video classification model based on the difference between the training label and the target prediction label until the convergence condition is met, and obtains the trained target video classification model. In this way, second-order attention information can improve the attention level corresponding to the video segments to be classified that are not critical to the classification. This means that the feature information of non-critical video segments can, to a certain extent, predict the feature information of keyframe video segments. Therefore, applying second-order attention information to model training can produce a target video classification model that can accurately classify both regular videos and videos with disturbed keyframes.

[0228] In one embodiment, three-dimensional feature extraction is performed on each training video segment to obtain motion features in the time dimension and image features in the spatial dimension of each training video segment, which form corresponding initial training feature information, including:

[0229] Through the three-dimensional convolutional network in the target video classification model, three-dimensional features are extracted from each training video clip to obtain the initial training feature information corresponding to each training video clip; the network parameters of the three-dimensional convolutional network are obtained by pre-training based on multiple training video clips corresponding to the training videos and the training labels corresponding to the training videos.

[0230] Specifically, the target video classification model includes a three-dimensional convolutional network, a preliminary attention network, a semi-soft attention network, and a fully connected network. The computer device can adjust the network parameters of all networks in the target video classification model based on the difference between the training labels and the target predicted labels until convergence conditions are met, thereby obtaining a trained target video classification model. However, to reduce the difficulty and complexity of training, the computer device can first train the network parameters of the three-dimensional convolutional network based on the training data to obtain relatively optimal network parameters for the three-dimensional convolutional network. Then, while maintaining the high-quality network parameters of the three-dimensional convolutional network, the network parameters of subsequent networks can be further trained based on the training data, ultimately obtaining a trained target video classification model. It can be understood that the three-dimensional convolutional network is used to extract three-dimensional features from each training video clip, obtaining initial training feature information corresponding to each training video clip. In other words, the three-dimensional convolutional network is used for feature extraction. The preliminary attention network, the semi-soft attention network, and the fully connected network are all used for feature processing. Networks with different functions can be trained separately. Compared with simultaneously training the network parameters of all networks, first training the network parameters of the three-dimensional convolutional network and then training the network parameters of other networks has lower hardware requirements for computer equipment and lower computational complexity, and can be applied to various types of computer equipment.

[0231] In one embodiment, pre-training of a 3D convolutional network includes the following steps:

[0232] Input each training video clip into the initial video classification model to be trained, the initial video classification model includes the three-dimensional convolutional network to be trained; perform three-dimensional feature extraction on each training video clip to obtain intermediate training feature information corresponding to each training video clip; obtain an initial prediction label corresponding to the training video based on each intermediate training feature information; adjust the model parameters of the initial video classification model based on the label difference between the initial prediction label and the training label until the convergence condition is met, thereby obtaining a trained initial video classification model; the trained initial video classification model includes the trained three-dimensional convolutional network.

[0233] Specifically, a computer device can establish an initial video classification model based on a three-dimensional convolutional network, and train the initial video classification model based on training data to obtain high-quality network parameters for the three-dimensional convolutional network. The initial video classification model can be a machine learning model consisting of a three-dimensional convolutional network and a classification layer. The computer device can input each training video clip into the initial video classification model to be trained. The three-dimensional convolutional network performs three-dimensional feature extraction on each training video clip to obtain intermediate training feature information corresponding to each training video clip. The classification layer uses the intermediate training feature information to obtain initial predicted labels corresponding to the training videos. The model parameters of the initial video classification model are adjusted based on the label difference between the initial predicted labels and the training labels until a convergence condition is met, thereby obtaining a trained initial video classification model. The convergence condition can be customized, such as when the number of iterations reaches an iteration threshold or when the difference between the training labels and the initial predicted labels reaches a minimum value. The model parameter adjustment can specifically include calculating the difference between the training labels and the initial predicted labels, adjusting the model parameters of the initial video classification model through backpropagation of the difference, and continuing training until the updated difference or the number of iterations meets the convergence condition, at which point the training is complete, resulting in a trained initial video classification model. When the initial video classification model training is completed, the computer device can obtain a three-dimensional convolutional network with high-quality network parameters. The computer device establishes a target video classification model to be trained based on the three-dimensional convolutional network, inputs multiple training video clips corresponding to the training video into the target video classification model, obtains the target prediction label corresponding to the training video, and adjusts other model parameters in the target video classification model except the three-dimensional convolutional network based on the difference between the training label and the target prediction label until the convergence condition is met, thereby obtaining the trained target video classification model.

[0234] refer to Figure 8 Specifically, the classification layer may be a softmax layer. The computer device may input each training video clip into the initial video classification model to be trained, and obtain the initial predicted label corresponding to the training video through data processing of the three-dimensional convolutional network and the softmax layer.

[0235] This application also provides an application scenario, which applies the above-mentioned video classification method. Specifically, the application of the video classification method in this application scenario is as follows:

[0236] Short video platforms are developing rapidly, with massive volumes of short videos produced and released every moment. As a result, these platforms are increasingly demonstrating their market value. Categorizing and tagging short videos on these platforms can optimize features like video recommendations and search.

[0237] refer to Figure 9The computer device extracts frames from the video to be classified according to the length of the video to obtain multiple video clips to be classified. The video clips to be classified are then input into the target video classification model. The three-dimensional convolutional network in the model is used to extract features to obtain initial video features to be classified, which are composed of the initial feature information to be classified corresponding to each video clip to be classified. The initial video features to be classified are then input into the hybrid attention module. The hybrid attention module allocates attention to the initial video features to obtain target attention information corresponding to each video clip to be classified. Based on the target attention information, the initial feature information to be classified corresponding to each video clip to be classified is fused to obtain target video features to be classified. Finally, the target video features to be classified are classified through the fully connected layer to obtain the video classification result (video label). The model outputs the video classification result.

[0238] 1. Extract frames from the classified video

[0239] The computer device extracts the first and last two frames of the video to be classified, and then extracts frames at equal intervals based on the total number of frames in the video to be classified, and obtains multiple video segments to be classified based on the extracted video frames. This can reduce computational complexity.

[0240] 2. 3D feature extraction through 3D convolutional network

[0241] The computer device simultaneously performs convolution operations on the two-dimensional pixel space and the third-dimensional spatiotemporal information of the video clips to be classified through a three-dimensional convolutional network, thereby obtaining the initial feature information to be classified corresponding to each video clip to be classified, forming the initial video features to be classified.

[0242] 3. Attention allocation through hybrid attention module

[0243] For each video, there are always some video clips that play a key role in determining which categories of information the video content contains. At the same time, there are also many video clips that are common to all categories of videos. Therefore, before classifying the initial video features to be classified, the proportion of unique information in each category of video can be increased, while reducing the impact of insignificant video clips (common to all categories) on the final classification results. Therefore, a hybrid attention module is specially designed. The hybrid attention module includes a soft attention network, a hard attention network, and a semi-soft attention network. Based on the attention information output by each attention network, the information of each initial feature to be classified is fused to obtain the target video features to be classified.

[0244] The soft attention network comprehensively considers the feature information of each video segment to be classified, obtains the first classification influence corresponding to each video segment to be classified, and forms a T-dimensional vector based on the first classification influence (where T represents the number of video segments obtained). The value of each vector represents the probability that the corresponding video segment affects the final classification result, and the sum of these vector values ​​is 1. The T-dimensional vector obtained based on the soft attention network is composed of the soft attention information corresponding to each video segment to be classified.

[0245] The hard attention network comprehensively considers the feature information of each video clip to be classified, obtains the second classification influence corresponding to each video clip to be classified, finds the video clip with the largest second classification influence among all the video clips to be classified, and sets the vector value of this video clip to 1. The vector values ​​of the remaining video clips to 0 are set to obtain another T-dimensional vector. The T-dimensional vector obtained by the soft attention network is composed of the hard attention information corresponding to each video clip to be classified.

[0246] Semi-soft attention network: Comprehensively consider the feature information of each video clip to be classified, and find out whether there are relatively critical video clips that have a certain degree of influence on the final classification results among the video clips to be classified that are ignored by the first two attention networks, that is, reduce the influence of the first two attention networks and ensure the robustness of the network. Comprehensively consider the feature information of each video clip to be classified, obtain the third classification influence corresponding to each video clip to be classified, set the vector value corresponding to each video clip to be classified with the top K third classification influence to 0, and then calculate the vector value based on the third classification influence in the remaining video clips to be classified, so that the sum of the vector values ​​of the remaining clips is 1, and obtain another T-dimensional vector. The T-dimensional vector obtained based on the semi-soft attention network is composed of the semi-soft attention information (i.e., second-order attention information) corresponding to each video to be classified.

[0247] 4. Classification through fully connected networks

[0248] The computer device can use the multi-layer perceptron as a fully connected network, input the target video features to be classified into the multi-layer perceptron, and output the video classification result.

[0249] In this embodiment, the image features and motion features in the video can be extracted through the three-dimensional convolutional network, and the accuracy of video classification can be improved by combining the image features and motion features to classify the video. Furthermore, based on the primary attention information, accurate classification of conventional videos can be achieved, and on this basis, combined with the second-order attention information, accurate classification of abnormal videos with key frames interfered with can be further achieved. The second-order attention information can improve the attention level corresponding to the video segments to be classified with non-critical classification influence, that is, improve the attention level corresponding to the non-critical video segments, and the feature information of the non-critical video segments can predict the feature information of the key frame video segments to a certain extent. Therefore, when classifying videos with key frames interfered with, the content of the video can also be identified based on the target video features to be classified that are fused with the feature information of the non-critical video segments. With the help of non-critical video segments, the accuracy of video classification of videos with key frames interfered with can be effectively improved.

[0250] It can be understood that the video classification method of the present application can be applied to the video classification of short videos with rich content, such as sports short videos and food short videos. It can classify videos with multiple labels and output hierarchical video labels from coarse-grained to fine-grained, thereby achieving a structured understanding of the video.

[0251] It should be understood that although Figure 2 、 5A The steps in the flowchart of FIG. 7 are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 、 5A At least part of the steps in 7 may include multiple steps or multiple stages. These steps or stages do not necessarily have to be performed at the same time, but can be performed at different times. The order of execution of these steps or stages is not necessarily one by one, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0252] In one embodiment, Figure 10 As shown, a video classification device is provided. The device can be a software module or a hardware module, or a combination of the two to form a part of a computer device. The device specifically includes: a video acquisition module 1002, a feature extraction module 1004, a primary attention allocation module 1006, a second-order attention allocation module 1008, an information fusion module 1010 and a video classification module 1012, wherein:

[0253] The video acquisition module 1002 is used to acquire multiple video segments to be classified corresponding to the video to be classified;

[0254] The feature extraction module 1004 is used to extract three-dimensional features from each video segment to be classified, obtain motion features in the time dimension and image features in the spatial dimension of each video segment to be classified, and form corresponding initial feature information to be classified;

[0255] A preliminary attention allocation module 1006 is configured to allocate preliminary attention information corresponding to the initial feature information to be classified corresponding to each video segment to be classified, wherein the preliminary attention information is used to represent the attention intensity corresponding to the video segment to be classified;

[0256] The second-order attention allocation module 1008 is configured to allocate corresponding second-order attention information to the initial feature information to be classified corresponding to each video segment to be classified based on the classification influence corresponding to each video segment to be classified. The second-order attention information is used to increase the attention intensity corresponding to the video segment to be classified with non-critical classification influence;

[0257] An information fusion module 1010 is configured to fuse the initial feature information to be classified corresponding to each video segment to be classified based on the primary attention information and the second-order attention information to obtain the target video feature to be classified;

[0258] The video classification module 1012 is configured to determine a video classification result corresponding to the video to be classified based on the features of the target video to be classified.

[0259] In one embodiment, the video acquisition module is also used to filter the head video frame and the tail video frame from the video to be classified to obtain a candidate video, perform equally spaced video frame extraction on each candidate video frame in the candidate video to obtain multiple intermediate video frames, sort the head video frame, the tail video frame and each intermediate video frame according to the time information corresponding to the video frame to obtain a target video frame sequence, and segment the target video frame sequence to obtain multiple video clips to be classified.

[0260] In one embodiment, the feature extraction module is also used to form corresponding three-dimensional video image blocks based on each video segment to be classified. The three dimensions of the three-dimensional video image block are the length, width and number of video frames of the current video segment to be classified. Through a three-dimensional convolutional network composed of a three-dimensional convolutional layer and a three-dimensional pooling layer, the motion information between temporally adjacent pixels in the three-dimensional video image block is extracted to obtain motion features, and the image information between spatially adjacent pixels in the three-dimensional video image block is extracted to obtain image features to form the corresponding initial feature information to be classified.

[0261] In one embodiment, the primary attention information includes soft attention information, and the primary attention allocation module is also used to calculate the first classification influence corresponding to each initial feature information to be classified, normalize each first classification influence, and obtain the soft attention information corresponding to each video clip to be classified.

[0262] In one embodiment, the primary attention information includes hard attention information, and the primary attention allocation module is also used to calculate the second classification influence corresponding to each initial feature information to be classified, and divide each video clip to be classified into a first category video clip and a second category video clip based on the second classification influence. The second classification influence corresponding to the first category video clip is greater than the second classification influence corresponding to the second category video clip. The attention intensity corresponding to the first category video clip is determined as a first preset value, and the attention intensity corresponding to the second category video clip is determined as a second preset value. The first preset value is greater than the second preset value. The attention intensity corresponding to each video clip to be classified is normalized to obtain the hard attention information corresponding to each video clip to be classified.

[0263] In one embodiment, the second-order attention allocation module is also used to calculate the third classification influence corresponding to each initial feature information to be classified, and divide each video clip to be classified into a third category video clip and a fourth category video clip based on the third classification influence. The third classification influence corresponding to the third category video clip is greater than the third classification influence corresponding to the fourth category video clip. The third classification influence corresponding to the third category video clip is a key classification influence, and the third classification influence corresponding to the fourth category video clip is a non-key classification influence. The attention intensity corresponding to the third category video clip is determined as a second preset value, and the third classification influence corresponding to the fourth category video clip is normalized to obtain the attention intensity corresponding to the fourth category video clip. Based on the attention intensity corresponding to each video clip to be classified, the second-order attention information corresponding to each video clip to be classified is obtained.

[0264] In one embodiment, the information fusion module is also used to fuse the primary attention information and the second-order attention information corresponding to the same initial feature information to be classified to obtain the target attention information corresponding to each initial feature information to be classified, perform weighted summation on the initial feature information to be classified and the target attention information corresponding to the same video clip to be classified to obtain the intermediate feature information to be classified corresponding to each video clip to be classified, and obtain the target video feature to be classified based on each intermediate feature information to be classified.

[0265] In one embodiment, the video classification result includes a motion label. The video classification module is also used to perform object recognition on the video to be classified based on the image features in the target video to be classified, to obtain multiple target objects in the video to be classified, to perform global trajectory recognition on the target object based on the motion features in the target video to be classified, to obtain the global motion trajectory corresponding to each target object, to determine the interaction information between each target object based on the global motion trajectory corresponding to each target object, and to obtain the motion label corresponding to the video to be classified based on the interaction information between each target object and the object type of the target object.

[0266] In one embodiment, the video classification module is further configured to generate interaction information between corresponding target objects based on the intersecting global motion trajectories and the motion directions of the global motion trajectories before and after the intersection.

[0267] In one embodiment, the video classification results include emotional labels. The video classification module is also used to perform object recognition on the video to be classified based on the image features in the target video to be classified, obtain multiple target objects in the video to be classified, perform key point trajectory recognition on the target object based on the motion features in the target video to be classified, determine the local motion trajectory corresponding to the key points of each target object, determine the state change information of the corresponding target object based on the local motion trajectory, and obtain the emotional label corresponding to the video to be classified based on the state change information of each target object and the object type of the target object.

[0268] In one embodiment, the video classification result is a video tag set, which includes hierarchical video tags from coarse-grained to fine-grained.

[0269] In one embodiment, the video acquisition module is further configured to input each video segment to be classified into a trained target video classification model. The feature extraction module is further configured to perform three-dimensional feature extraction on each video segment to be classified using the three-dimensional convolutional network of the target video classification model, obtaining motion features in the temporal dimension and image features in the spatial dimension for each video segment to be classified, thereby forming corresponding initial feature information to be classified. The primary attention allocation module is further configured to allocate corresponding primary attention information to the initial feature information to be classified corresponding to each video segment to be classified using the primary attention network of the target video classification model, wherein the primary attention network includes at least one of a soft attention network and a hard attention network. The second-order attention allocation module is further configured to allocate corresponding second-order attention information to the initial feature information to be classified corresponding to each video segment to be classified based on the classification influence corresponding to each video segment to be classified using the semi-soft attention network of the target video classification model. The information fusion module is further configured to fuse the initial feature information to be classified corresponding to each video segment to be classified based on the primary attention information and the second-order attention information, thereby obtaining target video features to be classified. The video classification module is also used to determine the video classification result corresponding to the video to be classified based on the target video features through the fully connected network of the target video classification model.

[0270] In one embodiment, the current attention network is a primary-order attention network or a second-order attention network. The primary-order attention allocation module and the second-order attention allocation module are further configured to perform primary-order fully connected processing on each initial feature information to be classified to obtain an initial classification influence corresponding to each video segment to be classified, perform nonlinear processing on each initial classification influence to obtain an intermediate classification influence corresponding to each video segment to be classified, perform advanced fully connected processing on each intermediate classification influence to obtain a current classification influence corresponding to each video segment to be classified, and based on the current classification influence corresponding to each video segment to be classified, allocate corresponding current-order attention information to the initial feature information to be classified corresponding to each video segment to be classified.

[0271] In the aforementioned video classification device, second-order attention information can improve the attention level corresponding to the video segments to be classified that have non-critical classification influences. Specifically, it can improve the attention level corresponding to non-critical video segments. The feature information of non-critical video segments can, to a certain extent, predict the feature information of keyframe video segments. Therefore, when classifying videos with keyframe interference, the content of the video can also be identified based on the features of the target video to be classified that are fused with the feature information of non-critical video segments. Using non-critical video segments can effectively improve the accuracy of video classification for videos with keyframe interference.

[0272] In one embodiment, Figure 11As shown, a video classification device is provided. The device can be a software module or a hardware module, or a combination of the two to form a part of a computer device. The device specifically includes: a video acquisition module 1102, a feature extraction module 1104, a primary attention allocation module 1106, a secondary attention allocation module 1108, an information fusion module 1110 and a parameter adjustment module 1112, wherein:

[0273] The video acquisition module 1102 is used to obtain multiple training video segments and training labels corresponding to the training videos, and input each training video segment into the target video classification model to be trained;

[0274] The feature extraction module 1104 is used to extract three-dimensional features from each training video segment to obtain motion features in the time dimension and image features in the spatial dimension of each training video segment to form corresponding initial training feature information;

[0275] A primary attention allocation module 1106 is configured to allocate corresponding primary attention information to the initial training feature information corresponding to each training video segment, where the primary attention information is used to represent the attention intensity corresponding to the training video segment;

[0276] A second-order attention allocation module 1108 is configured to allocate corresponding second-order attention information to the initial training feature information corresponding to each training video segment based on the classification influence corresponding to each training video segment. The second-order attention information is used to increase the attention intensity corresponding to the training video segment with non-critical classification influence;

[0277] An information fusion module 1110 is configured to fuse the initial training feature information corresponding to each training video segment based on the primary attention information and the second-order attention information to obtain target training video features;

[0278] The parameter adjustment module 1112 is used to determine the target prediction label corresponding to the training video based on the target training video features, and adjust the model parameters of the target video classification model based on the difference between the training label and the target prediction label until the convergence condition is met to obtain the trained target video classification model.

[0279] In one embodiment, the feature extraction module is also used to perform three-dimensional feature extraction on each training video clip through the three-dimensional convolutional network in the target video classification model to obtain initial training feature information corresponding to each training video clip. The network parameters of the three-dimensional convolutional network are pre-trained based on multiple training video clips corresponding to the training video and training labels corresponding to the training video.

[0280] In one embodiment, Figure 12 As shown, the device also includes:

[0281] The pre-training module 1101 is used to input each training video clip into the initial video classification model to be trained, the initial video classification model includes the three-dimensional convolutional network to be trained, perform three-dimensional feature extraction on each training video clip, obtain intermediate training feature information corresponding to each training video clip, obtain initial prediction labels corresponding to the training videos based on each intermediate training feature information, adjust model parameters of the initial video classification model based on the label difference between the initial prediction labels and the training labels until the convergence conditions are met, and obtain a trained initial video classification model, which includes the trained three-dimensional convolutional network.

[0282] In the aforementioned video classification device, second-order attention information can enhance the attention level corresponding to non-critical video segments to be classified. Specifically, it can enhance the attention level corresponding to non-critical video segments. The feature information of non-critical video segments can, to a certain extent, predict the feature information of keyframe video segments. Therefore, applying second-order attention information to model training can produce a target video classification model that can accurately classify both regular videos and videos with disrupted keyframes.

[0283] For the specific definition of the video classification device, please refer to the definition of the video classification method above, which will not be repeated here. The various modules in the above-mentioned video classification device can be implemented in whole or in part by software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0284] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 13 As shown. The computer device includes a processor, a memory and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as target video classification models, video classification results, and training videos. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a video classification method is implemented.

[0285] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 14As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a video classification method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0286] Those skilled in the art will understand that Figure 13 、 14 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0287] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0288] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0289] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of each of the above-described method embodiments.

[0290] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0291] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0292] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A video classification method, characterized in that the method comprises: Obtain multiple video clips to be classified corresponding to the video to be classified; Through the 3D convolutional network in the target video classification model, 3D features are extracted from each video clip to be classified, and the motion features in the time dimension and the image features in the spatial dimension of each video clip to be classified are obtained to form the corresponding initial feature information to be classified; Assigning corresponding primary attention information to the initial feature information to be classified corresponding to each video clip to be classified, the primary attention information being used to characterize the attention intensity corresponding to the video clip to be classified, the primary attention information including hard attention information, including: calculating the second classification influence corresponding to each initial feature information to be classified through the hard attention network in the target video classification model, dividing each video clip to be classified into a first category of video clips and a second category of video clips based on the second classification influence, the second classification influence corresponding to the first category of video clips being greater than the second classification influence corresponding to the second category of video clips, determining the attention intensity corresponding to the first category of video clips as a first preset value, determining the attention intensity corresponding to the second category of video clips as a second preset value, and obtaining the hard attention information corresponding to each video clip to be classified based on the attention intensity corresponding to each video clip to be classified, the first preset value being greater than the second preset value; Based on the classification influence corresponding to each of the video clips to be classified, corresponding second-order attention information is allocated to the initial feature information to be classified corresponding to each of the video clips to be classified, and the second-order attention information is used to improve the attention intensity corresponding to the video clips to be classified with non-critical classification influence, including: calculating the third classification influence corresponding to each of the initial feature information to be classified through the semi-soft attention network in the target video classification model, dividing each of the video clips to be classified into a third category video clip and a fourth category video clip based on the third classification influence, the third classification influence corresponding to the third category video clip is greater than the third classification influence corresponding to the fourth category video clip, the third classification influence corresponding to the third category video clip is a critical classification influence, and the third classification influence corresponding to the fourth category video clip is a non-critical classification influence, determining the attention intensity corresponding to the third category video clip as a second preset value, normalizing the third classification influence corresponding to the fourth category video clip to obtain the attention intensity corresponding to the fourth category video clip, and obtaining the second-order attention information corresponding to each of the video clips to be classified based on the attention intensity corresponding to each of the video clips to be classified; Based on the primary attention information and the second-order attention information, the initial feature information to be classified corresponding to each video clip to be classified is fused to obtain the target video feature to be classified; Determining a video classification result corresponding to the video to be classified based on the target video features through a fully connected network in the target video classification model; First, the network parameters of the three-dimensional convolutional network are trained, and then the network parameters of the three-dimensional convolutional network are fixed, and the network parameters of other networks are trained.

2. The method according to claim 1, wherein the preliminary attention information includes soft attention information, and the assigning of preliminary attention information corresponding to the initial feature information to be classified corresponding to each video segment to be classified comprises: Calculate the first classification influence corresponding to each initial feature information to be classified; The influence of each first classification is normalized to obtain the soft attention information corresponding to each video clip to be classified.

3. The method according to claim 1, wherein obtaining hard attention information corresponding to each video segment to be classified based on the attention intensity corresponding to each video segment to be classified comprises: The attention intensity corresponding to each video segment to be classified is normalized to obtain the hard attention information corresponding to each video segment to be classified.

4. The method according to claim 1, wherein obtaining a plurality of video segments to be classified corresponding to the video to be classified comprises: Filter the head video frame and the tail video frame from the video to be classified to obtain a candidate video; Extracting video frames at equal intervals from each candidate video frame in the candidate video to obtain a plurality of intermediate video frames; Sort the head video frame, the tail video frame, and each intermediate video frame according to time information corresponding to the video frame to obtain a target video frame sequence; The target video frame sequence is segmented to obtain a plurality of video segments to be classified.

5. The method according to claim 1, wherein the initial feature information to be classified corresponding to each video segment to be classified is fused based on the primary attention information and the secondary attention information to obtain the target video feature to be classified, comprising: The primary attention information and the second-order attention information corresponding to the same initial feature information to be classified are fused to obtain the target attention information corresponding to each initial feature information to be classified; Perform weighted summation on the initial feature information to be classified and the target attention information corresponding to the same video clip to be classified, and obtain the intermediate feature information to be classified corresponding to each video clip to be classified; The target video feature to be classified is obtained based on each intermediate feature information to be classified.

6. The method according to claim 1, wherein the video classification result includes a motion tag, and determining the video classification result corresponding to the video to be classified based on the target video features includes: Performing object recognition on the video to be classified based on image features in the target video to be classified to obtain multiple target objects in the video to be classified; Performing global trajectory recognition on the target object based on the motion features in the target to be classified video features to obtain the global motion trajectory corresponding to each target object; Determine interaction information between target objects based on the global motion trajectories corresponding to the target objects; A motion label corresponding to the video to be classified is obtained based on the interaction information between the target objects and the object type of the target objects.

7. The method according to claim 1, wherein the video classification result includes an emotion tag, and determining the video classification result corresponding to the target video to be classified based on the target video features comprises: Performing object recognition on the video to be classified based on image features in the target video to be classified to obtain multiple target objects in the video to be classified; Identify key point trajectories of target objects based on motion features in the target video features to be classified, and determine local motion trajectories corresponding to key points of each target object; Determine the state change information of the corresponding target object based on the local motion trajectory; The emotion label corresponding to the video to be classified is obtained based on the state change information of each target object and the object type of the target object.

8. The method according to claim 1, wherein the video classification result is a video tag set, and the video tag set includes hierarchical video tags from coarse granularity to fine granularity.

9. The method according to claim 1, wherein the step of extracting three-dimensional features from each video segment to be classified to obtain motion features in the time dimension and image features in the spatial dimension of each video segment to be classified, and forming corresponding initial feature information to be classified, comprises: forming a corresponding three-dimensional video image block based on each video segment to be classified, wherein the three dimensions of the three-dimensional video image block are respectively the length and width of the video frame and the number of video frames of the current video segment to be classified; Through a three-dimensional convolutional network composed of a three-dimensional convolutional layer and a three-dimensional pooling layer, motion information between temporally adjacent pixel points in the three-dimensional video image block is extracted to obtain motion features, and image information between spatially adjacent pixel points in the three-dimensional video image block is extracted to obtain image features to form corresponding initial feature information to be classified.

10. The method according to claim 6, wherein determining the interaction information between the target objects based on the global motion trajectories corresponding to the target objects comprises: Based on the intersecting global motion trajectories and the motion directions of the global motion trajectories before and after the intersection, the interaction information between the corresponding target objects is generated.

11. The method according to any one of claims 1 to 10, further comprising: Through the primary attention network of the target video classification model, corresponding primary attention information is assigned to the initial feature information to be classified corresponding to each video clip to be classified, and the primary attention network includes at least one of a soft attention network and a hard attention network.

12. The method according to claim 11, wherein the current attention network is the primary-order attention network or the second-order attention network, and the data processing of the input data by the current attention network comprises the following steps: Perform preliminary full-connection processing on each initial feature information to be classified to obtain the initial classification influence corresponding to each video clip to be classified; Performing nonlinear processing on each initial classification influence to obtain an intermediate classification influence corresponding to each video segment to be classified; Performing advanced full-connection processing on each intermediate classification influence to obtain the current classification influence corresponding to each video clip to be classified; Based on the current classification influence corresponding to each of the video segments to be classified, the corresponding current-order attention information is allocated to the initial feature information to be classified corresponding to each of the video segments to be classified.

13. A video classification method, characterized in that the method comprises: Obtain multiple training video clips and training labels corresponding to the training videos, and input each training video clip into the target video classification model to be trained; Performing 3D feature extraction on each training video segment through the 3D convolutional network in the target video classification model to obtain motion features in the time dimension and image features in the spatial dimension of each training video segment to form corresponding initial training feature information; Assigning corresponding primary attention information to the initial training feature information corresponding to each training video clip, the primary attention information is used to characterize the attention intensity corresponding to the training video clip, the primary attention information including hard attention information, including: calculating the second classification influence corresponding to each initial training feature information through the hard attention network in the target video classification model, dividing each training video clip into a first category of video clips and a second category of video clips based on the second classification influence, the second classification influence corresponding to the first category of video clips is greater than the second classification influence corresponding to the second category of video clips, determining the attention intensity corresponding to the first category of video clips as a first preset value, determining the attention intensity corresponding to the second category of video clips as a second preset value, and obtaining the hard attention information corresponding to each training video clip based on the attention intensity corresponding to each video clip to be classified, the first preset value being greater than the second preset value; Based on the classification influence corresponding to each training video clip, corresponding second-order attention information is allocated to the initial training feature information corresponding to each training video clip, and the second-order attention information is used to improve the attention intensity corresponding to the training video clip with non-critical classification influence, including: calculating the third classification influence corresponding to each initial training feature information through the semi-soft attention network in the target video classification model, dividing each training video clip into a third category video clip and a fourth category video clip based on the third classification influence, the third classification influence corresponding to the third category video clip is greater than the third classification influence corresponding to the fourth category video clip, the third classification influence corresponding to the third category video clip is a critical classification influence, and the third classification influence corresponding to the fourth category video clip is a non-critical classification influence, determining the attention intensity corresponding to the third category video clip as a second preset value, normalizing the third classification influence corresponding to the fourth category video clip to obtain the attention intensity corresponding to the fourth category video clip, and obtaining the second-order attention information corresponding to each training video clip based on the attention intensity corresponding to each video clip to be classified; Based on the primary attention information and the second-order attention information, the initial training feature information corresponding to each training video clip is fused to obtain the target training video feature; Determining a target prediction label corresponding to the training video based on the target training video features through a fully connected network in the target video classification model; Based on the difference between the training label and the target prediction label, the network parameters of the hard attention network, the semi-soft attention network and the fully connected network in the target video classification model are adjusted until the convergence conditions are met, thereby obtaining a trained target video classification model; the network parameters of the three-dimensional convolutional network are obtained by training a model composed of a three-dimensional convolutional network and a classification layer.

14. The method according to claim 13 is characterized in that the network parameters of the three-dimensional convolutional network are obtained by pre-training a model composed of a three-dimensional convolutional network and a classification layer based on multiple training video clips corresponding to the training video and training labels corresponding to the training video.

15. The method according to claim 14, wherein the pre-training of the three-dimensional convolutional network comprises the following steps: Inputting each training video clip into an initial video classification model to be trained, wherein the initial video classification model includes a three-dimensional convolutional network to be trained; Performing three-dimensional feature extraction on each of the training video clips to obtain intermediate training feature information corresponding to each of the training video clips; Obtaining an initial prediction label corresponding to the training video based on each intermediate training feature information; Based on the label difference between the initial predicted label and the training label, the model parameters of the initial video classification model are adjusted until the convergence condition is met, thereby obtaining a trained initial video classification model; the trained initial video classification model includes a trained three-dimensional convolutional network.

16. A video classification device, characterized in that the device comprises: A video acquisition module is used to acquire multiple video clips to be classified corresponding to the video to be classified; A feature extraction module is used to extract three-dimensional features from each video segment to be classified through the three-dimensional convolutional network in the target video classification model, obtain the motion features of each video segment to be classified in the time dimension and the image features in the spatial dimension, and form the corresponding initial feature information to be classified; a primary attention allocation module, configured to allocate corresponding primary attention information to initial feature information to be classified corresponding to each video clip to be classified, the primary attention information being used to characterize the attention intensity corresponding to the video clip to be classified, the primary attention information including hard attention information, including: calculating the second classification influence corresponding to each initial feature information to be classified through the hard attention network in the target video classification model, dividing each video clip to be classified into a first category of video clips and a second category of video clips based on the second classification influence, the second classification influence corresponding to the first category of video clips being greater than the second classification influence corresponding to the second category of video clips, determining the attention intensity corresponding to the first category of video clips as a first preset value, determining the attention intensity corresponding to the second category of video clips as a second preset value, and obtaining the hard attention information corresponding to each video clip to be classified based on the attention intensity corresponding to each video clip to be classified, the first preset value being greater than the second preset value; A second-order attention allocation module is used to allocate corresponding second-order attention information to the initial feature information to be classified corresponding to each video clip to be classified based on the classification influence corresponding to each video clip to be classified, and the second-order attention information is used to improve the attention intensity corresponding to the video clip to be classified with non-critical classification influence, including: calculating the third classification influence corresponding to each initial feature information to be classified through the semi-soft attention network in the target video classification model, dividing each video clip to be classified into a third category video clip and a fourth category video clip based on the third classification influence, the third classification influence corresponding to the third category video clip is greater than the third classification influence corresponding to the fourth category video clip, the third classification influence corresponding to the third category video clip is a critical classification influence, and the third classification influence corresponding to the fourth category video clip is a non-critical classification influence, determining the attention intensity corresponding to the third category video clip as a second preset value, normalizing the third classification influence corresponding to the fourth category video clip to obtain the attention intensity corresponding to the fourth category video clip, and obtaining the second-order attention information corresponding to each video clip to be classified based on the attention intensity corresponding to each video clip to be classified; An information fusion module is used to fuse the initial feature information to be classified corresponding to each video segment to be classified based on the primary attention information and the second-order attention information to obtain the target video feature to be classified; A video classification module is configured to determine a video classification result corresponding to the video to be classified based on the features of the target video to be classified through a fully connected network in the target video classification model; First, the network parameters of the three-dimensional convolutional network are trained, and then the network parameters of the three-dimensional convolutional network are fixed, and the network parameters of other networks are trained.

17. A video classification device, characterized in that the device comprises: A video acquisition module is used to obtain multiple training video clips and training labels corresponding to the training video, and input each training video clip into the target video classification model to be trained; A feature extraction module is used to extract three-dimensional features from each training video segment through the three-dimensional convolutional network in the target video classification model, thereby obtaining motion features in the time dimension and image features in the spatial dimension of each training video segment to form corresponding initial training feature information; a primary attention allocation module, configured to allocate corresponding primary attention information to initial training feature information corresponding to each training video clip, wherein the primary attention information is used to characterize the attention intensity corresponding to the training video clip, and the primary attention information includes hard attention information, including: calculating the second classification influence corresponding to each initial training feature information through the hard attention network in the target video classification model, dividing each training video clip into a first category of video clips and a second category of video clips based on the second classification influence, the second classification influence corresponding to the first category of video clips being greater than the second classification influence corresponding to the second category of video clips, determining the attention intensity corresponding to the first category of video clips as a first preset value, determining the attention intensity corresponding to the second category of video clips as a second preset value, and obtaining the hard attention information corresponding to each training video clip based on the attention intensity corresponding to each video clip to be classified, wherein the first preset value is greater than the second preset value; A second-order attention allocation module is used to allocate corresponding second-order attention information to the initial training feature information corresponding to each training video clip based on the classification influence corresponding to each training video clip, and the second-order attention information is used to improve the attention intensity corresponding to the training video clips with non-critical classification influence, including: calculating the third classification influence corresponding to each initial training feature information through the semi-soft attention network in the target video classification model, dividing each training video clip into a third category video clip and a fourth category video clip based on the third classification influence, the third classification influence corresponding to the third category video clip is greater than the third classification influence corresponding to the fourth category video clip, the third classification influence corresponding to the third category video clip is a critical classification influence, and the third classification influence corresponding to the fourth category video clip is a non-critical classification influence, determining the attention intensity corresponding to the third category video clip as a second preset value, normalizing the third classification influence corresponding to the fourth category video clip to obtain the attention intensity corresponding to the fourth category video clip, and obtaining the second-order attention information corresponding to each training video clip based on the attention intensity corresponding to each video clip to be classified; An information fusion module is used to fuse the initial training feature information corresponding to each training video clip based on the primary attention information and the second-order attention information to obtain the target training video features; A parameter adjustment module is used to determine the target prediction label corresponding to the training video based on the target training video features through the fully connected network in the target video classification model; based on the difference between the training label and the target prediction label, adjust the network parameters of the hard attention network, the semi-soft attention network and the fully connected network in the target video classification model until the convergence conditions are met, thereby obtaining a trained target video classification model; the network parameters of the three-dimensional convolutional network are obtained by training a model composed of a three-dimensional convolutional network and a classification layer.

18. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein the processor implements the steps of the method according to any one of claims 1 to 15 when executing the computer program.

19. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

20. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

Images