Video classification method, device, equipment and medium

Abstract

The present application discloses a video classification method, device, equipment and medium, and relates to the field of machine learning. The method includes: obtaining n multimodal features of a target video, wherein the n multimodal features include at least two of image features, audio features and text features, and n is a positive integer greater than 1; fusing the n multimodal features to obtain fused features; classifying the fused features according to m classification granularities to obtain overall classification features and m granularity classification features of the target video, wherein the m classification granularities are used to represent m interrelated granularities for classification under the target dimension, and the m granularity classification features are used to represent classification features corresponding to the m classification granularities; according to the overall classification features and the m granularity classification features, an m-level classification label of the target video is obtained, and the m-level classification label is a video label arranged according to the m classification granularities. The present application can fuse multimodal information to make video labels more accurate.

Description

Technical Field

[0001] The present application relates to the field of machine learning, and in particular to a video classification method, apparatus, device and medium. Background Art

[0002] After the video is uploaded to the video platform, a category label will be added to the video. For example, the category label of a video is technology-smartphone-domestic mobile phone.

[0003] Related technologies pre-train a machine learning model for video classification. Image features are extracted from the target video and input into the machine learning model, which processes the image features and outputs the classification label of the target video.

[0004] However, the relevant technology uses information related to image features in the target video, and the final output classification labels are inaccurate. Summary of the invention

[0005] The embodiment of the present application provides a video classification method, apparatus, device and medium. The method can combine multimodal features in the target video to determine the classification label of the target video from multiple modalities. The technical solution is as follows:

[0006] According to one aspect of the present application, a video classification method is provided, the method comprising:

[0007] Obtaining n multimodal features of a target video, wherein the n multimodal features include at least two of image features, audio features, and text features, and n is a positive integer greater than 1;

[0008] Fusing the n multimodal features to obtain a fused feature;

[0009] Classifying the fusion features according to m classification granularities to obtain an overall classification feature of the target video and m granularity classification features, wherein the m classification granularities are used to represent m interrelated granularities for classification under a target dimension, and the m granularity classification features are used to represent classification features corresponding to the m classification granularities;

[0010] According to the overall classification features and the m granularity classification features, m-level classification labels of the target video are obtained, and the m-level classification labels are video labels arranged according to the m classification granularities.

[0011] According to another aspect of the present application, a video classification device is provided, the device comprising:

[0012] A feature extraction module, used to obtain n multimodal features of a target video, wherein the n multimodal features include at least two of image features, audio features, and text features, and n is a positive integer greater than 1;

[0013] A fusion module, used for fusing the n multimodal features to obtain a fusion feature;

[0014] A classification module, used to classify the fusion features according to m classification granularities to obtain the overall classification features and m granularity classification features of the target video, wherein the m classification granularities are used to represent m interrelated granularities for classification under the target dimension, and the m granularity classification features are used to represent classification features corresponding to the m classification granularities, where m is a positive integer;

[0015] The classification module is further used to obtain m-level classification labels of the target video according to the overall classification features and the m granularity classification features, and the m-level classification labels are video labels arranged according to the m classification granularities.

[0016] According to one aspect of the present application, a training method for a video classification model is provided, the video classification model comprising a feature extraction network layer, a feature fusion network layer and a classification network layer, the method comprising:

[0017] Obtaining a sample training set, wherein the sample training set includes a sample video and a real annotation corresponding to the sample video;

[0018] Calling the feature extraction network layer to perform data processing on the sample video and outputting n sample multimodal features, wherein the n sample multimodal features include at least two of sample image features, sample audio features, and sample text features, where n is a positive integer greater than 1;

[0019] Calling the feature fusion network layer to fuse the n sample multimodal features and output sample fusion features;

[0020] Calling the classification network layer to perform data processing on the n sample multimodal features and sample fusion features, respectively, and outputting n sample granularity classification labels and sample fusion classification labels, respectively;

[0021] Calculating the cross entropy between the n sample granularity classification labels and the true annotations respectively to obtain n granularity cross entropies; calculating the cross entropy between the sample fusion classification labels and the true annotations to obtain fusion cross entropy;

[0022] The video classification model is trained according to the n granularity cross entropies and the fusion cross entropy.

[0023] According to one aspect of the present application, a training device for a video classification model is provided, wherein the video classification model includes a feature extraction network layer, a feature fusion network layer, and a classification network layer, and the device includes:

[0024] A sample acquisition module, used to acquire a sample training set, wherein the sample training set includes a sample video and a real annotation corresponding to the sample video;

[0025] A sample feature extraction module, used to call the feature extraction network layer, perform data processing on the sample video, and output n sample multimodal features, wherein the n sample multimodal features include at least two of sample image features, sample audio features, and sample text features, where n is a positive integer greater than 1;

[0026] A sample fusion module, used to call the feature fusion network layer, perform fusion processing on the n sample multimodal features, and output sample fusion features;

[0027] A sample classification module, used to call the classification network layer, perform data processing on the n sample multimodal features and sample fusion features, and output n sample granularity classification labels and sample fusion classification labels respectively;

[0028] The training module is used to respectively calculate the cross entropy between the n sample granularity classification labels and the true annotations to obtain n granularity cross entropies; calculate the cross entropy between the sample fusion classification labels and the true annotations to obtain fusion cross entropy;

[0029] The training module is also used to train the video classification model according to the n granularity cross entropies and the fusion cross entropy.

[0030] According to another aspect of the present application, a computer device is provided, which includes: a processor and a memory, wherein the memory stores at least one program loaded and executed by the processor to implement the video classification method described above, or the video classification model training method.

[0031] According to another aspect of the present application, a computer storage medium is provided, in which at least one program code is stored. The program code is loaded and executed by a processor to implement the video classification method described above, or the video classification model training method.

[0032] According to another aspect of the present application, a computer program product or a computer program is provided, wherein the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the video classification method described in the above aspect, or the training method of a video classification model.

[0033] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0034] This method fuses the multimodal features of the target video to obtain fused features, and determines the classification label of the target video based on the fused features. This method can model and mine finer granular information through multimodal features and deepen the understanding and extraction of video content. When classifying fused features, it makes full use of the dependencies and constraints between different modalities from a holistic perspective to improve the accuracy of video classification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0036] Figure 1 is a schematic diagram of the structure of a computer system provided by an exemplary embodiment of the present application;

[0037] Figure 2 is a schematic diagram of the structure of a computer system provided by an exemplary embodiment of the present application;

[0038] Figure 3 is a schematic diagram of a video classification model provided by an exemplary embodiment of the present application;

[0039] Figure 4 is a schematic diagram of a video classification model provided by an exemplary embodiment of the present application;

[0040] Figure 5 is a flowchart of a video classification method provided by an exemplary embodiment of the present application;

[0041] Figure 6 is a flowchart of a video classification method provided by an exemplary embodiment of the present application;

[0042] Figure 7 is a structural diagram of a classification module provided by an exemplary embodiment of the present application;

[0043] Figure 8 is a flowchart of a video classification method provided by an exemplary embodiment of the present application;

[0044] Fig. 9 is a schematic diagram of the structure of a fusion module provided by an exemplary embodiment of the present application;

[0045] Fig.10 is a flowchart of a feature fusion method provided by an exemplary embodiment of the present application;

[0046] Fig.11 is a flowchart of a feature fusion method provided by an exemplary embodiment of the present application;

[0047] Fig.12 is a flowchart of a method for training a video classification model provided by an exemplary embodiment of the present application;

[0048] Fig.13 is a schematic diagram of calculating relative entropy provided by an exemplary embodiment of the present application;

[0049] Fig.14 is a flowchart of a method for training a video classification model provided by an exemplary embodiment of the present application;

[0050] Fig.15 is a flowchart of a video recommendation method provided by an exemplary embodiment of the present application;

[0051] Fig.16 is a flowchart of a video recommendation method provided by an exemplary embodiment of the present application;

[0052] Fig.17 is a structural schematic diagram of a video classification device provided by an exemplary embodiment of the present application;

[0053] Fig.18 It is a structural schematic diagram of a training device for a video classification model provided by an exemplary embodiment of the present application;

[0054] Fig.19 It is a structural diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0056] First, the nouns involved in the embodiments of the present application are introduced:

[0057] Artificial Intelligence (AI): It is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0058] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0059] Computer Vision (CV): Computer vision is a science that studies how to make machines "see". To put it more specifically, it means using cameras and computers to replace human eyes, so that computers can process images that are more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0060] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0061] Machine Learning (ML): It is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0062] Relative Entropy: Also known as Kullback-Leibler (KL) divergence or information divergence, it is an asymmetric measure of the difference between two probability distributions. Relative entropy is equivalent to the difference between the information entropy (Shannon entropy) of two probability distributions.

[0063] Cross Entropy: It is used to measure the difference between two probability distributions. Cross entropy can be used as a loss function in machine learning. p represents the distribution of true labels, and q is the distribution of predicted labels of the trained model. Cross entropy can measure the similarity between p and q.

[0064] Mel-spectrogram: A spectrum obtained by converting the frequency scale (Hz, Hertz) of a normal spectrum into a Mel-frequency scale. The mapping relationship is as follows: Mel-frequency scale mel(f) = 2595*log 10 (1+f / 700), where f represents the frequency scale in Hertz.

[0065] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0066] Figure 1 The schematic diagram of the structure of a computer system provided by an exemplary embodiment of the present application is shown. The computer system 100 includes: a terminal 120 and a server 140 .

[0067] An application program related to video playback is installed on the terminal 120. The application program may be a small program in an app (application), a dedicated application program, or a web client. The terminal 120 includes at least one of a content provider and a content consumer, wherein the content provider refers to a terminal that provides videos, and the content consumer refers to a terminal that plays videos. Exemplarily, a user plays a video on the terminal 120. The terminal 120 is at least one of a smart phone, a tablet computer, an e-book reader, an MP3 player, an MP4 player, a laptop computer, and a desktop computer.

[0068] The terminal 120 is connected to the server 140 via a wireless network or a wired network.

[0069] The server 140 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The server 140 is used to provide background services for video playback applications and send videos and video-related information to the terminal 120. Optionally, the server 140 undertakes the main computing work and the terminal 120 undertakes the secondary computing work; or, the server 140 undertakes the secondary computing work and the terminal 120 undertakes the main computing work; or, the server 140 and the terminal 120 both adopt a distributed computing architecture for collaborative computing.

[0070] like Figure 2 As shown, in an optional design of the present application, the computer system includes a content production end 201, an upstream and downstream content interface server 202, a scheduling center server 203, a content database 204, a video classification model 205, a video classification service 206, a content deduplication service 207, a content distribution export service 208, a content consumption end 209, a manual review system 210, a content storage service 211, a download file system 212 and a video content frame extraction and audio separation 213.

[0071] The content provided by the content production end 201 includes at least one of PGC (Professional Generated Content), UGC (User Generated Content), MCN (Multi-Channel Network) and PUGC (Professional generated content + User Generated Content). The content production end 201 provides the video to the upstream and downstream content interface server 202 through the mobile terminal interface API (Application Programming Interface) system or the background interface API system. Optionally, the content production end 201 obtains the interface address of the upstream and downstream content interface server 202 by communicating with the upstream and downstream content interface server 202, and uploads the video to the upstream and downstream content interface server 202 through the aforementioned interface address.

[0072] The uplink and downlink content interface server 202 communicates directly with the content production terminal 201, writes the meta information of the video provided by the content production terminal 201 into the content database 204, and submits the video content to the scheduling center server 203. The meta information of the video includes at least one of the file size, cover image link, bit rate, file format, title, release time, and author of the video.

[0073] The dispatch center server 203 is used to be responsible for the entire dispatch process of video flow. Exemplarily, the dispatch center server 203 is used to receive the stored content through the uplink and downlink content interface servers 202, and obtain the meta information of the video from the content database 204. Exemplarily, the dispatch center server 203 is also used to dispatch the manual review system 210 and other processing systems to control the order and priority of the dispatch. Exemplarily, the dispatch center server 203 is also used to call the content deduplication service 207 to filter out unnecessary repeated similar content. For content that does not reach the repeated filtering, the system will output the content similarity and similar relationship chain for the system to break up and use. Exemplarily, the dispatch center server 203 is also used to recommend the video to the content consumer 209 through the content export distribution service 208 (for example, a recommendation engine or a search engine or an operator) by calling the content of the manual review system 210. Exemplarily, the dispatch center server 203 is also responsible for communicating with the video classification service to complete the multi-level classification and marking of the video.

[0074] The content database 204 stores the meta information of the video. Exemplarily, after receiving the video, the uplink and downlink content interface server 202 will transcode the video, and after the transcoding is completed, the meta information of the video will be saved in the content database 204. Optionally, the content database 204 also stores the manual review results of the manual review system 210. Optionally, the content database 204 also stores the deduplication results of the content deduplication service 207.

[0075] The video classification model 205 is used to generate video labels at different granularities using the multimodal features of the video. For example, the label of the video is "technology - smart phone - domestic mobile phone".

[0076] The video classification service 206 is used to service the video classification model 205 so as to realize the communication with the scheduling center server 203 .

[0077] The content deduplication service 207 is used to exclude similar uploaded videos. Optionally, the content deduplication service 207 vectorizes the uploaded video to obtain an uploaded video vector; and compares the similarity between the uploaded video vector and the existing video vector through the vector index. The content deduplication service 207 can realize the engineering and parallelization of deduplication services to avoid the activation of duplicate videos.

[0078] The content distribution export service 208 is used to provide the index information of the video to the content consumption end 209 so that the content consumption end 208 can download the streaming media information of the video and play the video through a video player.

[0079] The content consumption end 209 is used to obtain videos from the content distribution export service 208. Optionally, the content consumption end 209 obtains videos of different categories and display contents of different video channels according to the content distribution export service 208. Optionally, the content consumption end 209 directly downloads videos from the content storage service 211.

[0080] The manual review system 210 is used to conduct a secondary review of the content of the video. Exemplarily, the manual review system 210 is a system developed based on a web (network) database. Since the video classification model 205 may have errors, the accuracy and efficiency of the classification label of the video itself can be improved by performing a secondary manual review process.

[0081] The content storage service 211 is used to store videos uploaded by the content production terminal 201. Optionally, the content storage service 211 includes a data source for internal services and a data source for external services, and the data source for internal services and the data source for external services are deployed separately to avoid mutual influence. Optionally, the content storage service 211 implements acceleration through distributed caching of a CDN (full name Content Delivery Network) acceleration server.

[0082] The download file system 212 is used to control the speed and progress of video downloading. Optionally, the download file system is a group of parallel servers, including related task scheduling clusters and distribution clusters.

[0083] Video content frame extraction and audio separation 213 is used to sample and extract frames from the video and adopt and separate audio information. Exemplarily, when the video downloaded by the download file system 212 is provided to the video classification model 205, the video content frame extraction and audio separation 213 obtains the key frame of the video or calls the TSN (Time Sensitive Network) network for sampling.

[0084] Figure 3 The schematic diagram of the structure of a video classification model provided by an exemplary embodiment of the present application is shown. The video classification model 300 includes a feature extraction module 301 , a feature fusion module 302 and a classification module 303 .

[0085] The feature extraction module 301 is used to extract multimodal features of the target video. The feature extraction module 301 includes a first feature extraction network, a second feature extraction network, a third feature extraction network, ..., an nth feature extraction network, a total of n feature extraction network layers, each feature extraction network layer can extract features of different modalities of the target video. For example, the first feature extraction network is used to extract image features of the target video, and the second feature extraction network is used to extract audio features of the target video.

[0086] The feature fusion module 302 is used to fuse the multimodal features of the target video. The input of the feature fusion module 302 is the multimodal features output by the feature extraction module 301, and the output is the fused features 305. After the multimodal features of the target video are fused to obtain the fused features 305, the optimization module 304 is optionally called to perform compression processing, activation processing and context gating processing on the fused features 304 in sequence to obtain the fused features 305.

[0087] The classification module 303 is used to determine the labels of the target video at different granularities according to the fusion feature 305. The input of the classification module 303 is the fusion feature 305, and the output is the label of the target video at different granularities. Optionally, taking the output of the classification label of three granularities as an example for explanation, the classification module 303 includes a cascaded first-level label classifier 306, a second-level label classifier 307 and a third-level label classifier 308, wherein the classification labels output by the first-level label classifier 306, the second-level label classifier 307 and the third-level label classifier 308 are arranged from large to small according to the granularity size. The first-level label classifier 306 is called to process the data of the fusion feature 305, and the first-level classification feature is output. The first-level class label can be determined according to the first-level classification feature. The second-level label classifier 307 is called to process the data of the fusion feature 305 and the above-mentioned first-level classification feature, and the second-level classification feature is output. The second-level class label can be determined according to the second-level classification feature. The three-level label classifier 308 is called to process the fusion feature 305, the secondary class classification feature and the hidden layer feature of the secondary label classifier 307, and output the three-level class classification feature. The three-level class label can be determined according to the three-level class classification feature. Exemplarily, the classification module 205 outputs the classification labels of three granularities, namely, "technology-mobile phone-domestic mobile phone".

[0088] In the following embodiments, extraction of image features, audio features, and text features is used as an example for explanation. Figure 4 The schematic diagram of the structure of a video classification model provided by an exemplary embodiment of the present application is shown. The video classification model 400 includes an image feature extraction module 401 , an audio feature extraction module 402 , a text feature extraction module 403 , a feature fusion module 302 and a classification module 303 .

[0089] The image feature extraction module 401 is used to extract the image features of the entire target video. The input of the image feature extraction module is the target video 404, and the output is the image feature 407. In an optional design, the target video frame is extracted from the target video 3404 through a sparse sampling operation; the image feature extraction network layer 405 is called to perform data processing on the target video frame and output the video frame features corresponding to the target video frame; the image feature fusion network layer 406 is called to fuse the video frame features from the perspective of image features to obtain the image features 407 of the target video. Exemplarily, the target video frame is extracted from the target video 404 through a TSN network. Exemplarily, the image feature extraction network layer 406 is RestNet (residual network) or Xception (a type of Inception network). Exemplarily, the image feature fusion network layer 308 is a NeXtVLAD network (an improved VLAD network that adds nonlinear parameters of the VLAD layer in the VLAD network, where VLAD refers to Vector of Local Aggregated Descriptors). The NeXtVLAD network can decompose high-dimensional video frame features into a set of low-dimensional video frame features, and aggregate low-dimensional video frame features into low-dimensional image features.

[0090] The audio feature extraction module 402 is used to extract the audio features of the entire target video. The input of the audio feature extraction module is the spectrogram 408 of the audio of the target video, and the output is the audio features 412. In an optional design, the spectrogram 408 of the target video is converted into a Mel-spectrogram 409; the audio feature extraction network is called to perform data processing on the Mel-spectrogram 409, and the audio features 412 of the target video are output. Since the Mel-spectrogram 409 includes spectra of different frequency bands that do not overlap each other, the audio features output here include audio features of multiple frequency bands, so it is also necessary to call the audio feature fusion network 411 to fuse the audio features of multiple frequency bands to obtain the final audio features 412. Exemplarily, the audio feature extraction network 312 is a Vggish network. Exemplarily, the audio feature fusion network 411 is a NeXtVLAD network. Similarly, when the audio feature fusion network 411 is a NeXtVLAD network, the audio feature fusion network layer 411 in the audio feature extraction module 402 will also decompose the high-dimensional audio features in different frequency bands into a group of low-dimensional features, and aggregate the above low-dimensional features to represent the low-dimensional audio features of the entire target video. Optionally, select the 16kHz audio of the first 10 minutes of the target video, use a 25ms Hamming time window and a 10ms frame shift to perform a short-time Fourier transform on the audio to obtain a spectrogram 408, and then map the spectrogram to a 64-order Mel filter group to calculate a Mel spectrogram 409. The Mel spectrogram 409 is framed with a duration of 960ms, with no overlap between frames, each frame is 10ms long, and each frame contains 64 Mel frequency bands.

[0091] The text feature extraction module 403 is used to extract the text features of the entire target video. The input of the text feature extraction module is the title and text content 413, and the output is the text feature 416. In an optional design, the text content in the target video is extracted; the text feature extraction network layer 415 is called to perform data processing on the text content and the title of the target video, and the text feature 416 of the target video is output. Exemplarily, the text feature extraction network layer 415 is a corpus training BERT (Bidirectional Encoder Representation from Transformers) model based on the large-scale text corpus of the information flow. In practice, since some videos do not have titles or the information conveyed by the titles is insufficient, it is necessary to provide the text content of the target video. Optionally, the content of the target video is subjected to text recognition by OCR (Optical Character Recognition) to obtain the text content. However, there are also some problems with the text content obtained by OCR, such as: inaccurate OCR recognition during screen switching, fixed position OCR needs to be deduplicated, oral OCR needs to be retained, and news scrolling OCR needs to be deleted. Optionally, the text preprocessing network layer 414 is called to preprocess the text content. The preprocessing here includes but is not limited to filtering single-word / pure number / pure letter OCR, filtering OCR with small position offset between two adjacent frames of text and high text repetition rate, and filtering OCR with text at the bottom of the screen and small size.

[0092] It should be noted that the above-mentioned image feature extraction module 4014, audio feature extraction module 402, and text feature extraction module 403 are all feature extraction modules. In the embodiment of the present application, the number of feature extraction modules is at least two, and the number of feature extraction modules can be more.

[0093] The feature fusion module 304 is used to fuse the features of the target video. The feature fusion module 304 can fuse the image feature 407 , the audio feature 412 , and the text feature 416 to obtain the fused feature 303 .

[0094] The classification module 303 is used to determine the labels of the target video at different granularities according to the fusion features 303 .

[0095] Figure 5 FIG. 1 is a flow chart of a video classification method provided by an exemplary embodiment of the present application. Figure 1 The terminal 120 or the server 140 or other computer device shown is executed, and the method includes:

[0096] Step 502: Obtain n multimodal features of the target video.

[0097] The target video may be any video, a local video stored in a computer device, a video downloaded from the Internet, or a video provided by other computer devices, which is not limited in the present embodiment.

[0098] Multimodal features refer to features corresponding to the target video in different modes. Multimodal features include at least two features. In an embodiment of the present application, n multimodal features include at least two of image features, audio features, and text features, and n is a positive integer greater than 1.

[0099] In an optional design of the embodiment of the present application, a feature extraction network layer is used to obtain multimodal features. For example, an image feature extraction network layer is used to obtain image features; an audio feature extraction network layer is used to obtain audio features; and a text feature extraction network layer is used to obtain text features.

[0100] In the case where the n multimodal features include image features, a target video frame is extracted from the target video; an image feature extraction network is called to perform data processing on the target video frame and output the video frame features of the target video frame; the video frame features are fused to obtain the image features of the target video. The target video frame is a video frame extracted from the target video according to a preset frame extraction strategy. Exemplarily, a TSN network is called to extract the target video frame from the target video 306. Optionally, the image feature extraction network is RestNet or Xception.

[0101] In the case where the n multimodal features include audio features, a mel-spectrogram of the target video is obtained; an audio feature extraction network is called to perform data processing on the mel-spectrogram, and the audio features of the target video are output. Optionally, a Vggish network is called to perform data processing on the mel-spectrogram, and the audio features of the target video are output.

[0102] In the case where the n multimodal features include text features, the text content in the target video is extracted; the text feature extraction network is called to process the text content and the title of the target video, and the text features of the target video are output. In an optional design, the text content is obtained by performing text recognition on the content of the target video through OCR. In an optional design, the BERT model trained in advance based on the large-scale text corpus of the information flow is called to process the text content and the title of the target video, and the text features of the target video are output.

[0103] Step 504: Fuse n multimodal features to obtain fused features.

[0104] Fusion features refer to features that contain n multimodal features.

[0105] Optionally, a vector bitwise multiplication method is used to fuse n multimodal features.

[0106] Step 506: Classify the fused features according to m classification granularities to obtain the overall classification features of the target video and m granular classification features.

[0107] The m types of classification granularities are used to represent m types of interrelated granularities for classification under the target dimension, where m is a positive integer. The target dimension refers to the dimension used to classify the fused features, and the m types of granular classification features obtained are all features corresponding to the target dimension. For example, three granular classification features are obtained, and the three granular classification features all correspond to the technology dimension. Optionally, the target dimension is determined based on the video content of the target video. In an optional design, the target dimension is the largest classification granularity among the m types of classification granularities.

[0108] The m granularity classification features are used to represent the classification features corresponding to the m classification granularities. For example, assuming that there are three classification granularities, the target video is classified to obtain three classification granularity features. Among them, classification granularity feature 1 corresponds to "technology", which is the feature with the largest granularity, classification granularity feature 2 corresponds to "mobile phone", which is the feature with the middle granularity, and classification granularity feature 3 corresponds to "domestic mobile phone", which is the feature with the smallest granularity.

[0109] The overall classification feature is a feature that includes m granularity classification features. The overall classification feature may include the complete m granularity classification features, and may also include partial features of the m granularity classification features. Exemplarily, the overall classification feature extracts a portion of features from each granularity classification feature of the m granularity classification features, and fuses the extracted features to obtain the overall classification feature.

[0110] In an optional design, m cascaded label classifiers are called to perform data processing on the fused features to obtain m granular classification features. The hidden layer features of the m cascaded label classifiers are fused to obtain the overall classification features. Optionally, the m label classifiers belong to a fully connected feedforward network.

[0111] Step 508: Obtain m-level classification labels of the target video based on the overall classification features and the m granular classification features.

[0112] The applicant believes that the final classification label of a certain granularity should comprehensively consider the overall classification characteristics of the entire classification system, and use the overall classification characteristics as a reference to determine the m-level classification label of the target video.

[0113] In an optional design, the overall classification feature is combined with the m granularity classification features to obtain m target classification features; the m-level classification label of the target video is determined according to the m target classification features. The m target classification features are used to determine the classification labels at the m classification granularities.

[0114] In an optional design, the target classification feature includes probability values ​​of multiple candidate labels, and the candidate label corresponding to the maximum probability value in the target classification feature is determined as the classification label.

[0115] The m-level classification labels are video labels arranged according to m classification granularities. For example, the 3-level classification label of the target video is "Life - Food Preparation - Sichuan Cuisine Preparation".

[0116] In summary, this embodiment fuses the multimodal features of the target video to obtain fused features, and determines the classification label of the target video based on the fused features. This method can model and mine finer granular information through multimodal features and deepen the understanding and extraction of video content. When classifying the fused features, the dependencies and constraint information between different modalities are fully utilized from an overall perspective to improve the accuracy of the video classification results.

[0117] Figure 6 FIG. 1 is a flow chart of a video classification method provided by an exemplary embodiment of the present application. Figure 1 The terminal 120 or the server 140 or other computer device shown is executed, and the method includes:

[0118] Step 601: Obtain n multimodal features of a target video.

[0119] The target video may be any video, a local video stored in a computer device, a video downloaded from the Internet, or a video provided by other computer devices, which is not limited in the present embodiment.

[0120] Multimodal features refer to the features corresponding to the target video in different modes. Multimodal features include at least two features.

[0121] Step 602: Based on the attention mechanism, determine n weights corresponding to the n multimodal features.

[0122] Features of different modalities will describe the target video from different angles, and the contributions of different modalities to the classification results are inconsistent. For example, image features provide more information, while audio features provide less information. Therefore, it is necessary to assign weights to n multimodal features through the attention mechanism so that the subsequent fusion features are more accurate.

[0123] Exemplarily, based on the gated attention mechanism, n weights corresponding to n multimodal features are determined. The method is based on the weights of dimension-specific scalar multimodal features dynamically generated by the gated mechanism.

[0124] Step 603: Perform weighted calculation on the n multimodal features according to the n weights to obtain a fusion feature.

[0125] Optionally, the weighting is calculated by bitwise multiplication of the vectors.

[0126] Step 604: calling the cascaded m label classifiers, performing data processing on the fused features, and obtaining m granular classification features.

[0127] The m label classifiers are used to classify the fusion features from the m classification granularities to obtain m granularity classification features representing the m classification granularities. In an optional design, the connection between the m label classifiers can refer to Figure 3 or Figure 4 The connection mode of the label classifier in the classification module 303 is shown.

[0128] The m granularity classification features are used to represent the features corresponding to the m classification granularities.

[0129] Step 605: According to the cascade order of the m label classifiers, the hidden layer features of the m label classifiers are fused to obtain the overall classification features.

[0130] In an alternative design, the m label classifiers belong to a fully connected feed-forward network.

[0131] Exemplarily, the cascade order of the three label classifiers is "label classifier 1-label classifier 2-label classifier 3". When fusing the hidden layer features of the label classifiers, the hidden layer features of label classifier 1, label classifier 2 and label classifier 3 are fused in turn.

[0132] It should be noted that there is no particular order for step 604 and step 605. Step 604 may be performed first and then step 605, or step 605 may be performed first and then step 604. The embodiment of the present application does not limit this.

[0133] Step 606: Combine the overall classification features with the m granular classification features respectively to obtain m target classification features.

[0134] Optionally, m target classification features are obtained by splicing the m granular classification features to the overall classification features, or m target classification features are obtained by splicing the m granular classification features to the overall classification features.

[0135] Step 607: Determine the m-level classification label of the target video according to the m target classification features.

[0136] In an optional design, the target classification feature includes probability values ​​of multiple candidate labels, and the candidate label corresponding to the maximum probability value in the target classification feature is determined as the classification label.

[0137] In summary, this embodiment fuses the multimodal features of the target video to obtain fused features, and determines the classification label of the target video based on the fused features. This method can model and mine finer granular information through multimodal features and deepen the understanding and extraction of video content. When classifying the fused features, the dependencies and constraint information between different modalities are fully utilized from an overall perspective to improve the accuracy of the video classification results.

[0138] Moreover, when classifying the fused features, granular classification features are obtained from different granularities, which can mine finer-grained information and deepen the understanding of video content through multimodal hierarchical modeling, make full use of the overall classification features as constraint information, and improve the model prediction accuracy.

[0139] Figure 7 A schematic diagram of the structure of a classification module provided by an exemplary embodiment of the present application is shown.

[0140] The classification module 305 includes a label classifier 501, a label classifier 502, and a label classifier 503. The label classifier 501, the label classifier 502, and the label classifier 503 are a three-level hierarchical system from coarse to fine. The label classifier 501 constrains the label classifier 502, and the label classifier 502 constrains the label classifier 503. The input of the label classifier 501 is the fusion feature, and the output is the primary class label. The label classifier 501 will provide the primary class label to the label classifier 502. The input of the label classifier 502 is the primary class label and the fusion feature, and the output is the secondary class label. The label classifier 502 will provide its own hidden layer features and secondary class labels to the label classifier 503. The input of the label classifier 503 is the hidden layer features, secondary class labels, and fusion features of the label classifier 502, and the output is the third class label. In summary, the classification module 305 transmits information between the label classifiers through the above-mentioned hierarchical structure to generate classification labels of different granularities.

[0141] However, there is a problem with the classification module 305. The secondary class label is affected by the primary class label, and the tertiary class label is affected by the primary class label and the secondary class label. The primary class label and the secondary class label belong to the label embedding vector (labelembedding), which has an obvious constraint effect. In the case of an error in the primary class label, the secondary class label and the tertiary class label are bound to be wrong. Therefore, the classification module 305 is optimized and expanded to obtain the classification module 500.

[0142] The classification module 500 includes a label classifier 501, a label classifier 502, and a label classifier 503. However, the connection method of the label classifiers in the classification module 500 is different from that of the classification module 305. Among them, the hidden layer features of all label classifiers are fused to obtain an overall classification feature 504, and the overall classification feature 504 will be input into each label classifier as an input part. In addition, the target classification feature output by the adjacent label classifier will also be input into the label classifier. The input of each label classifier includes two parts: (1) the overall classification feature 504; (2) the target classification feature output by the adjacent label classifier. For example, the input of the label classifier 501 is the overall classification feature 504 and the target classification feature output by the label classifier 502, the input of the label classifier 502 is the overall classification feature 504, the label classifier 501 and the target classification feature output by the label classifier 503, and the input of the label classifier 503 is the overall classification feature 504 and the target classification feature output by the label classifier 502. The classification labels obtained in the above manner will include the overall classification features 504 and the classification labels output by the adjacent label classifier, which can effectively improve and enhance the accuracy of the classification labels. Moreover, a certain level of classification labels is affected by both the overall classification features and the adjacent classification labels, rather than only by the previous level of classification labels, which weakens the constraints on the classification labels. Even if the first level of classification labels are wrong, the second level and third level of classification labels may not be wrong.

[0143] Figure 8 FIG. 1 is a flow chart of a video classification method provided by an exemplary embodiment of the present application. Figure 1 The terminal 120 or the server 140 or other computer device shown is executed, and the method includes:

[0144] Step 801: For the ith granularity classification feature among the m granularity classification features, determine the neighbor granularity classification features adjacent to the ith granularity classification feature.

[0145] Wherein, i is a positive integer less than m+1, and the initial value of i is 1.

[0146] For example, there are 4 granularity classification features arranged in order, then the neighbor granularity classification feature corresponding to the first granularity classification feature is the second granularity classification feature, the neighbor granularity classification feature corresponding to the second granularity classification feature is the first granularity classification feature and the third granularity classification feature, the neighbor granularity classification feature corresponding to the third granularity classification feature is the second granularity classification feature and the third granularity classification feature, and the neighbor granularity classification feature corresponding to the fourth granularity classification feature is the third granularity classification feature.

[0147] Step 802: call the label classifier corresponding to the i-th granularity classification feature, perform data processing on the overall classification feature and the neighbor granularity classification feature, and output the i-th target classification feature corresponding to the i-th granularity classification feature.

[0148] Optionally, the label classifier is a fully connected feed-forward network. The label classifier is used to combine the overall classification feature and the neighbor granularity classification feature of the i-th granularity classification feature to obtain the i-th target classification feature.

[0149] Step 803: After updating i to i+1, repeat the above two steps until m target classification features are obtained.

[0150] The above steps 801 to 802 need to be iterated m times, and a target classification feature is obtained in each iteration until m target classification features are obtained.

[0151] In summary, this embodiment provides a method for determining target classification features, which can obtain target classification features for classification through overall classification features and neighbor granularity classification features adjacent to the granularity classification features. It can effectively integrate information between different features to improve and enhance the accuracy of classification labels. It can weaken the constraint effect of classification labels. For example, when the first-level class label is wrong, the second-level class label and the third-level class label may not be wrong.

[0152] Fig. 9 The following is a schematic diagram of the structure of a fusion module provided by an exemplary embodiment of the present application. The fusion of two multimodal features, image features and audio features, is used as an example for explanation:

[0153] On the one hand, the image 901 of the target video is input into the flip whitening module 903 to realize the flip whitening of the image 901 for the subsequent feature extraction process. The flip whitened image is input into the image feature extraction network layer 904 to obtain the image features of the target video. On the other hand, the audio 902 of the target video is input into the audio feature extraction network layer 905 to obtain the audio features of the target video.

[0154] The image features and audio features of the target video are input into the fusion module 906 to achieve the fusion of the image features and the audio features. Optionally, the method of fusing the image features and the audio features is at least one of a gating-based fusion method, an attention-based fusion method, and a tensor-based fusion method. Taking the gating-based fusion method as an example, the fusion method assigns weights to the image features and the audio features, calculates the weighted sum of the image features and the audio features, and obtains the fusion features.

[0155] The gating module 907 is used to reduce the structure of overfitting of the neural network. The gating module 907 is applied in the training phase of the network to prevent overfitting of the network. In the application phase of the network, the gating module 907 is not necessary. In an optional design, the gating module 907 is a dropout layer.

[0156] The fully connected layer 908 and the SE module 909 (S stands for Sequeeze, compression. E stands for Excitation, activation) are used to calibrate the fused features. In an optional design, the SE module 909 is SE Context Gating (Sequeeze & Excitation Context Gating, for compression and activation of upper and lower gating).

[0157] The classification module 910 is used to classify the calibrated fusion features to obtain the classification label of the target video.

[0158] Fig.10 FIG. 1 shows a flow chart of a feature fusion method provided by an exemplary embodiment of the present application. Figure 1 The terminal 120 or the server 140 or other computer device shown is executed, and the method includes:

[0159] Step 1001: Based on the attention mechanism, determine n weights corresponding to n multimodal features.

[0160] Features of different modalities will describe the target video from different angles, and the contributions of different modalities to the classification results are inconsistent. For example, image features provide more information, while audio features provide less information. Therefore, it is necessary to assign weights to n multimodal features through the attention mechanism so that the subsequent fusion features are more accurate.

[0161] Optionally, based on a gated attention mechanism, n weights corresponding to the n multimodal features are determined. The method is based on weights of dimension-specific scalar multimodal features dynamically generated by a gating mechanism.

[0162] Step 1002: Perform weighted calculation on n multimodal features according to n weights to obtain fused features.

[0163] Optionally, the weighting is calculated by bitwise multiplication of the vectors.

[0164] Step 1003: compress and activate the fused features, and output intermediate fused features.

[0165] The compression process is used to perform global average pooling on the fused features, converting the C×H×W fused features into 1×1×C.

[0166] The activation process is used to perform a nonlinear transformation on the fused features after compression.

[0167] Optionally, the compression process and the activation process are implemented by the SE module.

[0168] Step 1004: Calibrate the intermediate fusion features and output optimized fusion features.

[0169] Optionally, the calibration process can be expressed as the following formula: Y = σ(WX+b)*X, where X is the input intermediate fusion feature, Y is the optimized fusion feature, σ is the activation function (e.g., Sigmoid function, S-shaped growth curve), and W and b are parameters obtained through training. σ(WX+b) is a string of 01 vectors to activate or inhibit the intermediate fusion feature. This approach has two purposes: first, to reflect the internal correlation of each modal feature in the intermediate fusion feature through this activation mechanism, and second, to further calibrate the intermediate fusion feature.

[0170] It should be noted that step 1003 and step 1004 are optional steps, and step 1003 and step 1004 may be performed or may not be performed.

[0171] In summary, this embodiment provides a method for feature fusion, which can fuse multimodal features of a target video and optimize the fused features, so that the fused features can describe the target video from different modalities and can more accurately represent the target video.

[0172] In the following embodiments, the multimodal features of the target video including image features, audio features and text features are taken as an example for explanation. Fig.11 FIG. 1 is a flow chart showing a method for training a video classification model provided by an exemplary embodiment of the present application. Figure 1 The terminal 120 or the server 140 or other computer device shown is executed, and the method includes:

[0173] Step 1101: Based on the attention mechanism, determine the image weight corresponding to the image feature.

[0174] Optionally, based on a gated attention mechanism, image weights corresponding to image features are determined.

[0175] In another optional implementation, image weights corresponding to image features are determined based on a tensor method.

[0176] Step 1102: Based on the attention mechanism, determine the audio weight corresponding to the audio feature.

[0177] Optionally, based on a gated attention mechanism, an audio weight corresponding to the audio feature is determined.

[0178] In another optional implementation, the audio weight corresponding to the audio feature is determined based on a tensor method.

[0179] Step 1103: Based on the attention mechanism, determine the text weight corresponding to the image feature.

[0180] Optionally, based on a gated attention mechanism, a text weight corresponding to the text feature is determined.

[0181] In another optional implementation, a text weight corresponding to a text feature is determined based on a tensor method.

[0182] It should be noted that, when actually executing step 1101, step 1102 and step 1103, there is no particular order. For example, the order of step 1101, step 1102 and step 1103 is followed, or the order of step 1102, step 1103 and step 1101 is followed. The embodiment of the present application does not limit this.

[0183] Step 1104: Perform weighted calculation on the image features, audio features and text features according to the image weight, audio weight and text weight to obtain fusion features.

[0184] Optionally, the weighting is calculated by bitwise multiplication of the vectors.

[0185] Exemplarily, the image feature is denoted as a, the audio feature is denoted as b, the text feature is denoted as c, the image weight is denoted as k1, the audio weight is denoted as k2, and the text weight is denoted as k3, where k1+k2+k3=1, then the fusion feature d=a*k1+b*k2+c*k3.

[0186] Step 1105: compress and activate the fused features, and output intermediate fused features.

[0187] The compression process is used to perform global average pooling on the fused features, converting the C×H×W fused features into 1×1×C.

[0188] The activation process is used to perform a nonlinear transformation on the fused features after compression.

[0189] Optionally, the compression process and the activation process are implemented by the SE module.

[0190] Step 1106: Calibrate the intermediate fusion features and output optimized fusion features.

[0191] It should be noted that step 1105 and step 1106 are optional steps, and step 1105 and step 1106 may be performed or may not be performed.

[0192] In summary, this embodiment provides a method for fusing image features, audio features, and text features, which can fuse the image features, audio features, and text features of a target video and optimize the fused features, so that the fused features can describe the target video from the three modalities of image, audio, and text, and the fused features can more accurately represent the target video.

[0193] Fig.12 The flowchart of the training method of the video classification model provided by an exemplary embodiment of the present application is shown. The video classification model includes a feature extraction network layer, a feature fusion network layer and a classification network layer. Figure 1 The terminal 120 or the server 140 or other computer device shown is executed, and the method includes:

[0194] Step 1202: Obtain a sample training set.

[0195] The sample training set includes sample videos and real annotations corresponding to the sample videos;

[0196] The sample training set includes one or more sample videos.

[0197] Step 1204: Call the feature extraction network layer, perform data processing on the sample video, and output n sample multimodal features.

[0198] The n sample multimodal features include at least two sample multimodal features. Exemplarily, the n sample multimodal features include at least two of sample image features, sample audio features, and sample text features, and n is a positive integer greater than 1.

[0199] Step 1206: Call the feature fusion network layer to fuse the n sample multimodal features and output the sample fusion features.

[0200] Sample fusion features refer to features that contain n multimodal features.

[0201] Optionally, a vector bitwise multiplication method is used to fuse n types of sample multimodal features.

[0202] Step 1208: Call the classification network layer to perform data processing on the n sample multimodal features and sample fusion features, and output n sample granularity classification labels and sample fusion classification labels, respectively.

[0203] Optionally, the classification network layer includes n+1 cascaded label classifiers, and the n+1 cascaded label classifiers correspond one-to-one to the n sample multimodal features and sample fusion features.

[0204] Step 1210: Calculate the cross entropy between n sample granularity classification labels and the true annotations respectively to obtain n granularity cross entropies.

[0205] The n-granularity cross entropy is used to measure the difference information between the n-sample granularity classification labels and the true annotations.

[0206] Optionally, the n granularity cross entropies include image cross entropy, audio cross entropy and text cross entropy. Image cross entropy is the cross entropy corresponding to sample image features, audio cross entropy is the cross entropy corresponding to sample audio features, and text cross entropy is the cross entropy corresponding to sample text features.

[0207] For example, Fig.13 As shown, when the n sample multimodal features include sample image features 1301, sample audio features 1302, and sample text features 1303, the label classifier 1305 is called to process the sample image features 1301 to obtain the sample image classification labels; the image cross entropy is obtained according to the cross entropy between the sample image classification labels and the true annotations. The label classifier 1306 is called to process the sample audio features 1301 to obtain the sample audio classification labels; the audio cross entropy 1310 is obtained according to the cross entropy between the sample audio classification labels and the true annotations. The label classifier 1307 is called to process the sample text features 1301 to obtain the sample text classification labels; the text cross entropy 1311 is obtained according to the cross entropy between the sample text classification labels and the true annotations.

[0208] Step 1212: Calculate the cross entropy between the sample fusion classification label and the true annotation to obtain the fusion cross entropy.

[0209] For example, Fig.13 As shown, the label classifier 1308 is called to perform data processing on the sample fusion feature 1304 to obtain the sample fusion classification label; and the fusion cross entropy is obtained based on the cross entropy between the sample text classification label and the true annotation.

[0210] It should be noted that there is no particular order in which step 1210 and step 1212 are executed. Step 1210 may be executed first, and then step 1212. Step 1212 may also be executed first, and then step 1210.

[0211] Step 1214: Train the video classification model based on the n granularity cross entropies and the fusion cross entropy.

[0212] During the training process, if only one modality is trained, the video classification model may fit to the modality that is easy to converge, thus affecting the learning of network parameters. To avoid this, relative entropy is introduced to supervise the learning of the video classification model. At this time, this step includes the following sub-steps:

[0213] 1. Calculate the relative entropy of n sample multimodal features and sample fusion features respectively to obtain n granularity relative entropies.

[0214] Optionally, the n granularity relative entropies include at least two of image cross entropy, audio cross entropy and text cross entropy. The n granularity relative entropies are used to describe the information entropy difference between the n granularity cross entropies and the fusion cross entropy.

[0215] Exemplarily, when the n sample multimodal features include sample image features 1301, sample audio features 1302, and sample text features 1303, the relative entropy between the probability distribution of the sample image features and the probability distribution of the sample fusion features is calculated to obtain the image relative entropy 1312 among the n granular relative entropies. The relative entropy between the probability distribution of the sample audio features and the probability distribution of the sample fusion features is calculated to obtain the audio relative entropy 1313 among the n granular relative entropies. The relative entropy between the probability distribution of the sample text features and the probability distribution of the sample fusion features is calculated to obtain the text relative entropy 1314 among the n granular relative entropies.

[0216] 2. Calculate the sum of the relative entropies of n particles to obtain the relative entropy loss.

[0217] Optionally, the relative entropy loss is obtained by directly calculating the sum of the relative entropies of n granules. Alternatively, weights are assigned to the relative entropies of n granules, and the relative entropy loss is obtained by weighted calculation of the sum of the relative entropies of n granules.

[0218] 3. Taking minimizing the relative entropy loss as the training goal, the video classification model is trained based on n granular cross entropies and fused cross entropies.

[0219] Optionally, minimizing relative entropy loss is used as a training goal, and according to n granular cross entropies and fusion cross entropies, model parameters of the video classification model are adjusted according to an error back propagation algorithm to train the video classification model.

[0220] In summary, this embodiment trains the video classification model, which can effectively improve the accuracy of the model and accelerate the convergence speed of the model. Moreover, the introduction of relative entropy can enhance the synergy of multiple modes.

[0221] In the following embodiments, n sample multimodal features including sample image features, sample audio features and sample text features are taken as an example for explanation. Fig.14 The flowchart of the training method of the video classification model provided by an exemplary embodiment of the present application is shown. The video classification model includes a feature extraction network layer, a feature fusion network layer and a classification network layer. Figure 1 The terminal 120 or the server 140 or other computer device shown is executed, and the method includes:

[0222] Step 1401: Obtain a sample training set.

[0223] The sample training set includes sample videos and real annotations corresponding to the sample videos;

[0224] The sample training set includes one or more sample videos.

[0225] Step 1402: Call the image feature extraction network layer, perform data processing on the sample video, and output sample image features.

[0226] Optionally, extract a sample target video frame from the sample video; call an image feature extraction network to perform data processing on the sample target video frame, and output sample video frame features of the sample video frame; and fuse the sample video frame features to obtain sample image features of the sample video.

[0227] Step 1403: Call the audio feature extraction network layer, perform data processing on the sample video, and output the sample audio features.

[0228] Optionally, a mel-spectrogram of the sample video is obtained; an audio feature extraction network is called to perform data processing on the mel-spectrogram, and sample audio features of the sample video are output.

[0229] Step 1404: Call the text feature extraction network layer, perform data processing on the sample video, and output sample text features.

[0230] Optionally, extract text content from the sample video; call a text feature extraction network, perform data processing on the text content and the title of the sample video, and output sample text features of the sample video.

[0231] It should be noted that step 1402, step 1403 and step 1404 are executed in no particular order.

[0232] Step 1405: Call the feature fusion network layer to fuse the sample image features, sample audio features and sample text features, and output the sample fusion features.

[0233] Sample fusion features refer to features that include sample image features, sample audio features, and sample text features.

[0234] Optionally, a vector bitwise multiplication method is used to fuse sample image features, sample audio features, and sample text features.

[0235] Step 1406: Call the classification network layer to perform data processing on the sample image features, sample audio features, sample text features, and sample fusion features, and output the sample image classification label, sample audio classification label, sample text classification label, and sample fusion classification label, respectively.

[0236] Optionally, the classification network layer includes four cascaded label classifiers, and the four cascaded label classifiers correspond one-to-one to sample image features, sample audio features, sample text features and sample fusion features.

[0237] Step 1407: Calculate the cross entropy between the sample image classification label and the true annotation to obtain the image cross entropy.

[0238] For example, Fig.13 As shown, the label classifier 1305 is called to perform data processing on the sample image feature 1301 to obtain the sample image classification label; and the image cross entropy is obtained according to the cross entropy between the sample image classification label and the true annotation.

[0239] Step 1408: Calculate the cross entropy between the sample audio classification label and the true annotation to obtain the audio cross entropy.

[0240] For example, Fig.13 As shown, the label classifier 1306 is called to perform data processing on the sample audio feature 1301 to obtain the sample audio classification label; and the audio cross entropy 1310 is obtained based on the cross entropy between the sample audio classification label and the true annotation.

[0241] Step 1409: Calculate the cross entropy between the sample text classification label and the true annotation to obtain the text cross entropy.

[0242] For example, Fig.13 As shown, the label classifier 1307 is called to perform data processing on the sample text feature 1301 to obtain the sample text classification label; and the text cross entropy 1311 is obtained based on the cross entropy between the sample text classification label and the true annotation.

[0243] Step 1410: Calculate the cross entropy between the sample fusion classification label and the true annotation to obtain the fusion cross entropy.

[0244] For example, Fig.13 As shown, the label classifier 1308 is called to perform data processing on the sample fusion feature 1304 to obtain the sample fusion classification label; and the fusion cross entropy is obtained based on the cross entropy between the sample text classification label and the true annotation.

[0245] It should be noted that step 1407, step 1408, step 1409 and step 1410 are executed in no particular order.

[0246] Step 1411: Train the video classification model based on the image cross entropy, audio cross entropy, text cross entropy and fusion cross entropy.

[0247] In an optional design of the present application, relative entropy is introduced to supervise the learning of the video classification model: the relative entropy between the probability distribution of the sample image features and the probability distribution of the sample fusion features is calculated to obtain the image relative entropy among the n granularity relative entropies; the relative entropy between the probability distribution of the sample audio features and the probability distribution of the sample fusion features is calculated to obtain the audio relative entropy among the n granularity relative entropies; the relative entropy between the probability distribution of the sample text features and the probability distribution of the sample fusion features is calculated to obtain the text relative entropy among the n granularity relative entropies. The sum of the image relative entropy, audio relative entropy and text relative entropy is calculated to obtain the relative entropy loss. With minimizing the relative entropy loss as the training goal, the video classification model is trained according to the image cross entropy, audio cross entropy, text cross entropy and fusion cross entropy.

[0248] In summary, this embodiment trains the video classification model and introduces sample image features, sample audio features, and sample text features into the training of the video classification model. This can effectively improve the accuracy of the model and accelerate the convergence of the model. Moreover, the introduction of relative entropy can enhance the synergy of image, audio, and text modalities.

[0249] In the following embodiment, target videos are recommended to users based on the classification labels of the videos. Fig.15 A schematic diagram of a video recommendation method provided by an exemplary embodiment of the present application is shown. The method includes:

[0250] Step 1501: Obtain n multimodal features of the video to be recommended.

[0251] The video to be recommended may be any video, a local video stored in a computer device, a video downloaded from the Internet, or a video provided by other computer devices, which is not limited in the present embodiment.

[0252] In an embodiment of the present application, the n multimodal features include at least two of image features, audio features, and text features, and n is a positive integer greater than 1.

[0253] Step 1502: Fuse n multimodal features to obtain fused features.

[0254] Optionally, a vector bitwise multiplication method is used to fuse n multimodal features.

[0255] Step 1503: Classify the fused features according to m classification granularities to obtain the overall classification features of the video to be recommended and m granular classification features.

[0256] The m types of classification granularities are used to represent m types of interrelated granularities for classification under the target dimension, where m is a positive integer. The target dimension refers to the dimension used to classify the fused features, and the obtained m types of granular classification features are all features corresponding to the target dimension.

[0257] The overall classification feature is a feature that includes m granular classification features. The overall classification feature may include the complete m granular classification features, that is, it may include partial features of the m granular classification features.

[0258] Step 1504: Obtain m-level classification labels for the video to be recommended based on the overall classification features and the m granular classification features.

[0259] In an optional design, the overall classification feature is combined with the m granularity classification features to obtain m target classification features; the m-level classification label of the target video is determined according to the m target classification features. The m target classification features are used to determine the classification labels at the m classification granularities.

[0260] Step 1505: When the recommendation tag of the target user account matches the m-level classification tag, the video to be recommended is recommended to the target user account.

[0261] Optionally, the recommended tags of the target user account are determined based on the historical browsing records of the target user account. Alternatively, the recommended tags of the target user account are set by the user.

[0262] Optionally, when the target user account's recommendation tag is completely identical to the M-level classification tag, the video to be recommended is recommended to the user account. Optionally, when the target user account's recommendation tag is partially identical to the M-level classification tag, the video to be recommended is recommended to the user account.

[0263] For example, the recommended tag of the target user account is "Technology - Mobile Phones - Domestic Mobile Phones", the classification tag of the video 1 to be recommended is "Technology - Mobile Phones - Foreign Mobile Phones", and the classification tag of the video 2 to be recommended is "Life - Food - Chinese Food". Since the classification tag of the video 2 to be recommended is completely different from the recommended tag of the target user account, and the classification tag of the video 1 to be recommended is partially the same as the recommended tag of the target user account, the video 1 to be recommended is recommended to the target user account.

[0264] In summary, this embodiment provides a method for recommending videos to a user account. By obtaining the recommendation of the user account, a video is recommended to the user account in a targeted manner, ensuring that the recommended video meets the user's preferences.

[0265] In the following embodiment, target videos are recommended to users based on the classification labels of the videos. Fig.16A schematic flow chart of a video recommendation method provided by an exemplary embodiment of the present application is shown.

[0266] Step 1601: Obtain image features of the video to be recommended.

[0267] The video to be recommended may be any video, a local video stored in a computer device, a video downloaded from the Internet, or a video provided by other computer devices, which is not limited in the present embodiment.

[0268] Optionally, a target video frame is extracted from the video to be recommended; an image feature extraction network is called to perform data processing on the target video frame, and video frame features of the target video frame are output; and the video frame features are fused to obtain image features of the video to be recommended.

[0269] Step 1602: Obtain the audio features of the video to be recommended.

[0270] Optionally, obtain the mel-spectrogram of the video to be recommended; call the audio feature extraction network, perform data processing on the mel-spectrogram, and output the audio features of the video to be recommended. Optionally, call the Vggish network, perform data processing on the mel-spectrogram, and output the audio features of the video to be recommended.

[0271] Step 1603: Obtain text features of the video to be recommended.

[0272] Optionally, extract text content from the video to be recommended; call a text feature extraction network to perform data processing on the text content and the title of the video to be recommended, and output text features of the video to be recommended.

[0273] Step 1604: Fuse the image features, audio features, and text features to obtain fused features.

[0274] Optionally, a vector bitwise multiplication method is used to fuse n multimodal features.

[0275] Step 1605: Classify the fused features according to m classification granularities to obtain the overall classification features of the video to be recommended and m granular classification features.

[0276] The m types of classification granularities are used to represent m types of interrelated granularities for classification under the target dimension, where m is a positive integer. The target dimension refers to the dimension used to classify the fused features, and the obtained m types of granular classification features are all features corresponding to the target dimension.

[0277] The overall classification feature is a feature that includes m granular classification features. The overall classification feature may include the complete m granular classification features, that is, it may include partial features of the m granular classification features.

[0278] Step 1606: Obtain m-level classification labels for the video to be recommended based on the overall classification features and the m granular classification features.

[0279] In an optional design, the overall classification feature is combined with the m granularity classification features to obtain m target classification features; the m-level classification label of the target video is determined according to the m target classification features. The m target classification features are used to determine the classification labels at the m classification granularities.

[0280] Step 1605: When the recommendation tag of the target user account matches the m-level classification tag, the video to be recommended is recommended to the target user account.

[0281] Optionally, the recommended tags of the target user account are determined based on the historical browsing records of the target user account. Alternatively, the recommended tags of the target user account are set by the user.

[0282] Optionally, when the target user account's recommendation tag is completely identical to the M-level classification tag, the video to be recommended is recommended to the user account. Optionally, when the target user account's recommendation tag is partially identical to the M-level classification tag, the video to be recommended is recommended to the user account.

[0283] In summary, this embodiment provides a method for recommending videos to a user account. By obtaining the recommendation of the user account, a video is recommended to the user account in a targeted manner, ensuring that the recommended video meets the user's preferences.

[0284] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0285] Please refer to Fig.17 , which shows a block diagram of a video classification device provided by an embodiment of the present application. The above functions can be implemented by hardware, or by hardware executing corresponding software. The device 1700 includes:

[0286] A feature extraction module 1701 is used to obtain n multimodal features of a target video, wherein the n multimodal features include at least two of image features, audio features, and text features, where n is a positive integer greater than 1;

[0287] A fusion module 1702 is used to fuse the n multimodal features to obtain a fusion feature;

[0288] A classification module 1703 is used to classify the fusion features according to m classification granularities to obtain an overall classification feature of the target video and m granularity classification features, wherein the m classification granularities are used to represent m interrelated granularities for classification under a target dimension, and the m granularity classification features are used to represent classification features corresponding to the m classification granularities;

[0289] The classification module 1703 is further used to obtain m-level classification labels of the target video according to the overall classification features and the m granularity classification features, and the m-level classification labels are video labels arranged according to the m classification granularities.

[0290] In an optional design, the classification module 1703 is further used to combine the overall classification features with the m granular classification features respectively to obtain m target classification features; and determine the m-level classification label of the target video according to the m target classification features.

[0291] In an optional design, the classification module 1703 is also used to determine, for the i-th granularity classification feature among the m granularity classification features, a neighbor granularity classification feature adjacent to the i-th granularity classification feature, where i is a positive integer less than m+1, and the initial value of i is 1; call the label classifier corresponding to the i-th granularity classification feature, perform data processing on the overall classification feature and the neighbor granularity classification feature, and output the i-th target classification feature corresponding to the i-th granularity classification feature; after updating i to i+1, repeat the above two steps until the m target classification features are obtained.

[0292] In an optional design, the fusion module 1702 is also used to call the cascaded m label classifiers, perform data processing on the fused features, and obtain the m granular classification features; according to the cascade order of the m label classifiers, the hidden layer features of the m label classifiers are fused to obtain the overall classification features.

[0293] In an optional design, the fusion module 1702 is also used to determine n weights corresponding to the n multimodal features based on an attention mechanism; and perform weighted calculation on the n multimodal features according to the n weights to obtain the fusion feature.

[0294] In an optional design, the classification module 1703 is also used to determine the image weight of the image feature, the audio weight of the audio feature, and the text weight of the text feature based on the attention mechanism; the classification module 1703 is also used to perform weighted calculation on the image feature, the audio feature and the text feature according to the image weight, the audio weight and the text weight, and output the fused feature.

[0295] In an optional design, the fusion module 1702 is further used to perform compression processing and activation processing on the fusion features to output intermediate fusion features; perform calibration processing on the intermediate fusion features to output optimized fusion features.

[0296] In an optional design, the feature extraction module 1701 is also used to extract a target video frame from the target video; call an image feature extraction network to perform data processing on the target video frame, and output video frame features of the target video frame; and fuse the video frame features to obtain the image features of the target video.

[0297] In an optional design, the feature extraction module 1701 is also used to obtain the Mel-spectrogram of the target video; call the audio feature extraction network, perform data processing on the Mel-spectrogram, and output the audio features of the target video.

[0298] In an optional design, the feature extraction module 1701 is also used to extract text content in the target video; call a text feature extraction network to perform data processing on the text content and the title of the target video, and output the text features of the target video.

[0299] In summary, this embodiment fuses the multimodal features of the target video to obtain fused features, and determines the classification label of the target video based on the fused features. This method can make full use of the dependencies and constraint information between different modalities to improve the accuracy of video classification results. More multimodal content understanding features that characterize the video content are obtained, including multi-level fine-grained classification information of the video, to assist in the content distribution of the recommendation system; the understanding of the content can make full use of the title text, audio content, and video content of the video content as the basis, and the characterization is more comprehensive and accurate, improving the efficiency of content distribution; it can deepen the understanding of the video content by mining finer-grained information through multimodal hierarchical modeling, make full use of the category hierarchy dependencies and constraint information, and improve the accuracy of the model.

[0300] Please refer to Fig.18 , which shows a block diagram of a video classification model training device provided by an embodiment of the present application. The above functions can be implemented by hardware, or by hardware executing corresponding software. The device 1800 includes:

[0301] The sample acquisition module 1801 is used to acquire a sample training set, wherein the sample training set includes a sample video and a real annotation corresponding to the sample video;

[0302] A sample feature extraction module 1802 is used to call the feature extraction network layer, perform data processing on the sample video, and output n sample multimodal features, where the n sample multimodal features include at least two of sample image features, sample audio features, and sample text features, and n is a positive integer greater than 1;

[0303] The sample fusion module 1803 is used to call the feature fusion network layer to fuse the n sample multimodal features and output sample fusion features;

[0304] The sample classification module 1804 is used to call the classification network layer, perform data processing on the n sample multimodal features and sample fusion features, and output n sample granularity classification labels and sample fusion classification labels respectively;

[0305] The training module 1805 is used to respectively calculate the cross entropy between the n sample granularity classification labels and the true annotations to obtain n granularity cross entropies; calculate the cross entropy between the sample fusion classification label and the true annotation to obtain fusion cross entropy;

[0306] The training module 1805 is further used to train the video classification model according to the n granularity cross entropies and the fusion cross entropy.

[0307] In an optional design, the training module 1805 is also used to calculate the relative entropy between the n granularity cross entropies and the fused cross entropy respectively to obtain n granularity relative entropies; calculate the sum of the n granularity relative entropies to obtain a relative entropy loss; and train the video classification model according to the n granularity cross entropies and the fused cross entropy with the goal of minimizing the relative entropy loss.

[0308] In an optional design, the n sample multimodal features include the sample image features; the training module 1805 is also used to calculate the relative entropy between the probability distribution of the sample image features and the probability distribution of the sample fusion features, and obtain the image relative entropy among the n granularity relative entropies.

[0309] In an optional design, the n sample multimodal features include the sample audio features; the training module 1805 is also used to calculate the relative entropy between the probability distribution of the sample audio features and the probability distribution of the sample fusion features, and obtain the audio relative entropy among the n granular relative entropies.

[0310] In an optional design, the n sample multimodal features include the sample text feature; the training module 1805 is also used to calculate the relative entropy between the probability distribution of the sample text feature and the probability distribution of the sample fusion feature, and obtain the text relative entropy among the n granular relative entropies.

[0311] In summary, this embodiment trains the video classification model, which can effectively improve the accuracy of the model and accelerate the convergence speed of the model. Moreover, the introduction of relative entropy can enhance the synergy of multiple modes.

[0312] Fig.19 1 is a schematic diagram of the structure of a computer device according to an exemplary embodiment. The computer device 1900 includes a central processing unit (CPU) 1901, a system memory 1904 including a random access memory (RAM) 1902 and a read-only memory (ROM) 1903, and a system bus 1905 connecting the system memory 1904 and the central processing unit 1901. The computer device 1900 also includes a basic input / output system (I / O system) 1906 for helping to transmit information between various devices in the computer device, and a large-capacity storage device 1907 for storing an operating system 1913, an application program 1914 and other program modules 1915.

[0313] The basic input / output system 1906 includes a display 1908 for displaying information and an input device 1909 such as a mouse and a keyboard for user inputting information. The display 1908 and the input device 1909 are connected to the central processing unit 1901 through an input / output controller 1910 connected to the system bus 1905. The basic input / output system 1906 may also include an input / output controller 1910 for receiving and processing inputs from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 1910 also provides output to a display screen, a printer, or other types of output devices.

[0314] The mass storage device 1907 is connected to the central processing unit 1901 through a mass storage controller (not shown) connected to the system bus 1905. The mass storage device 1907 and its associated computer device readable medium provide non-volatile storage for the computer device 1900. That is, the mass storage device 1907 may include a computer device readable medium (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0315] Without loss of generality, the computer device readable medium may include computer device storage media and communication media. Computer device storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer device readable instructions, data structures, program modules or other data. Computer device storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), CD-ROM, Digital Video Disc (DVD) or other optical storage, cassette, tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will know that the computer device storage medium is not limited to the above. The above-mentioned system memory 1904 and mass storage device 1907 can be collectively referred to as memory.

[0316] According to various embodiments of the present disclosure, the computer device 1900 can also be connected to a remote computer device on a network through a network such as the Internet. That is, the computer device 1900 can be connected to the network 1911 through a network interface unit 1912 connected to the system bus 1905, or the network interface unit 1912 can be used to connect to other types of networks or remote computer device systems (not shown).

[0317] The memory also includes one or more programs, which are stored in the memory. The central processing unit 1901 implements the above-mentioned video classification method, or all or part of the steps of the training method of the video classification model by executing the one or more programs.

[0318] In an exemplary embodiment, a computer-readable storage medium is also provided, in which at least one instruction, at least one program, a code set or an instruction set is stored, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the video classification method provided by the above-mentioned various method embodiments, or the training method of the video classification model.

[0319] The present application also provides a computer-readable storage medium, which stores at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the video classification method provided in the above method embodiment, or the training method of the video classification model.

[0320] The present application also provides a computer program product or a computer program, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the video classification method provided in the above embodiment, or the training method of the video classification model.

[0321] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0322] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0323] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A video classification method, characterized in that: The method comprises: Obtaining n multimodal features of a target video, wherein the n multimodal features include at least two of image features, audio features, and text features, and n is a positive integer greater than 1; Fusing the n multimodal features to obtain a fused feature; Classifying the fusion features according to m classification granularities to obtain an overall classification feature and m granularity classification features of the target video, wherein the m classification granularities are used to represent m mutually related granularities for classification under the target dimension, and the m granularity classification features are used to represent classification features corresponding to the m classification granularities, m is a positive integer, and the overall classification feature is obtained by fusing a part of features taken from each feature of the m granularity classification features; Combining the overall classification features with the m granularity classification features respectively to obtain m target classification features; The m-level classification labels of the target video are determined according to the m target classification features, and the m-level classification labels are video labels arranged according to the m classification granularities.

2. The method according to claim 1, characterized in that The overall classification feature is combined with the m granularity classification features respectively to obtain m target classification features, including: For the i-th granularity classification feature among the m granularity classification features, determine a neighbor granularity classification feature adjacent to the i-th granularity classification feature, where i is a positive integer less than m+1, and an initial value of i is 1; Calling a label classifier corresponding to the i-th granularity classification feature, performing data processing on the overall classification feature and the neighbor granularity classification feature, and outputting an i-th target classification feature corresponding to the i-th granularity classification feature; After updating i to i+1, the above two steps are repeated until the m target classification features are obtained.

3. The method according to claim 1 or 2, characterized in that: The fusion features are classified according to m classification granularities to obtain the overall classification features of the target video and m granular classification features, including: Calling the cascaded m label classifiers to perform data processing on the fusion features to obtain the m granular classification features; According to the cascade order of the m label classifiers, the hidden layer features of the m label classifiers are fused to obtain the overall classification feature.

4. The method according to claim 1 or 2, characterized in that: The fusing of the n multimodal features to obtain a fused feature includes: Based on the attention mechanism, determining n weights corresponding to the n multimodal features; According to the n weights, weighted calculation is performed on the n multimodal features to obtain the fusion feature.

5. The method according to claim 4, characterized in that The n multimodal features include image features, audio features and text features; The determining, based on the attention mechanism, n weights corresponding to the n multimodal features includes: Based on the attention mechanism, determining the image weight of the image feature, determining the audio weight of the audio feature, and determining the text weight of the text feature; The step of performing weighted calculation on the n multimodal features according to the n weights to obtain the fusion feature includes: According to the image weight, the audio weight and the text weight, weighted calculation is performed on the image feature, the audio feature and the text feature, and the fusion feature is output.

6. The method according to claim 1 or 2, characterized in that: The method further comprises: Performing compression and activation processing on the fused features, and outputting intermediate fused features; The intermediate fusion features are calibrated and optimized fusion features are output.

7. The method according to claim 1 or 2, characterized in that: The n multimodal features include the image features; The method further comprises: Extracting a target video frame from the target video; Calling an image feature extraction network to perform data processing on the target video frame and outputting video frame features of the target video frame; The video frame features are fused to obtain the image features of the target video.

8. The method according to claim 1 or 2, characterized in that: The n multimodal features include the audio feature; The method further comprises: Obtaining a Mel-spectrogram of the target video; An audio feature extraction network is called to perform data processing on the Mel-spectrogram and output the audio features of the target video.

9. The method according to claim 1 or 2, characterized in that: The n multimodal features include the text feature; The method further comprises: Extracting text content from the target video; A text feature extraction network is called to perform data processing on the text content and the title of the target video, and the text features of the target video are output.

10. A method for training a video classification model, characterized in that: The video classification model includes a feature extraction network layer, a feature fusion network layer and a classification network layer, and the method includes: Obtaining a sample training set, wherein the sample training set includes a sample video and a real annotation corresponding to the sample video; Calling the feature extraction network layer to perform data processing on the sample video and outputting n kinds of sample multimodal features, wherein the n kinds of sample multimodal features include at least two of sample image features, sample audio features, and sample text features, where n is a positive integer greater than 1; Calling the feature fusion network layer to fuse the n types of sample multimodal features and output sample fusion features; Calling the classification network layer to perform data processing on the n types of sample multimodal features and sample fusion features, respectively, and outputting n types of sample granularity classification labels and sample fusion classification labels, respectively; Calculate the cross entropy between the n sample granularity classification labels and the true annotations respectively to obtain n granularity cross entropies; calculate the cross entropy between the sample fusion classification label and the true annotation to obtain fusion cross entropy; The video classification model is trained according to the n types of granularity cross entropies and the fusion cross entropy.

11. The method according to claim 10, characterized in that The step of training the video classification model according to the n types of granularity cross entropies and the fusion cross entropy includes: Relative entropies between the n types of sample multimodal features and the sample fusion features are calculated respectively to obtain n types of granularity relative entropies; Calculating the sum of the relative entropies of the n particle sizes to obtain a relative entropy loss; Taking minimizing the relative entropy loss as a training goal, the video classification model is trained according to the n types of granular cross entropies and the fusion cross entropy.

12. The method according to claim 11, characterized in that The n types of sample multimodal features include the sample image features; The method further comprises: The relative entropy between the probability distribution of the sample image feature and the probability distribution of the sample fusion feature is calculated to obtain the image relative entropy among the n types of granularity relative entropies.

13. The method according to claim 11, characterized in that The n sample multimodal features include the sample audio features; The method further comprises: The relative entropy between the probability distribution of the sample audio feature and the probability distribution of the sample fusion feature is calculated to obtain the audio relative entropy among the n types of granularity relative entropies.

14. The method according to claim 11, characterized in that The n types of sample multimodal features include the sample text features; The method further comprises: The relative entropy between the probability distribution of the sample text feature and the probability distribution of the sample fusion feature is calculated to obtain the text relative entropy among the n types of granularity relative entropies.

15. A video classification device, characterized in that: The device comprises: A feature extraction module, used to obtain n multimodal features of a target video, wherein the n multimodal features include at least two of image features, audio features, and text features, and n is a positive integer greater than 1; A fusion module, used for fusing the n multimodal features to obtain a fusion feature; A classification module is used to classify the fusion features according to m classification granularities to obtain an overall classification feature and m granularity classification features of the target video, wherein the m classification granularities are used to represent m mutually related granularities for classification under a target dimension, and the m granularity classification features are used to represent classification features corresponding to the m classification granularities, where m is a positive integer, and the overall classification feature is obtained by fusing a portion of features taken from each of the m granularity classification features; The classification module is also used to combine the overall classification features with the m granularity classification features respectively to obtain m target classification features; determine the m-level classification labels of the target video according to the m target classification features, and the m-level classification labels are video labels arranged according to the m classification granularities.

16. A training device for a video classification model, characterized in that: The video classification model includes a feature extraction network layer, a feature fusion network layer and a classification network layer, and the device includes: A sample acquisition module, used to acquire a sample training set, wherein the sample training set includes a sample video and a real annotation corresponding to the sample video; A sample feature extraction module, used to call the feature extraction network layer, perform data processing on the sample video, and output n kinds of sample multimodal features, wherein the n kinds of sample multimodal features include at least two of sample image features, sample audio features, and sample text features, where n is a positive integer greater than 1; A sample fusion module, used to call the feature fusion network layer, perform fusion processing on the n types of sample multimodal features, and output sample fusion features; A sample classification module is used to call the classification network layer, perform data processing on the n types of sample multimodal features and sample fusion features, and output n types of sample granularity classification labels and sample fusion classification labels respectively; The training module is used to respectively calculate the cross entropy between the n sample granularity classification labels and the true annotations to obtain n granularity cross entropies; calculate the cross entropy between the sample fusion classification labels and the true annotations to obtain fusion cross entropy; The training module is also used to train the video classification model according to the n types of granularity cross entropies and the fusion cross entropy.

17. A computer device, characterized in that: The computer device includes: a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the video classification method as described in any one of claims 1 to 9, or the training method of the video classification model as described in any one of claims 10 to 14.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one program code, and the program code is loaded and executed by a processor to implement the video classification method as described in any one of claims 1 to 9, or the training method of a video classification model as described in any one of claims 10 to 14.

19. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the video classification method according to any one of claims 1 to 9 is implemented, or the training method of the video classification model according to any one of claims 10 to 14 is implemented.

Images