Video classification method, device, equipment and medium

Abstract

The present application relates to a video classification method, apparatus, device and medium, the method comprising: constructing a multi-layer feature fusion network; extracting an image feature set and an audio feature set from a target video; using bottleneck unit vectors to alternately perform joint learning of image features and audio features in the multi-layer feature fusion network, obtaining target overall image features output by the image feature fusion unit of the last layer of feature fusion network and target overall audio features output by the audio feature fusion unit of the last layer of feature fusion network; inputting the target overall image features and target overall audio features into a classifier to determine the category of the target video. The present application obtains important feature information for classification during the fusion process of the image features and audio features of a video by setting bottleneck unit vectors and performing data interaction, thereby solving the problem that the feature fusion process is slow and prone to overfitting.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a video classification method, apparatus, device and medium. Background Art

[0002] Video classification is currently a key research topic in the field of computer vision. Accurately classifying videos can enhance the user experience and provide better recommendations. For companies, classifying large numbers of internet videos can also help them identify current market trends and implement appropriate strategic deployments. Currently, the commonly used fusion method for video classification is to simultaneously fuse the video's image and audio features before performing the classification. This simultaneous fusion of multiple image and audio features is slow and prone to overfitting.

[0003] Currently, no effective solution has been proposed to the above-mentioned problem that "simultaneously fusing multiple image features and multiple audio features of a video is slow and prone to overfitting." Summary of the Invention

[0004] The present application provides a video classification method, apparatus, device and medium to solve or at least partially solve the above-mentioned technical problem of "simultaneously fusing multiple image features and multiple audio features of a video, which is slow and prone to overfitting".

[0005] According to one aspect of an embodiment of the present application, the present application provides a video classification method, including: constructing a multi-layer feature fusion network, wherein each layer of the feature fusion network includes an image feature fusion unit and an audio feature fusion unit; extracting an image feature set and an audio feature set in the target video, wherein the image feature set is used as the input of the image feature fusion unit in the first layer of the feature fusion network, and the audio feature set is used as the input of the audio feature fusion unit in the first layer of the feature fusion network; using bottleneck unit vectors to alternately perform joint learning of image features and audio features in the multi-layer feature fusion network to obtain the target overall image features output by the image feature fusion unit of the last layer of the feature fusion network and the target overall audio features output by the audio feature fusion unit of the last layer of the feature fusion network; inputting the target overall image features and the target overall audio features into a classifier to determine the category of the target video.

[0006] Optionally, using bottleneck unit vectors alternately to perform joint learning of image features and audio features in a multi-layer feature fusion network includes performing joint learning of image features and audio features on the first-layer feature fusion network in the following manner: randomly generating bottleneck unit vectors and image feature representation vectors; splicing the image feature representation vector, the image feature set, and the bottleneck unit vector and inputting them into the image feature fusion unit of the first-layer feature fusion network; obtaining a new bottleneck unit vector output by the image feature fusion unit of the first-layer feature fusion network, and randomly generating an audio feature representation vector; splicing the audio feature representation vector, the audio feature set, and the bottleneck unit vector and inputting them into the audio feature fusion unit of the first-layer feature fusion network, so as to complete the joint learning of image features and audio features of the first-layer feature fusion network using the bottleneck unit vector.

[0007] Optionally, using bottleneck unit vectors alternately to perform joint learning of image features and audio features in a multi-layer feature fusion network also includes: obtaining a new image feature representation vector, a new image feature set, and a new bottleneck unit vector output by the image feature fusion unit of the previous layer feature fusion network; splicing the image feature representation vector, the image feature set, and the audio feature fusion unit and inputting them into the image feature fusion unit of the current layer feature fusion network; obtaining a new bottleneck unit vector output by the current layer image feature fusion unit, a new audio feature set, and a new audio feature representation vector output by the previous layer audio feature fusion unit; splicing the audio feature representation vector, the audio feature set, and the bottleneck unit vector and inputting them into the current layer audio feature fusion unit to complete the joint learning of image features and audio features of the current layer feature fusion network.

[0008] Optionally, after using bottleneck unit vectors to alternately perform joint learning of image features and audio features in a multi-layer feature fusion network, the method also includes obtaining the target overall image features output by the image feature fusion unit of the last layer of feature fusion network and the target overall audio features output by the audio feature fusion unit of the last layer of feature fusion network in the following manner: completing the joint learning of each layer of feature fusion network one by one until the last layer of image feature fusion unit outputs a new image feature representation vector and the last layer of audio feature fusion unit outputs a new audio feature representation vector, determining the image feature representation vector as the target overall image feature, and determining the audio feature representation vector as the target overall audio feature.

[0009] Optionally, extracting the image feature set and the audio feature set from the target video includes: extracting audio data and image data of the target video; determining multiple video frames in the image data, and generating multiple image feature vectors corresponding to the multiple video frames, wherein the image feature set includes multiple image feature vectors; performing feature extraction on the audio data, and generating multiple audio feature vectors corresponding to the multiple video frames, wherein the audio feature set includes multiple audio feature vectors.

[0010] Optionally, using bottleneck unit vectors alternately to perform joint learning of image features and audio features in a multi-layer feature fusion network also includes: after the audio feature fusion unit or the image feature fusion unit in each layer of the feature fusion network performs fusion learning, the obtained target fusion features are stored in the bottleneck unit vector.

[0011] Optionally, before randomly generating bottleneck unit vectors, the method includes determining the number of bottleneck unit vectors in the following manner: obtaining the number of feature vectors in the image feature set and / or the audio feature set; determining state information of the current network environment of the multi-layer feature fusion network, and using the state information and the number value to determine the number of bottleneck unit vectors.

[0012] According to another aspect of the embodiments of the present application, the present application also provides a video classification device, including: a construction module for constructing a multi-layer feature fusion network, wherein each layer of the feature fusion network includes an image feature fusion unit and an audio feature fusion unit; an extraction module for extracting an image feature set and an audio feature set in the target video, wherein the image feature set is used as the input of the image feature fusion unit in the first layer of the feature fusion network, and the audio feature set is used as the input of the audio feature fusion unit in the first layer of the feature fusion network; a learning module for using bottleneck unit vectors to alternately perform joint learning of image features and audio features in the multi-layer feature fusion network to obtain the target overall image features output by the image feature fusion unit of the last layer of the feature fusion network and the target overall audio features output by the audio feature fusion unit of the last layer of the feature fusion network; a determination module for inputting the target overall image features and the target overall audio features into a classifier to determine the category of the target video.

[0013] According to another aspect of an embodiment of the present application, the present application also provides an electronic device, including a memory, a processor, a communication interface and a communication bus, wherein the memory stores a computer program that can be run on the processor, the memory and the processor communicate through the communication bus and the communication interface, and the processor implements the steps of any of the above methods when executing the computer program.

[0014] According to another aspect of the embodiments of the present application, the present application also provides a computer-readable medium having a non-volatile program code executable by a processor, where the program code enables the processor to execute any of the above methods.

[0015] The technical solution of this application can be applied to the design of computer vision using deep learning technology.

[0016] The above technical solution provided by the embodiment of the present application has the following advantages compared with the related art:

[0017] The present application provides a video classification method, comprising: constructing a multi-layer feature fusion network, wherein each layer of the feature fusion network includes an image feature fusion unit and an audio feature fusion unit; extracting an image feature set and an audio feature set from a target video, wherein the image feature set is used as an input to the image feature fusion unit in the first layer of the feature fusion network, and the audio feature set is used as an input to the audio feature fusion unit in the first layer of the feature fusion network; using bottleneck unit vectors to alternately perform joint learning of image features and audio features in the multi-layer feature fusion network to obtain target overall image features output by the image feature fusion unit of the last layer of the feature fusion network and target overall audio features output by the audio feature fusion unit of the last layer of the feature fusion network; and inputting the target overall image features and the target overall audio features into a classifier to determine the category of the target video.

[0018] This application uses bottleneck unit vectors to perform data interaction during the fusion process of image features and audio features of a video to obtain important feature information for classification, thereby solving the problem that the feature fusion process is slow and prone to overfitting. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] Figure 1 A schematic diagram of a hardware environment for an optional video classification method provided according to an embodiment of the present application;

[0022] Figure 2 A flowchart of an optional video classification method provided according to an embodiment of the present application;

[0023] Figure 3A schematic diagram of the structure of an optional multi-layer feature fusion network provided according to an embodiment of the present application;

[0024] Figure 4 A block diagram of an optional video classification device provided according to an embodiment of the present application;

[0025] Figure 5 The present invention provides an optional schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0027] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of this application and have no specific meaning. Therefore, "module" and "component" can be used interchangeably.

[0028] Video classification is a key research topic in computer vision. Accurately categorizing videos can enhance user experience and provide better recommendations. For businesses, categorizing large amounts of internet videos can also help them identify current market trends and implement appropriate strategic plans.

[0029] An important attribute of video is that it contains not only the information of the image frames, but also the motion changes of the image frames, the background music in the video, the text descriptions in the image frames, etc. It is an information transmission medium with inherent multimodal characteristics. Therefore, the fine classification of videos is inseparable from the extraction and fusion of features from multiple modalities.

[0030] Existing technical solutions have already formed a certain pattern. For features of multiple modalities, existing pre-trained models are first used for feature extraction. Vit is used for feature extraction of image frames, and VGGish is used for background audio extraction. The main differences and research hotspots of existing technical solutions are concentrated on the fusion of multiple features. For example, mid-term fusion is to learn from the features of each modality that are aligned in time. The learned modal features are then used to generate a feature representing the video. This mid-term fusion method can avoid the problems caused by uneven information between the modalities to a certain extent, but it is slow and prone to overfitting.

[0031] In order to solve the problems mentioned in the background technology, according to one aspect of the embodiments of the present application, an embodiment of a data backup method is provided.

[0032] Optionally, in the embodiment of the present application, the above data backup method can be applied to Figure 1 In the hardware environment composed of the terminal 101 and the server 103 shown in FIG. Figure 1 As shown, the server 103 is connected to the terminal 101 via a network, and can be used to provide services (video classification service, image feature extraction service, audio feature extraction service, etc.) for the terminal or the client installed on the terminal. A database 105 can be set on the server or independently of the server to provide data storage services for the server 103. The above-mentioned network includes but is not limited to: wide area network, metropolitan area network or local area network, and the terminal 101 includes but is not limited to PC, mobile phone, tablet computer, etc.

[0033] A data backup method in the embodiment of the present application can be executed by the server 103, or can be executed by the server 103 and the terminal 101 together, such as Figure 2 Shown, including:

[0034] Step 201: construct a multi-layer feature fusion network, wherein each layer of the feature fusion network includes an image feature fusion unit and an audio feature fusion unit;

[0035] Step 203: extracting an image feature set and an audio feature set from the target video, wherein the image feature set is used as an input to an image feature fusion unit in a first-layer feature fusion network, and the audio feature set is used as an input to an audio feature fusion unit in the first-layer feature fusion network;

[0036] Step 205: Jointly learning image features and audio features in a multi-layer feature fusion network using the bottleneck unit vectors alternately, to obtain target overall image features output by the image feature fusion unit of the last layer of the feature fusion network and target overall audio features output by the audio feature fusion unit of the last layer of the feature fusion network.

[0037] Step 207: Input the target overall image features and the target overall audio features into a classifier to determine the category of the target video.

[0038] The multi-layer feature fusion network includes a multi-layer feature fusion network, and each layer of the feature fusion network includes an image feature fusion unit and an audio feature fusion unit. The image feature fusion unit is used to perform fusion learning of image feature vectors, and the audio feature fusion unit is used to perform fusion learning of audio feature vectors.

[0039] Optionally, the bottleneck unit vector is used to alternately perform joint learning of image features and audio features in the multi-layer feature fusion network, so that the bottleneck unit vector can participate in each fusion learning, and the important features after each fusion learning are obtained and transmitted to the next fusion unit.

[0040] This application constructs a multi-layer feature fusion network and generates bottleneck unit vectors, so that the bottleneck unit vectors can alternately perform joint learning of image features and audio features in the multi-layer feature fusion network, and then input the two features that can respectively represent the overall image features and the overall audio features into the classifier for classification, so as to obtain the classification results for the entire video.

[0041] As an optional embodiment, using bottleneck unit vectors alternately to perform joint learning of image features and audio features in a multi-layer feature fusion network includes performing joint learning of image features and audio features on the first-layer feature fusion network in the following manner: randomly generating bottleneck unit vectors and image feature representation vectors; splicing the image feature representation vector, the image feature set, and the bottleneck unit vector and inputting them into the image feature fusion unit of the first-layer feature fusion network; obtaining a new bottleneck unit vector output by the image feature fusion unit of the first-layer feature fusion network, and randomly generating an audio feature representation vector; splicing the audio feature representation vector, the audio feature set, and the bottleneck unit vector and inputting them into the audio feature fusion unit of the first-layer feature fusion network, so as to complete the joint learning of image features and audio features of the first-layer feature fusion network using the bottleneck unit vector.

[0042] Optionally, the bottleneck unit vector, image feature representation vector and audio feature representation vector are all randomly generated, and carry more or less interference information after generation. The interference information will be updated in subsequent iterative learning. During the iterative learning process, the reverse gradient propagation algorithm can be used to perform iterative updates in each round.

[0043] Specifically, the image feature representation vector, the image feature set and the bottleneck unit vector are spliced and input into the image feature fusion unit of the first-layer feature fusion network. After completing the image fusion learning of the first layer, a new image feature representation vector, a new image feature set and a new bottleneck unit vector are output. At this time, the new bottleneck unit vector includes the effective information for classification saved after calculating the correlation of each feature in the image feature set. The new bottleneck unit vector is used as the input of the audio feature fusion unit of the first-layer feature fusion network, so that the effective information of the first image fusion can be brought into the first audio fusion.

[0044] The new image feature representation vector includes a vector of overall features obtained after the first image fusion learning, and the image features in the new image feature set are new features obtained after calculating the correlation between the various image features.

[0045] Specifically, the audio feature representation vector, the audio feature set and the new bottleneck unit vector are spliced and input into the audio feature fusion unit of the first-layer feature fusion network. After completing the audio fusion learning of the first layer, the new audio feature representation vector, the new audio feature set and the new bottleneck unit vector are output. At this time, the bottleneck unit vector has participated in and obtained effective information of the two fusions.

[0046] The new audio feature representation vector includes a vector of the overall audio features learned after the first audio fusion learning, and the audio features in the new audio feature set are new features obtained after calculating the correlation between the various audio features.

[0047] Optionally, both the image feature fusion unit and the audio feature fusion unit can use a tansformer model.

[0048] At this point, the learning of the first layer of feature fusion network is completed, and then the fusion learning needs to be repeated several times in the following way.

[0049] As an optional embodiment, using bottleneck unit vectors alternately to perform joint learning of image features and audio features in a multi-layer feature fusion network also includes: obtaining a new image feature representation vector, a new image feature set, and a new bottleneck unit vector output by the image feature fusion unit of the previous layer feature fusion network; splicing the image feature representation vector, the image feature set, and the audio feature fusion unit and inputting them into the image feature fusion unit of the current layer feature fusion network; obtaining a new bottleneck unit vector output by the current layer image feature fusion unit, a new audio feature set, and a new audio feature representation vector output by the previous layer audio feature fusion unit; splicing the audio feature representation vector, the audio feature set, and the bottleneck unit vector and inputting them into the current layer audio feature fusion unit to complete the joint learning of image features and audio features of the current layer feature fusion network.

[0050] Starting from the fusion learning of the second-layer feature fusion network, there is no need to regenerate the image feature representation vector, audio feature representation vector and bottleneck unit vector. Instead, the new image feature representation vector, audio feature representation vector and bottleneck unit vector output after learning in the previous layer are used as the input of the new layer. The specific input method and fusion learning method are the same as the fusion learning process of the first-layer feature fusion network mentioned above.

[0051] Specifically, the output of the previous layer will serve as the input of the current layer.

[0052] This section is as follows: Taking the image feature fusion unit as an example (the same applies to audio): Our image is a sequence of information. Feature 1 is the feature extracted from the first frame, and Feature 2 is the feature extracted from the second frame. The image feature fusion unit at each layer receives images from all frames. The role of the image feature fusion unit is to calculate the correlation between each frame and all frames, and then obtain the new features of this frame based on the correlation. The new features of all frames are then sent to the next image feature fusion unit. By jointly learning the features of all video frames and the features of the bottleneck unit vector, the bottleneck unit vector can save the most useful information for the classification result after calculating the correlation of all video frames. For example, in a live broadcast of an outdoor scene, many frames are on the host's face, but some are purely outdoor scenes. These frames are very useful.

[0053] The feature fusion network is multi-layered, so the number of fusion learning times can be set to correspond to the number of layers of the feature fusion network. The number of fusion learning times can also be adjusted according to actual needs, and there is no limitation on this.

[0054] As an optional embodiment, after using bottleneck unit vectors to alternately perform joint learning of image features and audio features in a multi-layer feature fusion network, the method also includes obtaining the target overall image features output by the image feature fusion unit of the last layer of feature fusion network and the target overall audio features output by the audio feature fusion unit of the last layer of feature fusion network in the following manner: completing the joint learning of each layer of feature fusion network one by one until the last layer of image feature fusion unit outputs a new image feature representation vector and the last layer of audio feature fusion unit outputs a new audio feature representation vector, determining the image feature representation vector as the target overall image feature, and determining the audio feature representation vector as the target overall audio feature.

[0055] In each round of fusion learning, the image feature representation vector is used as the output of the network structure from the very beginning. Therefore, in the process of continuous learning and iteration, the image feature representation vector is constantly learning the expression of the entire network. Therefore, after the final training, it can be considered that the result of the image feature representation vector can represent the expression of the network.

[0056] When the last layer of image feature fusion unit outputs a new image feature representation vector and the last layer of audio feature fusion unit outputs a new audio feature representation vector, the new image feature representation vector can represent the comprehensive features of the entire image feature set, and the new audio feature representation vector can represent the comprehensive features of the entire audio feature set. At this time, the new image feature representation vector is determined as the target overall image feature, and the new audio feature representation vector is determined as the target overall audio feature.

[0057] The target overall image features and the target overall audio features are input into the classifier for classification.

[0058] Optionally, the target overall image features and the target overall audio features may be fused first to generate target overall video features, and then the target overall video features are input into a classifier for classification, and the result displayed by the classifier is determined as the category result of the target video.

[0059] Optionally, the target overall image features and the target overall audio features are respectively input into the classifier for classification to obtain a first classification result of the audio and a second classification result of the image, and then the first classification result and the second classification result of the image are processed (for example, taking the average value) to finally obtain the video classification result.

[0060] As an optional embodiment, extracting an image feature set and an audio feature set from a target video includes: extracting audio data and image data from the target video; determining multiple video frames in the image data, and generating multiple image feature vectors corresponding to the multiple video frames, wherein the image feature set includes multiple image feature vectors; performing feature extraction on the audio data, and generating multiple audio feature vectors corresponding to the multiple video frames, wherein the audio feature set includes multiple audio feature vectors.

[0061] Optionally, to determine multiple video frames in the image data, the corresponding video frames can be directly captured according to the number of video frames, or video frames at a preset time interval can be captured (for example, once every 1 second). After obtaining the video frames, feature extraction is performed to obtain an image feature set. The feature extraction method can be VIT or other extraction methods.

[0062] After determining the number of video frames, audio extraction is performed according to the audio points corresponding to the video frames. After obtaining multiple audio point data, feature extraction is performed to obtain an audio feature set. The feature extraction method can be VGGish or other extraction methods.

[0063] Specifically, the number of features in the audio feature set and the image feature set is the same.

[0064] As an optional embodiment, using bottleneck unit vectors alternately to perform joint learning of image features and audio features in a multi-layer feature fusion network also includes: after the audio feature fusion unit or the image feature fusion unit in each layer of the feature fusion network performs fusion learning, the obtained target fusion features are stored in the bottleneck unit vector.

[0065] This application reduces the amount of correlation calculation by utilizing bottleneck unit vectors to perform data interaction in each layer of the feature fusion network, thereby improving the network's operating speed.

[0066] As an optional embodiment, before randomly generating bottleneck unit vectors, the method includes determining the number of bottleneck unit vectors in the following manner: obtaining the number of feature vectors in the image feature set and / or the audio feature set; determining the status information of the current network environment of the multi-layer feature fusion network, and using the status information and the number value to determine the number of bottleneck unit vectors.

[0067] The number of bottleneck unit vectors is determined using the state information and the quantity value. Typically, the number of bottleneck unit vectors can be determined based on the number of feature vectors in the image feature set and / or audio feature set (for example, the number of bottleneck unit vectors is set to one-third of the quantity value. This is because it is believed that only one-third of the video frames are useful for classification, and the remaining frames may not be very useful for classification).

[0068] Specifically, when there are specified network requirements or classification requirements (mainly accuracy), it is necessary to combine state information and quantity values to jointly determine the optimal number of bottleneck unit vectors.

[0069] The design of feature fusion networks requires hyperparameters to control their operation in different scenarios. For example, when performing predictions on a large server, where computing resources and speed are not important and only higher accuracy is desired, more bottleneck unit vectors can be used. However, when performing predictions on mobile devices or in real-time scenarios, where faster results with fewer resources are desired, fewer bottleneck unit vectors can be used. The specific settings depend on the specific video classification task and the network training results. All settings must be based on the actual needs of real data in real scenarios.

[0070] Figure 3 A schematic diagram of the structure of a multi-layer feature fusion network provided by the present application. The video transformer calculates the relationship between the video feature vector and the BFU, and the audio transformer calculates the relationship between the audio feature vector and the BFU. The BFU in the figure corresponds to the bottleneck unit vector of the present application, the image frame feature set in the figure corresponds to the image feature set of the present application, the audio feature set in the figure corresponds to the audio feature set of the present application, the video transformer in the figure corresponds to the image feature fusion unit of the present application, the audio transformer in the figure corresponds to the audio feature fusion unit of the present application, the CLS_V in the figure corresponds to the image feature representation vector of the present application, and the CLS_A in the figure corresponds to the audio feature representation vector of the present application.

[0071] The present application provides a video classification method, comprising: constructing a multi-layer feature fusion network, wherein each layer of the feature fusion network includes an image feature fusion unit and an audio feature fusion unit; extracting an image feature set and an audio feature set from a target video, wherein the image feature set is used as an input to the image feature fusion unit in the first layer of the feature fusion network, and the audio feature set is used as an input to the audio feature fusion unit in the first layer of the feature fusion network; using bottleneck unit vectors to alternately perform joint learning of image features and audio features in the multi-layer feature fusion network to obtain target overall image features output by the image feature fusion unit of the last layer of the feature fusion network and target overall audio features output by the audio feature fusion unit of the last layer of the feature fusion network; and inputting the target overall image features and the target overall audio features into a classifier to determine the category of the target video.

[0072] This application solves the problem that the feature fusion process is slow and prone to overfitting by utilizing bottleneck unit vectors to obtain important feature information for classification during the fusion process of image features and audio features of the video for data interaction.

[0073] The advantages of this application plan are:

[0074] 1. Flexible network structure. Parameters are used to control the number of bottleneck unit vectors. When there are more bottleneck unit vectors, more information can be used for image and audio interaction. When there are fewer bottleneck unit vectors, the network can run faster.

[0075] 2. Suppress overfitting. When using a small number of features for interaction, the bottleneck unit vector learns the most valuable parts of the two modalities, preventing the risk of overfitting the network on the training set due to information redundancy.

[0076] 3. Faster operation speed. The network structure no longer calculates the correlation between two modes, but only needs to calculate the correlation between a certain mode and the bottleneck unit vector, which reduces the amount of calculation and improves the network operation speed.

[0077] According to another aspect of the embodiment of the present application, the present application also provides a video classification device, such as Figure 4 Shown, including:

[0078] A construction module 402 is used to construct a multi-layer feature fusion network, wherein each layer of the feature fusion network includes an image feature fusion unit and an audio feature fusion unit;

[0079] Extraction module 404, used to extract an image feature set and an audio feature set from the target video, wherein the image feature set is used as an input to the image feature fusion unit in the first-layer feature fusion network, and the audio feature set is used as an input to the audio feature fusion unit in the first-layer feature fusion network;

[0080] A learning module 406 is configured to perform joint learning of image features and audio features in a multi-layer feature fusion network using bottleneck unit vectors alternately, thereby obtaining target overall image features output by the image feature fusion unit of the last layer of the feature fusion network and target overall audio features output by the audio feature fusion unit of the last layer of the feature fusion network.

[0081] The determination module 408 is configured to input the target overall image features and the target overall audio features into a classifier to determine the category of the target video.

[0082] It should be noted that the construction module 402 in this embodiment can be used to execute step 201 in the embodiment of the present application, the extraction module 404 in this embodiment can be used to execute step 203 in the embodiment of the present application, the learning module 406 in this embodiment can be used to execute step 205 in the embodiment of the present application, and the determination module 408 in this embodiment can be used to execute step 207 in the embodiment of the present application.

[0083] Optionally, the learning module 406 is also used to perform joint learning of image features and audio features on the first-layer feature fusion network in the following manner: randomly generate a bottleneck unit vector and an image feature representation vector; splice the image feature representation vector, the image feature set and the bottleneck unit vector and input them into the image feature fusion unit of the first-layer feature fusion network; obtain a new bottleneck unit vector output by the image feature fusion unit of the first-layer feature fusion network, and randomly generate an audio feature representation vector; splice the audio feature representation vector, the audio feature set and the bottleneck unit vector and input them into the audio feature fusion unit of the first-layer feature fusion network to complete the joint learning of image features and audio features of the first-layer feature fusion network using the bottleneck unit vector.

[0084] Optionally, the learning module 406 is also used to obtain a new image feature representation vector, a new image feature set, and a new bottleneck unit vector output by the image feature fusion unit of the previous layer feature fusion network; splice the image feature representation vector, the image feature set, and the audio feature fusion unit and input them into the image feature fusion unit of the current layer feature fusion network; obtain a new bottleneck unit vector output by the current layer image feature fusion unit, a new audio feature set, and a new audio feature representation vector output by the previous layer audio feature fusion unit; splice the audio feature representation vector, the audio feature set, and the bottleneck unit vector and input them into the current layer audio feature fusion unit to complete the joint learning of the image features and audio features of the current layer feature fusion network.

[0085] Optionally, the learning module 406 is also used to obtain the target overall image features output by the image feature fusion unit of the last layer of feature fusion network and the target overall audio features output by the audio feature fusion unit of the last layer of feature fusion network in the following manner: completing the joint learning of each layer of feature fusion network one by one until the last layer of image feature fusion unit outputs a new image feature representation vector and the last layer of audio feature fusion unit outputs a new audio feature representation vector, determining the image feature representation vector as the target overall image feature, and determining the audio feature representation vector as the target overall audio feature.

[0086] Optionally, the extraction module 404 is also used to extract audio data and image data of the target video; determine multiple video frames in the image data, and generate multiple image feature vectors corresponding to the multiple video frames, wherein the image feature set includes multiple image feature vectors; perform feature extraction on the audio data, and generate multiple audio feature vectors corresponding to the multiple video frames, wherein the audio feature set includes multiple audio feature vectors.

[0087] Optionally, the learning module 406 is further configured to store the obtained target fusion features into the bottleneck unit vector after the audio feature fusion unit or the image feature fusion unit in the feature fusion network of each layer performs fusion learning.

[0088] Optionally, the learning module 406 is also used to determine the number of bottleneck unit vectors before randomly generating bottleneck unit vectors in the following manner: obtaining the number of feature vectors in the image feature set and / or the audio feature set; determining the status information of the current network environment of the multi-layer feature fusion network, and using the status information and the number value to determine the number of bottleneck unit vectors.

[0089] It should be noted here that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments.

[0090] According to another aspect of the embodiment of the present application, Figure 5 As shown, the present application provides an electronic device, including a memory 51, a processor 52, a communication interface 53 and a communication bus 54. The memory 51 stores a computer program that can be run on the processor 52. The memory 51 and the processor 52 communicate through the communication bus 54 and the communication interface 53. When the processor 52 executes the computer program, the steps of the above method are implemented.

[0091] The memory and processor in the electronic device communicate via a communication bus and a communication interface. The communication bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus may be divided into an address bus, a data bus, a control bus, and the like.

[0092] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0093] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0094] According to another aspect of the embodiments of the present application, the present application provides a computer-readable medium having a non-volatile program code executable by a processor, where the program code enables the processor to execute the steps of any of the above methods.

[0095] Optionally, in an embodiment of the present application, a computer-readable medium is configured to store program code for the processor to execute the above method steps.

[0096] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0097] When implementing the embodiments of the present application, reference may be made to the above embodiments, which have corresponding technical effects.

[0098] It is understood that the embodiments described herein may be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or a combination thereof.

[0099] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0100] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0101] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0102] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0103] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0104] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0105] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application are essentially or partly contributed to the prior art or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard drive, a ROM, a RAM, a magnetic disk, or an optical disk. It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such a process, method, article or device. Without further constraints, an element defined by the phrase "comprises a..." does not preclude the existence of additional identical elements in the process, method, article or apparatus that includes the element.

[0106] The foregoing is merely a list of specific embodiments of the present application, intended to enable those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the broadest scope consistent with the principles and novel features of the present application.

Claims

1. A video classification method, characterized in that: include: Constructing a multi-layer feature fusion network, wherein each layer of the feature fusion network includes an image feature fusion unit and an audio feature fusion unit; Extracting an image feature set and an audio feature set from a target video, wherein the image feature set is used as an input to the image feature fusion unit in the first-layer feature fusion network, and the audio feature set is used as an input to the audio feature fusion unit in the first-layer feature fusion network; Using bottleneck unit vectors alternately to perform joint learning of image features and audio features in multiple layers of the feature fusion network, obtaining target overall image features output by the image feature fusion unit of the last layer of the feature fusion network and target overall audio features output by the audio feature fusion unit of the last layer of the feature fusion network; Inputting the target overall image feature and the target overall audio feature into a classifier to determine the category of the target video; The joint learning of image features and audio features alternately performed in the multi-layer feature fusion network using bottleneck unit vectors includes performing joint learning of image features and audio features on the feature fusion network of the first layer in the following manner: randomly generating the bottleneck unit vector and the image feature representation vector; splicing the image feature representation vector, the image feature set, and the randomly generated bottleneck unit vector, and inputting the spliced image feature fusion unit of the first layer feature fusion network; obtaining the new bottleneck unit vector output by the image feature fusion unit of the first layer feature fusion network, and randomly generating an audio feature representation vector; splicing the audio feature representation vector, the audio feature set, and the new bottleneck unit vector, and inputting the spliced audio feature fusion unit of the first layer feature fusion network, so as to complete the joint learning of image features and audio features of the first layer feature fusion network using the bottleneck unit vector; The joint learning of image features and audio features by alternating the use of bottleneck unit vectors in multiple layers of the feature fusion network also includes: obtaining the new image feature representation vector, the new image feature set, and the new bottleneck unit vector output by the image feature fusion unit of the feature fusion network of the previous layer; splicing the new image feature representation vector, the new image feature set, and the new bottleneck unit vector output by the audio feature fusion unit, and inputting them into the image feature fusion unit of the feature fusion network of the current layer; obtaining the new bottleneck unit vector output by the image feature fusion unit of the current layer, the new audio feature set, and the new audio feature representation vector output by the audio feature fusion unit of the previous layer; splicing the new audio feature representation vector, the new audio feature set, and the new bottleneck unit vector output by the image feature fusion unit of the current layer, and inputting them into the audio feature fusion unit of the current layer, so as to complete the joint learning of image features and audio features of the feature fusion network of the current layer.

2. The method according to claim 1, characterized in that After the bottleneck unit vectors are alternately used to perform joint learning of image features and audio features in multiple layers of the feature fusion network, the method further includes obtaining target overall image features output by the image feature fusion unit of the last layer of the feature fusion network and target overall audio features output by the audio feature fusion unit of the last layer of the feature fusion network in the following manner: Complete the joint learning of each layer of the feature fusion network one by one until the image feature fusion unit of the last layer outputs a new image feature representation vector and the audio feature fusion unit of the last layer outputs a new audio feature representation vector, and determine the new image feature representation vector as the target overall image feature, and determine the new audio feature representation vector as the target overall audio feature.

3. The method according to claim 1, characterized in that The extracting of the image feature set and the audio feature set in the target video includes: Extracting audio data and image data of the target video; determining a plurality of video frames in the image data, and generating a plurality of image feature vectors corresponding to the plurality of video frames, wherein the image feature set includes the plurality of image feature vectors; Feature extraction is performed on the audio data to generate a plurality of audio feature vectors corresponding to the plurality of video frames, wherein the audio feature set includes the plurality of audio feature vectors.

4. The method according to claim 3, characterized in that The method of using bottleneck unit vectors to perform joint learning of image features and audio features in multiple layers of the feature fusion network also includes: After the audio feature fusion unit or the image feature fusion unit in the feature fusion network of each layer performs fusion learning, the obtained target fusion feature is stored in the new bottleneck unit vector.

5. The method according to claim 1, wherein Before randomly generating the bottleneck unit vectors, the method includes determining the number of the bottleneck unit vectors in the following manner: Obtaining a quantity value of feature vectors in the image feature set and / or the audio feature set; Determine state information of a network environment in which the multi-layer feature fusion network currently resides, and determine the number of the bottleneck unit vectors using the state information and the quantity value.

6. A video classification device, characterized in that: include: A construction module is used to construct a multi-layer feature fusion network, wherein each layer of the feature fusion network includes an image feature fusion unit and an audio feature fusion unit; An extraction module, configured to extract an image feature set and an audio feature set from a target video, wherein the image feature set is used as an input to the image feature fusion unit in the first-layer feature fusion network, and the audio feature set is used as an input to the audio feature fusion unit in the first-layer feature fusion network; A learning module, configured to perform joint learning of image features and audio features in multiple layers of the feature fusion network using bottleneck unit vectors alternately, to obtain target overall image features output by the image feature fusion unit of the last layer of the feature fusion network and target overall audio features output by the audio feature fusion unit of the last layer of the feature fusion network; a determination module, configured to input the target overall image feature and the target overall audio feature into a classifier to determine the category of the target video; The learning module is further configured to perform joint learning of image features and audio features on the feature fusion network of the first layer in the following manner: randomly generating the bottleneck unit vector and the image feature representation vector; concatenating the image feature representation vector, the image feature set, and the randomly generated bottleneck unit vector, and inputting the concatenated vectors into the image feature fusion unit of the feature fusion network of the first layer; obtaining the new bottleneck unit vector output by the image feature fusion unit of the feature fusion network of the first layer, and randomly generating an audio feature representation vector; concatenating the audio feature representation vector, the audio feature set, and the new bottleneck unit vector, and inputting the concatenated vectors into the audio feature fusion unit of the feature fusion network of the first layer, so as to complete the joint learning of image features and audio features of the feature fusion network of the first layer using the bottleneck unit vector; The learning module is also used to: obtain the new image feature representation vector, the new image feature set, and the new bottleneck unit vector output by the image feature fusion unit of the feature fusion network of the previous layer; splice the new image feature representation vector, the new image feature set, and the new bottleneck unit vector output by the audio feature fusion unit, and input them into the image feature fusion unit of the feature fusion network of the current layer; obtain the new bottleneck unit vector output by the image feature fusion unit of the current layer, the new audio feature set, and the new audio feature representation vector output by the audio feature fusion unit of the previous layer; splice the new audio feature representation vector, the new audio feature set, and the new bottleneck unit vector output by the image feature fusion unit of the current layer, and input them into the audio feature fusion unit of the current layer, so as to complete the joint learning of the image features and audio features of the feature fusion network of the current layer.

7. An electronic device comprising a memory, a processor, a communication interface, and a communication bus, wherein the memory stores a computer program that can be run on the processor, and the memory and the processor communicate via the communication bus and the communication interface, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable medium having a non-volatile program code executable by a processor, characterized in that The program code enables the processor to execute the method according to any one of claims 1 to 5.

Images