Video classification methods, devices, electronic devices, and computer-readable media

Abstract

This disclosure relates to a video classification method, apparatus, electronic device, and computer-readable medium, belonging to the field of video processing technology. The method includes: inputting various types of information from the video into a video classification model; extracting video information features corresponding to each type of information through various feature extraction networks; fusing the various video information features through a feature fusion network in the video classification model to obtain fused video features; processing the fused video features through a hybrid expert system in the video classification model to obtain classification features at each level corresponding to each level of the video; and inputting the classification features at each level of the video into the classification networks corresponding to each level of the respective category to obtain the classification results at each level of the video. This disclosure, by extracting and fusing various types of information features from the video and using a hybrid expert system to differentiate the fused video features, can output multi-level classification results for the video in one go, improving video classification efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of video processing technology, and more specifically, to a video classification method, a video classification device, an electronic device, and a computer-readable medium. Background Technology

[0002] For short video platforms with massive amounts of video content, video categorization is both in high demand and of great significance. Generally, the video category system on short video platforms is a multi-level, mutually exclusive tree structure. Comprehensively and accurately identifying the vertical category information of videos at each level is of great value in video operation, distribution, and search scenarios.

[0003] For video classification tasks, most mainstream methods are based on deep learning, utilizing related technologies to extract spatiotemporal features of videos and then fusing them to output the classification results. However, for multi-level video classification tasks, they are typically modeled as multiple independent tasks, and the results of these tasks are then fused to output the final multi-level category information. This approach requires training a separate model for each level of classification task, resulting in significant overall resource consumption.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide a video classification method, a video classification device, an electronic device, and a computer-readable medium, which can at least to some extent output multi-level classification results of videos at one time, improve video classification efficiency, and reduce resource consumption costs.

[0006] According to a first aspect of this disclosure, a method for classifying videos is provided, comprising:

[0007] Various types of information from the video are input into the video classification model, and various feature extraction networks in the video classification model are used to extract the video information features corresponding to each type of information in the video.

[0008] The feature fusion network in the video classification model performs feature fusion on the video information features corresponding to each type of information to obtain the fused video features of the video.

[0009] The fused video features are processed by the hybrid expert system in the video classification model to obtain the classification features corresponding to each level of the video category.

[0010] The classification features of the video at each level are input into the classification networks corresponding to each level of the category to obtain the classification results of the video at each level.

[0011] In an exemplary embodiment of this disclosure, the video information features include visual features, text features, and audio features; the feature extraction network includes a visual feature extraction network, a text feature extraction network, and an audio feature extraction network; and the step of extracting video information features corresponding to various types of information in the video through the various feature extraction networks in the video classification model includes:

[0012] Multiple video frames are extracted from the video, and image features are extracted from the multiple video frames respectively through a visual feature extraction network. The image features in the multiple video frames are then fused to obtain the visual features of the video. The video frames include the cover frame and multiple content frames of the video.

[0013] Multiple video texts from the video are obtained, and categorized text features are extracted from the multiple video texts through a text feature extraction network. The categorized text features are then fused to obtain the text features of the video. The video texts include title text, character text, and speech recognition text from the video.

[0014] The audio vector features in the video are extracted, and the audio vector features are processed by an audio feature extraction network to obtain the audio features of the video.

[0015] In one exemplary embodiment of this disclosure, the video information features further include video meta-features and author meta-features, and the extraction of video information features corresponding to various types of information in the video includes:

[0016] Multiple video meta-information of the video are obtained, and the corresponding classification video meta-features of each type of video meta-information are extracted. The classification video meta-features are then fused to obtain the video meta-features of the video. The video meta-information includes video duration, video word frequency, and video type.

[0017] The video's author information is obtained, and the corresponding categorical author meta-features for each type of author information are extracted. These categorical author meta-features are then fused to obtain the video's author meta-features. The author information includes the video author's author type, author level, and author vertical category. In an exemplary embodiment of this disclosure, the hybrid expert system includes multi-level classification tasks corresponding to multiple levels of categories and multiple expert networks. The process of processing the fused video features through the hybrid expert system in the video classification model to obtain the classification features corresponding to each level of the video's categories includes:

[0018] The fused video features are input into the hybrid expert system, and the fused video features are differentiated by multiple expert networks in the hybrid expert system to obtain classification features at each level of the video that respectively represent the multi-level classification task.

[0019] In one exemplary embodiment of this disclosure, the method further includes a method for training the video classification model, the method comprising:

[0020] Various types of information from the training video are input into the video classification model to be trained. Through various feature extraction networks in the video classification model, video information features corresponding to various types of information from the training video are extracted respectively.

[0021] The feature fusion network in the video classification model is used to fuse the video information features of various types to obtain the fused video features of the training video.

[0022] The hybrid expert system in the video classification model processes the fused video features of the training video to obtain the classification features of each level of the training video corresponding to each level of category.

[0023] The classification features of each level of the training video are input into the classification networks corresponding to each level of the category to obtain the classification prediction results of each level of the training video.

[0024] Based on the classification prediction results of each level of the training video, the cross-entropy loss of each level of the classification network and the matching loss between each two hierarchical categories with a subordinate relationship are obtained respectively, and the overall loss is obtained based on the cross-entropy loss of each level and the matching loss.

[0025] The overall loss is used to train the various feature extraction networks, the feature fusion network, the hybrid expert system, and the classification networks at each level in the video classification model.

[0026] In one exemplary embodiment of this disclosure, obtaining the cross-entropy loss for each level of the classification network and the matching loss between every two hierarchical categories with a subordinate relationship based on the classification prediction results for each level of the training video includes:

[0027] Based on the classification prediction results at each level of the training video and the true labels of the training video, the cross-entropy loss at each level of the classification network is obtained.

[0028] Based on the classification prediction results of each level of the training video, the prediction probability distribution of each level category is obtained, and based on the prediction probability distribution of each pair of hierarchical categories with a subordinate relationship, the matching loss between each pair of hierarchical categories with a subordinate relationship is obtained.

[0029] In one exemplary embodiment of this disclosure, obtaining the overall loss based on the cross-entropy losses at each level and the matching loss includes:

[0030] The total matching loss is obtained by summing the matching losses between any two hierarchical categories that have a subordinate relationship;

[0031] The overall loss is obtained by weighting the losses at each level and the total matching loss.

[0032] According to a second aspect of this disclosure, a video classification apparatus is provided, comprising:

[0033] The video feature extraction module is configured to input various types of information from the video into a video classification model, and extract video information features corresponding to each type of information from the video through various feature extraction networks in the video classification model.

[0034] The video feature fusion module is configured to perform feature fusion on the video information features corresponding to each type of information through the feature fusion network in the video classification model to obtain the fused video features of the video.

[0035] The classification feature determination module is configured to process the fused video features through a hybrid expert system in the video classification model to obtain the classification features at each level corresponding to each category of the video.

[0036] The classification result determination module is configured to input the classification features of the video at each level into the classification networks corresponding to each level of the category, and obtain the classification results of the video at each level.

[0037] In one exemplary embodiment of this disclosure, the video feature extraction module includes:

[0038] The visual feature extraction unit is configured to extract multiple video images from the video, extract image features from the multiple video images respectively through a visual feature extraction network, and perform feature fusion on the image features from the multiple video images to obtain the visual features of the video, wherein the video images include the cover frame and multiple content frames of the video.

[0039] The text feature extraction unit is configured to acquire multiple video texts in the video, extract categorized text features from the multiple video texts through a text feature extraction network, and perform feature fusion on the categorized text features to obtain the text features of the video, wherein the video texts include title text, character text, and speech recognition text in the video;

[0040] The audio feature extraction unit is configured to extract audio vector features from the video and process the audio vector features through an audio feature extraction network to obtain the audio features of the video.

[0041] In one exemplary embodiment of this disclosure, the video feature extraction module further includes:

[0042] The video meta-feature extraction unit is configured to acquire multiple video meta-information of the video, extract the categorized video meta-features corresponding to each type of video meta-information, and perform feature fusion on the categorized video meta-features to obtain the video meta-features of the video, wherein the video meta-information includes video duration, video word frequency, and video type;

[0043] The author meta-feature extraction unit is configured to obtain the author information of the video, extract the categorical author meta-features corresponding to various types of author information, and perform feature fusion on the categorical author meta-features to obtain the author meta-features of the video. The author information includes the author type, author level, and author vertical category of the video author.

[0044] In one exemplary embodiment of this disclosure, the classification feature determination module includes:

[0045] The video feature differentiation unit is configured to input the fused video features into the hybrid expert system, and perform feature differentiation on the fused video features through multiple expert networks in the hybrid expert system to obtain classification features at each level of the video that respectively characterize the multi-level classification task.

[0046] In one exemplary embodiment of this disclosure, the video classification apparatus further includes a video classification model training module, the video classification model training module comprising:

[0047] The video feature extraction unit is configured to input various types of information from the training video into the video classification model to be trained, and extract video information features corresponding to various types of information from the training video through various feature extraction networks in the video classification model.

[0048] The video feature fusion unit is configured to perform feature fusion on various video information features through the feature fusion network in the video classification model to obtain the fused video features of the training video;

[0049] The classification feature determination unit is configured to process the fused video features of the training video through the hybrid expert system in the video classification model to obtain the classification features of each level of the training video corresponding to each level of category.

[0050] The classification result prediction unit is configured to input the classification features of each level of the training video into the classification networks corresponding to each level of the category, and obtain the classification prediction results of each level of the training video.

[0051] The network loss determination unit is configured to perform the following operations: obtaining the cross-entropy loss of each level of the classification network corresponding to each level based on the classification prediction results of each level of the training video, and the matching loss between each pair of hierarchical categories with a subordinate relationship; and obtaining the overall loss based on the cross-entropy loss of each level and the matching loss.

[0052] The classification model training unit is configured to train various feature extraction networks, the feature fusion network, the hybrid expert system, and the classification networks at each level in the video classification model using the overall loss.

[0053] In one exemplary embodiment of this disclosure, the network loss determination unit includes:

[0054] Each level of loss determination unit is configured to perform the calculation of each level of cross-entropy loss corresponding to the classification network based on the classification prediction results of each level of the training video and the true labels of the training video.

[0055] The matching loss determination unit is configured to perform operations to obtain the prediction probability distribution of each level category based on the classification prediction results of each level of the training video, and to obtain the matching loss between each pair of hierarchical categories with a subordinate relationship based on the prediction probability distribution corresponding to each pair of hierarchical categories with a subordinate relationship.

[0056] In one exemplary embodiment of this disclosure, the network loss determination unit further includes:

[0057] The total matching loss determination unit is configured to perform a total matching loss by summing the matching losses between every two hierarchical categories with a subordinate relationship;

[0058] The overall loss determination unit is configured to perform a weighted summation of the losses at each level and the total matching loss to obtain the overall loss.

[0059] According to a third aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the video classification method described in any of the preceding claims.

[0060] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the video classification method described in any of the preceding claims.

[0061] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the video classification method described in any one of the preceding claims.

[0062] The exemplary embodiments disclosed herein can have the following beneficial effects:

[0063] In the video classification method of this exemplary embodiment, various types of information in the video are input into a video classification model. Various feature extraction networks within the video classification model extract video information features corresponding to each type of information. Then, a feature fusion network within the video classification model fuses these video information features to obtain fused video features. A hybrid expert system within the video classification model processes these fused video features to obtain classification features corresponding to each level of the video's categories. Finally, these classification features are input into the classification networks corresponding to each level of the categories to obtain the classification results for each level of the video. This video classification method, on the one hand, improves the comprehensiveness of feature extraction by extracting multiple modal features from various types of video information, thereby enhancing the accuracy of video classification. On the other hand, by using a hybrid expert system to differentiate the fused video features, classification features at each level representing different levels of classification tasks can be obtained, thus outputting multi-level categories of the video at once, improving video classification efficiency and reducing resource consumption costs.

[0064] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0065] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0066] Figure 1 A flowchart illustrating a video classification method according to an exemplary embodiment of this disclosure is shown;

[0067] Figure 2 A flowchart illustrating the extraction of video information features corresponding to various types of information in a video according to an exemplary embodiment of this disclosure is shown.

[0068] Figure 3 A flowchart illustrating a training method for a video classification model according to an exemplary embodiment of the present disclosure is shown.

[0069] Figure 4 A flowchart illustrating the process of determining the cross-entropy loss at each level and the matching loss between categories at each level in an exemplary embodiment of this disclosure is shown.

[0070] Figure 5 A flowchart of a two-level category video classification method according to a specific embodiment of this disclosure is shown;

[0071] Figure 6 A block diagram of a video classification apparatus according to an exemplary embodiment of the present disclosure is shown;

[0072] Figure 7 A schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure is shown. Detailed Implementation

[0073] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0074] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein.

[0075] The following exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0076] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0077] In some relevant embodiments, for video classification tasks, deep learning can be used to extract the spatiotemporal features of the video using relevant techniques, and the fused features can then be used to output the video classification result. For example, 3D and 2D convolutions can be used separately to extract features of the video in the temporal and spatial directions, which can then be concatenated and classified to output the video category.

[0078] While the methods described in the above embodiments can identify video category information, they have the following drawbacks:

[0079] 1. High training and deployment costs

[0080] The above method cannot directly output the multi-level categories of the video end-to-end in one go. Instead, it requires training a separate model for each level of classification task, which has a large training cost and a large overall resource consumption cost when the service is launched.

[0081] 2. Multimodal information of the video was not considered.

[0082] In addition to visual modalities, videos also contain information on audio, text, video meta, author meta, and other modalities. The methods mentioned above cannot effectively model these modalities.

[0083] 3. The hierarchical relationship of categories was not considered.

[0084] The multi-level classification of videos has a clear hierarchical relationship. This independent classification scheme cannot model this hierarchical relationship, which leads to certain limitations in the overall performance indicators.

[0085] To address the aforementioned problems, this exemplary implementation first provides a video classification method. (Reference) Figure 1 As shown, the above video classification method may include the following steps:

[0086] Step S110. Input various types of information from the video into the video classification model, and extract the video information features corresponding to each type of information from the video through various feature extraction networks in the video classification model.

[0087] Step S120. The video information features corresponding to each type of information are fused using the feature fusion network in the video classification model to obtain the fused video features.

[0088] Step S130. The fused video features are processed by the hybrid expert system in the video classification model to obtain the classification features corresponding to each level of the video category.

[0089] Step S140. Input the classification features of each level of the video into the classification networks corresponding to each level of the category to obtain the classification results of each level of the video.

[0090] In the video classification method of this exemplary embodiment, various types of information in the video are input into a video classification model. Various feature extraction networks within the video classification model extract video information features corresponding to each type of information. Then, a feature fusion network within the video classification model fuses these video information features to obtain fused video features. A hybrid expert system within the video classification model processes these fused video features to obtain classification features corresponding to each level of the video's categories. Finally, these classification features are input into the classification networks corresponding to each level of the category to obtain the classification results for each level of the video. This video classification method, in its exemplary embodiment, improves the comprehensiveness of feature extraction by extracting multiple modal features from various types of video information, thereby enhancing the accuracy of video classification. Furthermore, by using a hybrid expert system to differentiate the fused video features, classification features at each level representing different levels of classification tasks can be obtained, enabling the output of multi-level video categories at once, improving video classification efficiency, and reducing resource consumption costs.

[0091] Below, in conjunction with Figures 2 to 4 The steps described above in this example implementation will be explained in more detail.

[0092] In step S110, various types of information from the video are input into the video classification model, and various feature extraction networks in the video classification model are used to extract the video information features corresponding to each type of information in the video.

[0093] In this example implementation, the various types of information in the video can include visual information, text information, audio information, video meta-information, author meta-information, etc. The video information features corresponding to each type of information refer to the features of each modality corresponding to each type of information, including visual features, text features, audio features, video meta-features, and author meta-features. The feature extraction network can include visual feature extraction networks, text feature extraction networks, and audio feature extraction networks, etc.

[0094] In this example implementation, as Figure 2 As shown, the method for extracting video information features corresponding to various types of information in a video can specifically include the following steps:

[0095] Step S210. Extract multiple frames of video images from the video, extract image features from the multiple frames of video images respectively through a visual feature extraction network, and perform feature fusion on the image features from the multiple frames of video images to obtain the visual features of the video.

[0096] In this example implementation, the video image may include a cover frame and multiple content frames. For instance, the cover frame and four content frames can be extracted first, and then a visual feature extraction network, such as the swinT-V2 (shiftwindows transformer V2) model, can be used to extract image features from the aforementioned five frames. These features are then fused to obtain the visual features of the video.

[0097] Step S220. Obtain multiple video texts from the video, extract categorized text features from the multiple video texts using a text feature extraction network, and perform feature fusion on the categorized text features to obtain the text features of the video.

[0098] In this example implementation, the video text may include the title text, character text, and speech-recognized text from the video. The character text can be obtained using OCR (Optical Character Recognition) technology, and the speech-recognized text can be obtained using ASR (Automatic Speech Recognition). A text feature extraction network, such as mT5-encoder (multilingual-T5-encoder), extracts the categorical text features of the aforementioned three types of text, and these features are then fused to obtain the video's text features.

[0099] Step S230. Extract audio vector features from the video, and process the audio vector features through an audio feature extraction network to obtain the audio features of the video.

[0100] In this example implementation, an audio feature extraction network, such as a Transformer-encoder, can be used to extract audio features. Specifically, the VGGish (audio VGG model) features of the audio can be extracted first, and then the Transformer-encoder can be used to process the VGGish features to obtain the video audio features.

[0101] Step S240. Obtain multiple video meta-information of the video, extract the classification video meta-features corresponding to each type of video meta-information, and perform feature fusion on the classification video meta-features to obtain the video meta-features of the video.

[0102] In this example implementation, video metadata may include video duration, video word frequency, and video type. Video metadata can be extracted using a lookup table method. After discretizing the video metadata, the lookup table method is used to extract the corresponding categorized video metadata for each type of video metadata, and the resulting categorized video metadata is obtained after fusion.

[0103] Step S250. Obtain the author information of the video, extract the corresponding category author meta-features of each type of author information, and perform feature fusion on the category author meta-features to obtain the author meta-features of the video.

[0104] In this example implementation, author information may include the video author's author type, author level, and author category. Author meta-features can also be extracted using a lookup table method. After discretizing the author information for each type, the corresponding categorical author meta-features are extracted using a lookup table method, and then fused to obtain the author meta-features.

[0105] In step S120, the feature fusion network in the video classification model performs feature fusion on the video information features corresponding to each type of information to obtain the fused video features.

[0106] In this example implementation, after extracting the video information features corresponding to various types of information in the video, the feature fusion network in the video classification model can be used to fuse these video information features and output fused video features representing the entire video. The feature fusion network in the video classification model is a network structure with an attention mechanism, which can capture the relationships between different modalities, thereby achieving feature fusion.

[0107] In step S130, the fused video features are processed by the hybrid expert system in the video classification model to obtain the classification features corresponding to each level of the video category.

[0108] In this example implementation, a Hybrid Expert System (MMOE) in a video classification model can be used to process the fused video features obtained in the above steps. The MMOE includes multi-level classification tasks corresponding to multiple categories and multiple expert networks. By inputting the fused video features into the MMOE, and through feature differentiation by the multiple expert networks within the MMOE, classification features at each level representing the multi-level classification tasks can be obtained. Compared to the original video features, the classification features at each level corresponding to each category can better represent the related multi-level classification tasks.

[0109] In step S140, the classification features of the video at each level are input into the classification networks corresponding to each level of the category to obtain the classification results of the video at each level.

[0110] The video classification method in this example implementation can directly output multi-level video categories. Each level of category is predefined. Taking a two-level category as an example, the first-level category can represent a broader classification, such as "Sports". Under the first-level category, the second-level categories represent more detailed classifications, such as "Basketball" and "Soccer". Alternatively, the first-level category could be "Games", and the second-level categories could be "Shooting Games" and "Competitive Games". Adjacent categories have a parent-child relationship.

[0111] In this example implementation, the classification features corresponding to each level of the video category are input into the corresponding classification network at each level to obtain the classification results of the video at each level.

[0112] In addition to the above, the video classification method provided in this example embodiment also includes a training method for the video classification model, such as... Figure 3 As shown, the training method for a video classification model can specifically include the following steps:

[0113] Step S310. Input various types of information from the training video into the video classification model to be trained, and extract the video information features corresponding to each type of information from the training video through various feature extraction networks in the video classification model.

[0114] The video information features corresponding to various types of information in the training video mainly include visual features, text features, audio features, video meta-features, and author meta-features. The specific feature extraction methods are similar to those mentioned above and will not be repeated here.

[0115] Step S320. Use the feature fusion network in the video classification model to fuse the features of various video information to obtain the fused video features of the training video.

[0116] By fusing features of various video information features through the feature fusion network in the video classification model, the fused video features of the training video can be obtained.

[0117] Step S330. The fused video features of the training video are processed by the hybrid expert system in the video classification model to obtain the classification features of each level of the training video corresponding to each level of category.

[0118] The fused video features of the training videos are input into the hybrid expert system in the video classification model to be trained, and the system outputs the classification features corresponding to each level of the training video for each level of category.

[0119] Step S340. Input the classification features of each level of the training video into the classification networks corresponding to each level of the category to obtain the classification prediction results of each level of the training video.

[0120] The classification features of each level of the training video output by the hybrid expert system are input into the classification networks corresponding to each level of the category to obtain the classification prediction results of each level of the training video. The prediction results are the probability values ​​of each category.

[0121] Step S350. Based on the classification prediction results of each level of the training video, obtain the cross-entropy loss of each level of the classification network, the matching loss between each two hierarchical categories with a subordinate relationship, and obtain the overall loss based on the cross-entropy loss and the matching loss of each level.

[0122] In this example implementation, as Figure 4 As shown, based on the classification prediction results of each level of the training video, the cross-entropy loss corresponding to each level of the classification network and the matching loss between each pair of hierarchical categories with a subordinate relationship are obtained. Specifically, this may include the following steps:

[0123] Step S410. Based on the classification prediction results of each level of the training video and the real labels of the training video, obtain the cross-entropy loss of each level of the classification network.

[0124] For each individual level of the classification task, cross-entropy loss can be used to model the corresponding loss. Taking a two-level classification task as an example, the first-level cross-entropy loss and the second-level cross-entropy loss are as follows:

[0125]

[0126]

[0127] Where N is the number of samples, Y represents the probability value corresponding to the prediction result, and m is the index of the true value of the nth sample. For example, if there are 30 categories under the first-level category, labeled with numbers from 0 to 29, m refers to the index of the nth sample within that category.

[0128] Step S420. Based on the classification prediction results of each level of the training video, obtain the prediction probability distribution of each level category, and based on the prediction probability distribution of each pair of subordinate level categories, obtain the matching loss between each pair of subordinate level categories.

[0129] To better model the hierarchical relationships between categories, this example implementation proposes a matching loss. Taking a two-level category hierarchy as an example, the matching loss L between the first-level and second-level categories with a hierarchical relationship is... match The calculation formula is:

[0130]

[0131]

[0132] Where i is the first-level category index, j is the second-level category index, and P 1 ={P1 1 P2 1 ,…,P r 1} represents the probability distribution of the first-level classification head output, r is the number of first-level categories, and P is the probability distribution of the first-level classification head output. 2 ={P1 2 P2 2 ,…,P s 2 Let} represent the probability distribution of the second-level classification head output, and s represent the number of second-level categories. TopK refers to selecting the top K largest data points from n data points, and children represent the sub-second-level categories. Matching loss can make the parent-child relationship between first-level and second-level categories more consistent.

[0133] In this example implementation, the total matching loss is obtained by summing the matching losses between every two hierarchical categories with a subordinate relationship, and the overall loss is obtained by weighting the losses at each level and the total matching loss, i.e.:

[0134] L=α*L1+β*L2+γ*L match

[0135] Where α, β, and γ are weighting coefficients, which can be determined according to business needs. If there are only two levels of categories, then L match This represents the matching loss between first-level and second-level categories. If multiple levels of categories exist, then L... match This is the sum of the matching losses between any two hierarchical categories that have a subordinate relationship.

[0136] In this example implementation, by constructing various different losses, especially matching losses, the dependencies between different levels of tasks can be learned, which can make video classification more accurate.

[0137] Step S360. Train the various feature extraction networks, feature fusion networks, hybrid expert systems, and classification networks at all levels in the video classification model using the overall loss.

[0138] Finally, the video classification model is trained using the overall loss, including various feature extraction networks, feature fusion networks, hybrid expert systems, and classification networks at all levels, to obtain the trained video classification model.

[0139] like Figure 5 The diagram shown is a flowchart of a two-level category video classification method according to a specific embodiment of this disclosure. It illustrates the above steps in this example embodiment. The specific content of the flowchart is as follows:

[0140] First, extract features corresponding to various modalities from the video, including visual features, text features, audio features, video meta-features, and author meta-features.

[0141] Secondly, the modality fusion module is used to fuse the above features and output video features representing the entire video. This model fusion module is a network structure with an attention mechanism, which can capture the relationships between different modalities, thereby achieving fusion.

[0142] Then, a hybrid expert system is used to process the video features. This system comprises two tasks and five expert networks, each with a multilayer perceptron (MLP) structure. The hybrid expert system differentiates the video features, obtaining relevant features representing the primary and secondary classification tasks respectively. Compared to the original video features, this provides a better representation of the relevant tasks. These multi-level task features are then input into the corresponding classification networks to obtain the two-layer classification prediction results.

[0143] Finally, we construct multi-level cross-entropy loss and matching loss to learn the dependencies between tasks at different levels.

[0144] After testing, the video classification method in this example embodiment significantly improves the recognition performance (accuracy) of the first-level classification task by approximately 15 percentage points (pp) compared to related technical solutions; significantly improves the recognition performance (accuracy) of the second-level classification task by approximately 20 pp; and requires less machine resources for online services, approximately 40%.

[0145] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0146] It should be noted that the user information involved in this disclosure (including but not limited to user device information, user personal information, etc.) is all information authorized by the user or fully authorized by all parties.

[0147] Furthermore, this disclosure also provides a video classification device. (See reference) Figure 6 As shown, the video classification device may include a video feature extraction module 610, a video feature fusion module 620, a classification feature determination module 630, and a classification result determination module 640. Wherein:

[0148] The video feature extraction module 610 is configured to input various types of information in the video into the video classification model, and extract the video information features corresponding to each type of information in the video through various feature extraction networks in the video classification model;

[0149] The video feature fusion module 620 is configured to perform feature fusion on the video information features corresponding to various types of information through the feature fusion network in the video classification model to obtain the fused video features of the video.

[0150] The classification feature determination module 630 is configured to process the fused video features through a hybrid expert system in the video classification model to obtain the classification features corresponding to each level of the video category.

[0151] The classification result determination module 640 is configured to input the classification features of each level of the video into the classification networks corresponding to each level of the category, and obtain the classification results of each level of the video.

[0152] In some exemplary embodiments of this disclosure, the video feature extraction module 610 may include a visual feature extraction unit, a text feature extraction unit, and an audio feature extraction unit. Wherein:

[0153] The visual feature extraction unit is configured to extract multiple video images from the video, extract image features from the multiple video images through the visual feature extraction network, and perform feature fusion on the image features from the multiple video images to obtain the visual features of the video. The video images include the cover frame and multiple content frames of the video.

[0154] The text feature extraction unit is configured to acquire multiple video texts in the video, extract categorized text features from the multiple video texts through the text feature extraction network, and perform feature fusion on the categorized text features to obtain the text features of the video. The video text includes the title text, character text and speech recognition text in the video.

[0155] The audio feature extraction unit is configured to extract audio vector features from the video and process the audio vector features through the audio feature extraction network to obtain the audio features of the video.

[0156] In some exemplary embodiments of this disclosure, the video feature extraction module 610 may further include a video meta-feature extraction unit and an author meta-feature extraction unit. Wherein:

[0157] The video meta-feature extraction unit is configured to acquire multiple video meta-information of a video, extract the classification video meta-features corresponding to each type of video meta-information, and perform feature fusion on the classification video meta-features to obtain the video meta-features of the video. The video meta-information includes video duration, video word frequency, and video type.

[0158] The author meta-feature extraction unit is configured to obtain the author information of the video, extract the corresponding categorical author meta-features for each type of author information, and perform feature fusion on the categorical author meta-features to obtain the author meta-features of the video. The author information includes the author type, author level, and author vertical category of the video author.

[0159] In some exemplary embodiments of this disclosure, the classification feature determination module 630 may include a video feature differentiation unit, configured to input fused video features into a hybrid expert system, and perform feature differentiation on the fused video features through multiple expert networks in the hybrid expert system to obtain classification features at each level that respectively characterize the video for the multi-level classification task.

[0160] In some exemplary embodiments of this disclosure, a video classification apparatus provided by this disclosure may further include a video classification model training module. This video classification model training module may include a video feature extraction unit, a video feature fusion unit, a classification feature determination unit, a classification result prediction unit, a network loss determination unit, and a classification model training unit. Wherein:

[0161] The video feature extraction unit is configured to input various types of information from the training video into the video classification model to be trained, and extract the video information features corresponding to each type of information in the training video through various feature extraction networks in the video classification model.

[0162] The video feature fusion unit is configured to fuse various video information features through the feature fusion network in the video classification model to obtain the fused video features of the training video;

[0163] The classification feature determination unit is configured to process the fused video features of the training video through a hybrid expert system in the video classification model to obtain the classification features of each level of the training video corresponding to each level of category.

[0164] The classification result prediction unit is configured to input the classification features of each level of the training video into the classification networks corresponding to each level of the category, and obtain the classification prediction results of each level of the training video.

[0165] The network loss determination unit is configured to obtain the cross-entropy loss of each level of the classification network and the matching loss between each two hierarchical categories with a subordinate relationship based on the classification prediction results of each level of the training video, and obtain the overall loss based on the cross-entropy loss and the matching loss.

[0166] The classification model training unit is configured to train various feature extraction networks, feature fusion networks, hybrid expert systems, and classification networks at all levels in the video classification model using the overall loss.

[0167] In some exemplary embodiments of this disclosure, the network loss determination unit may include loss determination units at various levels and a matching loss determination unit. Wherein:

[0168] Each loss determination unit is configured to obtain the cross-entropy loss of each level of the classification network based on the classification prediction results of each level of the training video and the true labels of the training video.

[0169] The matching loss determination unit is configured to obtain the predicted probability distribution of each level category based on the classification prediction results of each level of the training video, and to obtain the matching loss between each pair of hierarchical categories with a subordinate relationship based on the predicted probability distribution of each pair of hierarchical categories with a subordinate relationship.

[0170] In some exemplary embodiments of this disclosure, the network loss determination unit may further include a sum-matching loss determination unit and an overall loss determination unit. Wherein:

[0171] The total matching loss determination unit is configured to obtain the total matching loss based on the sum of the matching losses between every two hierarchical categories with a subordinate relationship;

[0172] The overall loss determination unit is configured to obtain the overall loss by weighted sum of the losses at each level and the total matching loss.

[0173] The specific details of each module / unit in the above video classification device have been described in detail in the corresponding method embodiment section, and will not be repeated here.

[0174] Figure 7A schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure is shown.

[0175] It should be noted that, Figure 7 The computer system 700 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0176] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 702 or programs loaded from storage section 708 into random access memory (RAM) 703. The RAM 703 also stores various programs and data required for system operation. The CPU 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0177] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0178] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by central processing unit (CPU) 701, it performs various functions defined in the system of this application.

[0179] It should be noted that the computer-readable medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0180] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0181] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0182] It should be noted that although several modules for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided and embodied by multiple modules.

[0183] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0184] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A video classification method, characterized in that, include: Various types of information from the video are input into the video classification model, and various feature extraction networks in the video classification model are used to extract the video information features corresponding to each type of information in the video. The feature fusion network in the video classification model performs feature fusion on the video information features corresponding to each type of information to obtain the fused video features of the video. The fused video features are processed by the hybrid expert system in the video classification model to obtain the classification features corresponding to each level of the video category. The hybrid expert system includes multi-level classification tasks corresponding to multiple levels of categories and multiple expert networks; the step of processing the fused video features through the hybrid expert system in the video classification model to obtain the classification features at each level corresponding to each level of the video includes: inputting the fused video features into the hybrid expert system, and performing feature differentiation on the fused video features through multiple expert networks in the hybrid expert system to obtain the classification features at each level of the video that respectively represent the multi-level classification tasks; The classification features of the video at each level are input into the classification networks corresponding to each level of the category to obtain the classification results of the video at each level.

2. The video classification method according to claim 1, characterized in that, The video information features include visual features, text features, and audio features. The feature extraction network includes a visual feature extraction network, a text feature extraction network, and an audio feature extraction network. The step of extracting video information features corresponding to each type of information in the video through the various feature extraction networks in the video classification model includes: Multiple video frames are extracted from the video, and image features are extracted from the multiple video frames respectively through a visual feature extraction network. The image features in the multiple video frames are then fused to obtain the visual features of the video. The video frames include the cover frame and multiple content frames of the video. Multiple video texts from the video are obtained, and categorized text features are extracted from the multiple video texts through a text feature extraction network. The categorized text features are then fused to obtain the text features of the video. The video texts include title text, character text, and speech recognition text from the video. The audio vector features in the video are extracted, and the audio vector features are processed by an audio feature extraction network to obtain the audio features of the video.

3. The video classification method according to claim 2, characterized in that, The video information features also include video meta-features and author meta-features. Extracting the video information features corresponding to each type of information in the video includes: Multiple video meta-information of the video are obtained, and the corresponding classification video meta-features of each type of video meta-information are extracted. The classification video meta-features are then fused to obtain the video meta-features of the video. The video meta-information includes video duration, video word frequency, and video type. The author information of the video is obtained, the category author meta-features corresponding to each type of author information are extracted, and the category author meta-features are fused to obtain the author meta-features of the video. The author information includes the author type, author level and author vertical category of the video author.

4. The video classification method according to claim 1, characterized in that, The method also includes a training method for the video classification model, the training method for the video classification model comprising: Various types of information from the training video are input into the video classification model to be trained. Through various feature extraction networks in the video classification model, video information features corresponding to various types of information from the training video are extracted respectively. The feature fusion network in the video classification model is used to fuse the video information features of various types to obtain the fused video features of the training video. The hybrid expert system in the video classification model processes the fused video features of the training video to obtain the classification features of each level of the training video corresponding to each level of category. The classification features of each level of the training video are input into the classification networks corresponding to each level of the category to obtain the classification prediction results of each level of the training video. Based on the classification prediction results of each level of the training video, the cross-entropy loss of each level of the classification network and the matching loss between each two hierarchical categories with a subordinate relationship are obtained respectively, and the overall loss is obtained based on the cross-entropy loss of each level and the matching loss. The overall loss is used to train the various feature extraction networks, the feature fusion network, the hybrid expert system, and the classification networks at each level in the video classification model.

5. The video classification method according to claim 4, characterized in that, The step of obtaining the cross-entropy loss for each level of the classification network corresponding to each level based on the classification prediction results of the training video, and the matching loss between each pair of hierarchical categories with a subordinate relationship, includes: Based on the classification prediction results at each level of the training video and the true labels of the training video, the cross-entropy loss at each level of the classification network is obtained. Based on the classification prediction results of each level of the training video, the prediction probability distribution of each level category is obtained, and based on the prediction probability distribution of each pair of hierarchical categories with a subordinate relationship, the matching loss between each pair of hierarchical categories with a subordinate relationship is obtained.

6. The video classification method according to claim 5, characterized in that, The process of obtaining the overall loss based on the cross-entropy loss at each level and the matching loss includes: The total matching loss is obtained by summing the matching losses between any two hierarchical categories that have a subordinate relationship; The overall loss is obtained by weighting the cross-entropy loss at each level and the total matching loss.

7. A video classification device, characterized in that, include: The video feature extraction module is configured to input various types of information from the video into a video classification model, and extract video information features corresponding to each type of information from the video through various feature extraction networks in the video classification model. The video feature fusion module is configured to perform feature fusion on the video information features corresponding to each type of information through the feature fusion network in the video classification model to obtain the fused video features of the video. The classification feature determination module is configured to process the fused video features through a hybrid expert system in the video classification model to obtain the classification features at each level corresponding to each category of the video. The hybrid expert system includes multi-level classification tasks corresponding to multiple levels of categories and multiple expert networks; the step of processing the fused video features through the hybrid expert system in the video classification model to obtain the classification features at each level corresponding to each level of the video includes: inputting the fused video features into the hybrid expert system, and performing feature differentiation on the fused video features through multiple expert networks in the hybrid expert system to obtain the classification features at each level of the video that respectively represent the multi-level classification tasks; The classification result determination module is configured to input the classification features of the video at each level into the classification networks corresponding to each level of the category, and obtain the classification results of the video at each level.

8. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the video classification method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, wherein instructions in the computer-readable storage medium, when executed by a processor of an electronic device, enable the electronic device to perform a video classification method as described in any one of claims 1 to 6.

Images