Multimodal classification model training, video classification method and apparatus, medium and device

Abstract

The present disclosure relates to a multi-modal classification model training, a video classification method and device, a medium and equipment. The multi-modal classification model comprises a plurality of classification heads, a feature extractor and a splicing module, the splicing module is connected with each classification head respectively; the multi-modal classification model training method comprises: for each classification head, extracting deep features and shallow features of a sample video through the feature extractor obtained after pre-training; generating splicing features through the splicing module according to the deep features and the shallow features; training the classification head according to the splicing features and the classification label. In this way, the classification tasks can share the feature extractor and the splicing module, which can reduce the resource consumption and the mutual influence of multiple downstream classification heads, and improve the model iteration flexibility. The classification head learns the deep features and the shallow features of the video at the same time, provides multi-perspective features for the classification head without adding new feature extraction units, and improves the classification effect of the multi-modal classification model.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a multimodal classification model training, video classification method, apparatus, medium, and device. Background Technology

[0002] With the development of computer technology, multimodal models can enhance cross-modal semantic alignment capabilities and connect the relationships between modalities. Compared with single-modal models such as text, speech, and vision, they can provide richer semantic representations. Therefore, multimodal models are increasingly being applied in content understanding business, such as video classification, i.e., multimodal models are video classification models. However, in the existing training process of multimodal models for classification, the following two methods are usually used: (1) For each classification task, a corresponding model is trained separately. However, as the number of classification tasks increases, the number of models deployed may expand to hundreds or thousands. The cost of modeling each task separately is too high and the resources are difficult to bear; (2) For all classification tasks, a unified classification model is trained. Compared with modeling each task independently, it can save resources. However, the model iteration is not flexible enough and the models between tasks are seriously coupled. Summary of the Invention

[0003] This summary section is provided to briefly introduce the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] In a first aspect, this disclosure provides a method for training a multimodal classification model. The multimodal classification model includes: multiple classification heads corresponding one-to-one with multiple classification tasks, a feature extractor and a concatenation module connected in series, wherein the concatenation module is connected to each of the classification heads. The method includes: for each classification head, obtaining a sample video and a classification label of the sample video under the classification task corresponding to the classification head; extracting deep features and shallow features of the sample video through the pre-trained feature extractor; generating concatenated features through the concatenation module based on the deep features and the shallow features; and training the classification head based on the concatenated features and the classification label.

[0005] Secondly, this disclosure provides a video classification method, comprising: acquiring a video to be classified; generating classification results of the video to be classified under different classification tasks using a pre-trained multimodal classification model, wherein the multimodal classification model is trained using the multimodal classification model training method provided in the first aspect of this disclosure.

[0006] Thirdly, this disclosure provides a multimodal classification model training device. The multimodal classification model includes: multiple classification heads corresponding one-to-one with multiple classification tasks, a feature extractor and a concatenation module connected in series, wherein the concatenation module is connected to each of the classification heads respectively; the device includes: a first acquisition module, used to acquire a sample video and a classification label of the sample video under the classification task corresponding to the classification head for each classification head; a feature extraction module, used to extract deep features and shallow features of the sample video through a pre-trained feature extractor; a feature processing module, used to generate concatenated features through the concatenation module based on the deep features and the shallow features; and a classification head training module, used to train the classification head based on the concatenated features and the classification labels.

[0007] Fourthly, this disclosure provides a video classification apparatus, comprising: a second acquisition module for acquiring a video to be classified; and a classification module for generating classification results of the video to be classified under different classification tasks using a pre-trained multimodal classification model, wherein the multimodal classification model is trained using the multimodal classification model training method provided in the first aspect of this disclosure.

[0008] Fifthly, this disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the multimodal classification model training method provided in the first aspect of this disclosure or the steps of the video classification method provided in the second aspect of this disclosure.

[0009] In a sixth aspect, this disclosure provides an electronic device, comprising: a storage device having a computer program stored thereon; and a processing device for executing the computer program in the storage device to implement the steps of the multimodal classification model training method provided in the first aspect of this disclosure or the steps of the video classification method provided in the second aspect of this disclosure.

[0010] In the above technical solution, the multimodal classification model includes multiple classification heads, a cascaded feature extractor, and a stitching module. The stitching module is connected to each classification head, allowing different classification tasks to share the feature extractor and stitching module. This reduces resource consumption and minimizes mutual interference among multiple downstream classification heads, improving model iteration flexibility. Furthermore, after pre-training the feature extractor of the multimodal classification model, the model parameters are frozen. The classification head is then trained based on the deep and shallow features of the sample video. This allows a simple classification head to simultaneously learn both deep and shallow features of the video, providing multi-view features without adding new feature extraction units, thus improving the classification performance of the multimodal classification model. Moreover, after the feature extractor pre-training is complete, only the model parameters of the classification head need to be adjusted, without adjusting the model parameters of the feature extractor, significantly reducing the amount of parameter tuning and improving model training efficiency. Furthermore, a single multimodal classification model can perform multi-dimensional classification of videos, saving the resource costs of running multimodal classification models and promoting their application in practical business.

[0011] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale. In the drawings:

[0013] Figure 1 This is a schematic diagram illustrating the current process of training and deploying multimodal models for classification.

[0014] Figure 2 This is a schematic diagram illustrating the structure of a multimodal classification model according to an exemplary embodiment.

[0015] Figure 3 This is a flowchart illustrating a multimodal classification model training method according to an exemplary embodiment.

[0016] Figure 4 This is a schematic diagram illustrating the process of training and deploying a multimodal classification model according to an exemplary embodiment.

[0017] Figure 5A This is a schematic diagram illustrating a multimodal classification model training process according to an exemplary embodiment.

[0018] Figure 5BThis is a schematic diagram illustrating a multimodal classification model training process according to another exemplary embodiment.

[0019] Figure 6 This is a flowchart illustrating a video classification method according to an exemplary embodiment.

[0020] Figure 7 This is a block diagram illustrating a multimodal classification model training apparatus according to an exemplary embodiment.

[0021] Figure 8 This is a block diagram illustrating a video classification device according to an exemplary embodiment.

[0022] Figure 9 This is a schematic diagram of the structure of an electronic device according to an exemplary embodiment. Detailed Implementation

[0023] As discussed in the background section, current multimodal models for classification suffer from a trade-off between resource consumption and flexibility in model iteration. Specifically, regarding the first model training method mentioned in the background section, such as... Figure 1 As shown, the typical approach involves using massive amounts of data (hundreds of millions) to perform a first-stage pre-training of the backbone network, resulting in a pre-trained model. Then, in various downstream business scenarios, different business data is used to fine-tune the corresponding linear head and pre-trained model based on different classification objectives (i.e., classification tasks). Finally, the pre-trained model and linear head, fine-tuned in the second stage, are deployed online. However, modeling each task individually is too costly, requires significant resources, and involves long training and deployment times. Furthermore, as... Figure 1 As shown in the deployment phase, each classification task requires deploying the pre-trained model and classification head together. The cost of deploying the model is too high, requiring a large amount of GPU resources, which is difficult for online resources to bear. In addition, the model training only uses the deep representation of the pre-trained model, missing a lot of shallow representation information, resulting in poor model classification performance.

[0024] In view of this, this disclosure provides a multimodal classification model training, video classification method, apparatus, medium and device to simultaneously take into account resource consumption and the flexibility of model iteration.

[0025] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0026] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0027] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0028] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0029] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0030] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0031] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0032] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0033] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0034] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0035] Meanwhile, it is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0036] Before introducing the training method of the multimodal classification model of this disclosure, the structure of the multimodal classification model in this disclosure will be described first. For example... Figure 2 As shown, a multimodal classification model can include: multiple classification heads that correspond one-to-one with multiple classification tasks (i.e., downstream business). Figure 2 The example uses N classification heads (N > 1), a feature extractor, and a concatenation module. The concatenation module is connected to each classification head, and the feature extractor is connected in series with the concatenation module. Each classification head can include one or more fully connected layers.

[0037] The feature extractor is used to extract deep and shallow features from the video; the stitching module is used to generate stitched features based on the deep and shallow features; and each classification head is used to generate the classification result for the corresponding classification task based on the stitched features.

[0038] For example, multiple classification tasks may include: classifying by video content, such as news, sports, music, movies, games, animation, education, documentaries, etc.; classifying by video format, such as advertisements, music videos, live broadcasts, dramas, short films, documentaries, radio dramas, etc.; classifying by video theme, such as food, travel, fashion, technology, art, culture, health, etc.; and classifying by whether a pre-defined object (e.g., the ocean, a house, etc.) exists in the video.

[0039] Figure 3 This is a flowchart illustrating a multimodal classification model training method according to an exemplary embodiment. For example... Figure 3 As shown, the training method for this multimodal classification model may include S101 to S104.

[0040] In S101, for each classifier head of the multimodal classification model, the sample video and the classification label of the sample video under the corresponding classification task of that classifier head are obtained.

[0041] In this disclosure, the sample video can be general video data or video data specific to the business scenario of the classification task corresponding to the classification head, such as... Figure 4 As shown, business data is used to train the classification heads corresponding to the respective classification tasks.

[0042] Preferably, the sample video is video data from the business scenario to which the classification task corresponding to the classification head belongs. This can improve the training efficiency and classification effect of the classification head.

[0043] In S102, deep and shallow features of the sample video are extracted using a feature extractor obtained after pre-training.

[0044] In this disclosure, before training each classifier head of the multimodal classification model, the feature extraction of the multimodal classification model needs to be pre-trained. Specifically, as follows: Figure 4 As shown, the feature extractor of the multimodal classification model can be pre-trained using a large amount of data. Then, based on the feature extractor obtained after training (hereinafter referred to as the pre-trained model), each classifier head is trained separately. During the training of the classifier head, the model parameters of the pre-trained model are frozen, and only the model parameters of the classifier head are adjusted.

[0045] In S103, spliced ​​features are generated by the splicing module of the multimodal classification model based on deep and shallow features.

[0046] In S104, the classification head is trained based on the splicing features and classification labels.

[0047] Specifically, the spliced ​​features can be input into the classification head to obtain the predicted classification result of the sample video under the classification task corresponding to the classification head; then, the model parameters of the classification head are adjusted based on the comparison result of the predicted classification result of the sample video under the classification task corresponding to the classification head and the classification label under the classification task corresponding to the classification head.

[0048] After the multimodal classification model has been trained, it can be deployed, such as... Figure 4 As shown, the pre-trained model and each classifier head are deployed separately, and the stitching module is also deployed separately from each classifier head. In this way, each classification task can share the pre-trained model and stitching module, which can not only reduce resource consumption, but also reduce the mutual influence of multiple downstream classifier heads and improve the flexibility of model iteration.

[0049] In the above technical solution, the multimodal classification model includes multiple classification heads, a cascaded feature extractor, and a stitching module. The stitching module is connected to each classification head, allowing different classification tasks to share the feature extractor and stitching module. This reduces resource consumption and minimizes mutual interference among multiple downstream classification heads, improving model iteration flexibility. Furthermore, after pre-training the feature extractor of the multimodal classification model, the model parameters are frozen. The classification head is then trained based on the deep and shallow features of the sample video. This allows a simple classification head to simultaneously learn both deep and shallow features of the video, providing multi-view features without adding new feature extraction units, thus improving the classification performance of the multimodal classification model. Moreover, after the feature extractor pre-training is complete, only the model parameters of the classification head need to be adjusted, without adjusting the model parameters of the feature extractor, significantly reducing the amount of parameter tuning and improving model training efficiency. Furthermore, a single multimodal classification model can perform multi-dimensional classification of videos, saving the resource costs of running multimodal classification models and promoting their application in practical business.

[0050] like Figure 2 , Figure 5A as well as Figure 5B As shown, the feature extractor of a multimodal classification model may include a cascaded backbone network and a fusion module, wherein the fusion module includes multiple cascaded modality fusion layers (wherein, Figure 2 , Figure 5A as well as Figure 5B (Taking the example of a fusion module consisting of two cascaded modal fusion layers as an illustration), the splicing module is connected to each modal fusion layer, and the backbone network is connected to each modal fusion layer (not shown in the figure).

[0051] In this disclosure, the backbone network is used to extract textual and visual features from sample videos and inputs the textual and visual features into each modality fusion layer.

[0052] like Figure 2 , Figure 5A as well as Figure 5B As shown, the backbone network can include a visual encoder and a text encoder. After acquiring the sample video, it can be sliced ​​(i.e., segmented). Then, for each video segment, the video can be shortened and frames extracted. Next, the extracted video frames are input into the visual encoder to obtain the corresponding visual features. Simultaneously, text information (including the sample video title, abstracted video frame subtitles, and the extracted optical character recognition results of the video frames) is extracted from the extracted video frames and input into the text encoder to obtain the text features of that video segment.

[0053] In this disclosure, the backbone network can be a multimodal model such as a contrastive language-image pre-training (CLIP) model based on a shifted window (Swin) or a visual-linguistic BERT (VL-BERT) model. Specifically, when the backbone network is a Swin-based CLIP model, the text encoder is a bidirectional encoder representation from transformers (BERT) model, and the visual encoder is a Swin Transformer model (a novel Transformer framework).

[0054] When the backbone network is a Swing-based CLIP model, the multimodal classification model has a better generalization effect because there is no batch normalization layer in the model. Furthermore, all modules in this backbone network only need to propagate features once, which greatly reduces the resource consumption of the multimodal classification model.

[0055] Each modality fusion layer in the fusion module is used to fuse the features input to that modality fusion layer to obtain deep or shallow features, and then input the deep or shallow features into the next modality fusion layer and the concatenation module. Between two adjacent modality fusion layers, the output of the previous modality fusion layer serves as the input of the next modality fusion layer. At the same time, the backbone network inputs the extracted text features and visual features into each modality fusion layer.

[0056] For example, such as Figure 2 , Figure 5A As shown in 5B, the fusion module includes a modal fusion layer E and a modal fusion layer F. Modal fusion layer F is used to fuse the text features and visual features input into this layer to obtain shallow features, and input the shallow features into modal fusion layer E and the splicing module. Modal fusion layer E fuses the shallow features input from modal fusion layer F, the text features and visual features input from the backbone network to obtain deep features, and inputs the deep features into the splicing module.

[0057] For example, each modality fusion layer can use the Transformer framework.

[0058] In the above implementation, the features generated by each modality fusion layer in the multimodal classification model are concatenated and then used to train the classification head. In this way, the simple classification head can learn both deep and shallow features of the video at the same time, that is, it can learn features at different levels of the video simultaneously. Without adding new feature extraction units, it provides multi-view features for the classification head, thereby improving the classification effect of the multimodal classification model.

[0059] The following provides a detailed explanation of the specific implementation method for generating spliced ​​features based on deep and shallow features using the splicing module in S103 above. Specifically, this can be achieved in various ways; in one implementation, such as... Figure 5A As shown, the classification token cls-token in the deep features or the cls-token in the shallow features of each modality fusion layer can be concatenated using the concatenation module to obtain concatenated features.

[0060] In this disclosure, the various classification tags cls-tokens input into the concatenation module can be concatenated horizontally. This disclosure does not specify the concatenation order of the cls-tokens. The classification tag cls-token refers to a representation in deep or shallow features that bears the [cls] label. It is a special token representing a vector representation of deep or shallow features. Deep or shallow features can include cls-tokens and embeddings.

[0061] In another implementation, such as Figure 5B As shown, the above-mentioned stitching module may include stitching units and multiple dimensionality reduction units corresponding one-to-one with multiple modality fusion layers. Each dimensionality reduction unit is connected to its corresponding modality fusion layer, and each stitching unit is connected to each dimensionality reduction unit. The modality fusion layer is used to input the classification token (cls-token) from the deep features or the cls-token from the shallow features into its corresponding dimensionality reduction unit. Then, based on the deep and shallow features, the stitching module can generate stitched features in the following way:

[0062] Each dimensionality reduction unit performs dimensionality reduction processing on the cls-token in the deep features or the cls-token in the shallow features of the input modality fusion layer corresponding to it.

[0063] The splicing unit splices the cls-tokens obtained after each dimensionality reduction process to obtain the spliced ​​features.

[0064] For example, the dimensionality reduction unit maps a 768-dimensional cls-token to a 128-dimensional cls-token.

[0065] For example, Figure 5BAs shown, the fusion module includes a modal fusion layer E and a modal fusion layer F. The stitching module includes a stitching unit, a dimensionality reduction unit G corresponding to the modal fusion layer E, and a dimensionality reduction unit H corresponding to the modal fusion layer F. The dimensionality reduction unit G is connected to its corresponding modal fusion layer E, the dimensionality reduction unit H is connected to its corresponding modal fusion layer F, and the stitching unit is connected to the dimensionality reduction unit G and the dimensionality reduction unit H, respectively.

[0066] The modality fusion layer F is used to fuse the text features and visual features input into this layer to obtain shallow features, and then inputs the shallow features into the modality fusion layer E. The cls-token in the shallow features is input into the dimensionality reduction unit H. The modality fusion layer E fuses the shallow features input from the modality fusion layer F, the text features and visual features input from the backbone network to obtain deep features, and then inputs the cls-token in the deep features into the dimensionality reduction unit G.

[0067] Dimensionality reduction unit H is used to reduce the dimensionality of the cls-token in the shallow features input to the modality fusion layer F; dimensionality reduction unit G is used to reduce the dimensionality of the cls-token in the deep features input to the modality fusion layer E. The concatenation unit concatenates the cls-token obtained after dimensionality reduction by dimensionality reduction unit H and the cls-token obtained after dimensionality reduction by dimensionality reduction unit G to obtain the concatenated features.

[0068] For example, the dimensionality reduction unit mentioned above can be a single-layer neural network.

[0069] In the above implementation, the bandwidth can be reduced by concatenating the cls-token in the deep features and the cls-token in the shallow features after dimensionality reduction.

[0070] In addition, when the stitching module includes dimensionality reduction units, the model parameters of each dimensionality reduction unit need to be adjusted simultaneously with the model parameters of the classification head to ensure that the cls-token output by the dimensionality reduction unit meets the preset dimensionality requirements. Note that the model parameters of the feature extractor are frozen while updating the model parameters of the classification head and dimensionality reduction units. Figure 5B The asterisk (*) in the figure indicates that the model parameters of the frozen feature extractor are frozen. Specifically, the above method may further include the following steps:

[0071] For each dimensionality reduction unit, adjust the model parameters of that dimensionality reduction unit according to the dimension of the cls-token input into that dimensionality reduction unit.

[0072] Figure 6 This is a flowchart illustrating a video classification method according to an exemplary embodiment. Figure 6 As shown, the video classification method may include S201 and S202.

[0073] In S201, obtain the video to be classified.

[0074] In S202, a pre-trained multimodal classification model is used to generate classification results for the video to be classified under different classification tasks.

[0075] In this disclosure, the multimodal classification model is trained using the multimodal classification model training method provided above.

[0076] In the above technical solution, the multimodal classification model includes multiple classification heads, a cascaded feature extractor, and a stitching module. The stitching module is connected to each classification head, allowing different classification tasks to share the feature extractor and stitching module. This reduces resource consumption and minimizes mutual interference among multiple downstream classification heads, improving model iteration flexibility. Furthermore, after pre-training the feature extractor of the multimodal classification model, the model parameters are frozen. The classification head is then trained based on the deep and shallow features of the sample video. This allows a simple classification head to simultaneously learn both deep and shallow features of the video, providing multi-view features without adding new feature extraction units, thus improving the classification performance of the multimodal classification model. Moreover, after the feature extractor pre-training is complete, only the model parameters of the classification head need to be adjusted, without adjusting the model parameters of the feature extractor, significantly reducing the amount of parameter tuning and improving model training efficiency. Furthermore, a single multimodal classification model can perform multi-dimensional classification of videos, saving the resource costs of running multimodal classification models and promoting their application in practical business.

[0077] Figure 7 This is a block diagram illustrating a multimodal classification model training apparatus according to an exemplary embodiment.

[0078] The multimodal classification model includes: multiple classification heads corresponding one-to-one with multiple classification tasks, a feature extractor and a concatenation module connected in series, wherein the concatenation module is connected to each of the classification heads. Figure 7 As shown, the multimodal classification model training device 300 includes:

[0079] The first acquisition module 301 is used to acquire a sample video and a classification label for the sample video under the classification task corresponding to the classification head for each classification head; the feature extraction module 302 is used to extract deep features and shallow features of the sample video through a feature extractor obtained after pre-training; the feature processing module 303 is used to generate spliced ​​features through the splicing module based on the deep features and the shallow features; and the classification head training module 304 is used to train the classification head based on the spliced ​​features and the classification label.

[0080] In the above technical solution, the multimodal classification model includes multiple classification heads, a cascaded feature extractor, and a stitching module. The stitching module is connected to each classification head, allowing different classification tasks to share the feature extractor and stitching module. This reduces resource consumption and minimizes mutual interference among multiple downstream classification heads, improving model iteration flexibility. Furthermore, after pre-training the feature extractor of the multimodal classification model, the model parameters are frozen. The classification head is then trained based on the deep and shallow features of the sample video. This allows a simple classification head to simultaneously learn both deep and shallow features of the video, providing multi-view features without adding new feature extraction units, thus improving the classification performance of the multimodal classification model. Moreover, after the feature extractor pre-training is complete, only the model parameters of the classification head need to be adjusted, without adjusting the model parameters of the feature extractor, significantly reducing the amount of parameter tuning and improving model training efficiency. Furthermore, a single multimodal classification model can perform multi-dimensional classification of videos, saving the resource costs of running multimodal classification models and promoting their application in practical business.

[0081] Optionally, the feature extractor includes a cascaded backbone network and a fusion module, wherein the fusion module includes multiple cascaded modal fusion layers, the splicing module is connected to each of the modal fusion layers respectively, and the backbone network is connected to each of the modal fusion layers respectively;

[0082] The backbone network is used to extract text features and visual features from the sample video, and input the text features and visual features into each modality fusion layer;

[0083] Each modality fusion layer is used to fuse the features input into the modality fusion layer to obtain deep features or shallow features, and input the deep features or shallow features into the next modality fusion layer and the splicing module.

[0084] Optionally, the feature processing module 303 is used to concatenate the classification marker cls-token in the deep features or the cls-token in the shallow features of each modality fusion layer input by the concatenation module to obtain concatenated features.

[0085] Optionally, the splicing module includes a splicing unit and a plurality of dimensionality reduction units corresponding one-to-one with the plurality of modality fusion layers, wherein the dimensionality reduction unit is connected to its corresponding modality fusion layer, the splicing unit is connected to each of the dimensionality reduction units respectively, and the modality fusion layer is used to input the classification tag cls-token in the deep features or the cls-token in the shallow features to its corresponding dimensionality reduction unit;

[0086] The feature processing module 303 includes:

[0087] The first processing submodule is used to perform dimensionality reduction processing on the cls-token in the deep features or the cls-token in the shallow features input by the modality fusion layer corresponding to each of the dimensionality reduction units.

[0088] The second processing submodule is used to concatenate the cls-tokens obtained after each dimensionality reduction process through the concatenation unit to obtain the concatenated features.

[0089] Optionally, the multimodal classification model training device 300 further includes:

[0090] The parameter adjustment module is used to adjust the model parameters of each dimensionality reduction unit according to the dimension of the cls-token input into the dimensionality reduction unit.

[0091] Optionally, the sample video is video data from the business scenario to which the classification task corresponding to the classification head belongs.

[0092] Figure 8 This is a block diagram illustrating a video classification apparatus according to an exemplary embodiment. Figure 8 As shown, the video classification device 400 includes:

[0093] The second acquisition module 401 is used to acquire the video to be classified.

[0094] The classification module 402 is used to generate classification results of the video to be classified under different classification tasks using a pre-trained multimodal classification model, wherein the multimodal classification model is trained using the multimodal classification model training method provided in this disclosure.

[0095] In the above technical solution, the multimodal classification model includes multiple classification heads, a cascaded feature extractor, and a stitching module. The stitching module is connected to each classification head, allowing different classification tasks to share the feature extractor and stitching module. This reduces resource consumption and minimizes mutual interference among multiple downstream classification heads, improving model iteration flexibility. Furthermore, after pre-training the feature extractor of the multimodal classification model, the model parameters are frozen. The classification head is then trained based on the deep and shallow features of the sample video. This allows a simple classification head to simultaneously learn both deep and shallow features of the video, providing multi-view features without adding new feature extraction units, thus improving the classification performance of the multimodal classification model. Moreover, after the feature extractor pre-training is complete, only the model parameters of the classification head need to be adjusted, without adjusting the model parameters of the feature extractor, significantly reducing the amount of parameter tuning and improving model training efficiency. Furthermore, a single multimodal classification model can perform multi-dimensional classification of videos, saving the resource costs of running multimodal classification models and promoting their application in practical business.

[0096] This disclosure also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the multimodal classification model training method or the video classification method described above.

[0097] The following is for reference. Figure 9 The diagram illustrates a structural schematic of an electronic device (e.g., a terminal device or a server) 600 suitable for implementing embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 9 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0098] like Figure 9As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0099] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 9 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0100] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0101] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0102] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0103] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0104] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: for each classifier in a multimodal classification model, acquire a sample video and a classification label for the sample video under the classification task corresponding to that classifier, wherein the multimodal classification model includes: multiple classifiers corresponding one-to-one with multiple classification tasks, a feature extractor and a stitching module connected in series, the stitching module being connected to each classifier respectively; extract deep features and shallow features of the sample video through the pre-trained feature extractor; generate stitched features through the stitching module based on the deep features and the shallow features; and train the classifier based on the stitched features and the classification label.

[0105] Alternatively, the aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire a video to be classified; and generate classification results of the video to be classified under different classification tasks using a pre-trained multimodal classification model, wherein the multimodal classification model is trained using the multimodal classification model training method provided in this disclosure.

[0106] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0107] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0108] The modules described in the embodiments of this disclosure can be implemented in software or in hardware. The names of the modules are not necessarily limiting in certain circumstances; for example, the second acquisition module can also be described as a "module for acquiring videos to be classified".

[0109] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0110] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0111] According to one or more embodiments of this disclosure, Example 1 provides a method for training a multimodal classification model. The multimodal classification model includes: multiple classification heads corresponding one-to-one with multiple classification tasks, a feature extractor and a concatenated module, wherein the concatenated module is connected to each of the classification heads. The method includes: for each classification head, obtaining a sample video and a classification label of the sample video under the classification task corresponding to the classification head; extracting deep features and shallow features of the sample video through the pre-trained feature extractor; generating concatenated features through the concatenated module based on the deep features and the shallow features; and training the classification head based on the concatenated features and the classification label.

[0112] According to one or more embodiments of this disclosure, Example 2 provides the method of Example 1, wherein the feature extractor includes a cascaded backbone network and a fusion module, wherein the fusion module includes multiple cascaded modal fusion layers, the stitching module is connected to each of the modal fusion layers respectively, and the backbone network is connected to each of the modal fusion layers respectively; the backbone network is used to extract text features and visual features of the sample video, and input the text features and the visual features into each of the modal fusion layers; each modal fusion layer is used to fuse the features input into the modal fusion layer to obtain deep features or shallow features, and input the deep features or the shallow features into the next modal fusion layer and the stitching module.

[0113] According to one or more embodiments of this disclosure, Example 3 provides the method of Example 1, wherein generating spliced ​​features by the splicing module based on the deep features and the shallow features includes: splicing the classification token cls-token in the deep features or the cls-token in the shallow features input to each modality fusion layer by the splicing module to obtain spliced ​​features.

[0114] According to one or more embodiments of this disclosure, Example 4 provides the method of Example 2, wherein the splicing module includes a splicing unit and a plurality of dimensionality reduction units corresponding one-to-one with the plurality of modality fusion layers, wherein the dimensionality reduction unit is connected to its corresponding modality fusion layer, and the splicing unit is connected to each of the dimensionality reduction units respectively, and the modality fusion layer is used to input the classification tag cls-token in the deep features or the cls-token in the shallow features to its corresponding dimensionality reduction unit; the step of generating spliced ​​features by the splicing module according to the deep features and the shallow features includes: each dimensionality reduction unit performing dimensionality reduction processing on the cls-token in the deep features or the cls-token in the shallow features input by its corresponding modality fusion layer; the splicing unit splices the cls-token obtained after each dimensionality reduction processing to obtain the spliced ​​features.

[0115] According to one or more embodiments of this disclosure, Example 5 provides the method of Example 4, the method further comprising: for each dimensionality reduction unit, adjusting the model parameters of the dimensionality reduction unit according to the dimension of the cls-token input into the dimensionality reduction unit.

[0116] According to one or more embodiments of this disclosure, Example 6 provides a method as described in any one of Examples 1-5, wherein the sample video is video data from the business scenario to which the classification task corresponding to the classification head belongs.

[0117] According to one or more embodiments of this disclosure, Example 7 provides a video classification method, including: acquiring a video to be classified; generating classification results of the video to be classified under different classification tasks using a pre-trained multimodal classification model, wherein the multimodal classification model is trained using the multimodal classification model training method described in any one of Examples 1-6.

[0118] According to one or more embodiments of this disclosure, Example 8 provides a multimodal classification model training apparatus. The multimodal classification model includes: multiple classification heads corresponding one-to-one with multiple classification tasks, a feature extractor and a concatenation module connected in series, wherein the concatenation module is connected to each of the classification heads respectively; the apparatus includes: a first acquisition module, used to acquire a sample video and a classification label of the sample video under the classification task corresponding to the classification head for each classification head; a feature extraction module, used to extract deep features and shallow features of the sample video through a pre-trained feature extractor; a feature processing module, used to generate concatenated features through the concatenation module based on the deep features and the shallow features; and a classification head training module, used to train the classification head based on the concatenated features and the classification labels.

[0119] According to one or more embodiments of this disclosure, Example 9 provides a video classification apparatus, including: a second acquisition module for acquiring a video to be classified; and a classification module for generating classification results of the video to be classified under different classification tasks using a pre-trained multimodal classification model, wherein the multimodal classification model is trained using the multimodal classification model training method described in any one of Examples 1-6.

[0120] According to one or more embodiments of the present disclosure, Example 10 provides a computer-readable medium having a computer program stored thereon that, when executed by a processing device, implements the steps of the method described in any one of Examples 1-7.

[0121] According to one or more embodiments of this disclosure, Example 11 provides an electronic device including: a storage device having a computer program stored thereon; and a processing device for executing the computer program in the storage device to implement the steps of any one of Examples 1-7.

[0122] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0123] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0124] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative forms of implementing the claims. Regarding the apparatus in the above embodiments, the specific manner in which the various modules perform their operations has been described in detail in the embodiments relating to the method, and will not be elaborated upon here.

Claims

1. A method for training a multimodal classification model, characterized in that, The multimodal classification model includes: multiple classification heads corresponding one-to-one with multiple classification tasks, a series-connected feature extractor and a splicing module, wherein the feature extractor includes a series-connected backbone network and a fusion module, wherein the fusion module includes multiple series-connected modal fusion layers, the splicing module is connected to each of the modal fusion layers, the backbone network is connected to each of the modal fusion layers, and the splicing module is connected to each of the classification heads; The method includes: For each classification head, obtain the sample video and the classification label of the sample video under the classification task corresponding to the classification head; The deep and shallow features of the sample video are extracted by a feature extractor obtained after pre-training; wherein, the backbone network is used to extract the text features and visual features of the sample video, and input the text features and visual features into each modality fusion layer; each modality fusion layer is used to fuse the features input into the modality fusion layer to obtain deep features or shallow features, and input the deep features or shallow features into the next modality fusion layer and the stitching module; Based on the deep features and the shallow features, the splicing module generates spliced ​​features; The classification head is trained based on the splicing features and the classification labels.

2. The method according to claim 1, characterized in that, The step of generating stitched features through the stitching module based on the deep features and the shallow features includes: The stitching module stitches together the classification token cls-token from the deep features or the cls-token from the shallow features input to each modality fusion layer to obtain stitched features.

3. The method according to claim 1, characterized in that, The splicing module includes a splicing unit and multiple dimensionality reduction units corresponding one-to-one with the multiple modality fusion layers. The dimensionality reduction unit is connected to its corresponding modality fusion layer, and the splicing unit is connected to each of the dimensionality reduction units. The modality fusion layer is used to input the classification tag cls-token in the deep features or the cls-token in the shallow features into its corresponding dimensionality reduction unit. The step of generating stitched features through the stitching module based on the deep features and the shallow features includes: Each of the dimensionality reduction units performs dimensionality reduction processing on the cls-token in the deep features or the cls-token in the shallow features input to the modality fusion layer corresponding to it; The splicing unit splices the cls-tokens obtained after each dimensionality reduction process to obtain spliced ​​features.

4. The method according to claim 3, characterized in that, The method further includes: For each dimensionality reduction unit, the model parameters of the dimensionality reduction unit are adjusted according to the dimension of the cls-token input into the dimensionality reduction unit.

5. The method according to any one of claims 1-4, characterized in that, The sample video is video data from the business scenario to which the classification task corresponding to this classification head belongs.

6. A video classification method, characterized in that, include: Get the videos to be categorized; The classification results of the video to be classified under different classification tasks are generated by a pre-trained multimodal classification model, wherein the multimodal classification model is trained by the multimodal classification model training method of any one of claims 1-5.

7. A multimodal classification model training device, characterized in that, The multimodal classification model includes: multiple classification heads corresponding one-to-one with multiple classification tasks, a series-connected feature extractor and a splicing module, wherein the feature extractor includes a series-connected backbone network and a fusion module, wherein the fusion module includes multiple series-connected modal fusion layers, the splicing module is connected to each of the modal fusion layers, the backbone network is connected to each of the modal fusion layers, and the splicing module is connected to each of the classification heads; The device includes: The first acquisition module is used to acquire, for each classification head, a sample video and the classification label of the sample video under the classification task corresponding to that classification head; The feature extraction module is used to extract deep and shallow features of the sample video using a pre-trained feature extractor; wherein, the backbone network is used to extract text and visual features of the sample video, and input the text and visual features into each modality fusion layer; each modality fusion layer is used to fuse the features input into the modality fusion layer to obtain deep or shallow features, and input the deep or shallow features into the next modality fusion layer and the stitching module; The feature processing module is used to generate spliced ​​features based on the deep features and the shallow features through the splicing module; The classification head training module is used to train the classification head based on the spliced ​​features and the classification labels.

8. A video classification device, characterized in that, include: The second acquisition module is used to acquire the video to be classified; The classification module is used to generate classification results of the video to be classified under different classification tasks using a pre-trained multimodal classification model, wherein the multimodal classification model is trained by the multimodal classification model training method according to any one of claims 1-5.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When executed by the processing device, the program implements the steps of the method according to any one of claims 1-6.

10. An electronic device, characterized in that, include: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the method according to any one of claims 1-6.

Images