Training method, resource classification method and device for multimedia resource classification model

By jointly training pre-defined discriminative and generative models, the problem of high computational cost in multimedia resource classification is solved, and the model training efficiency and application flexibility are improved.

CN123115883APending Publication Date: 2026-11-17TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510615569.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-14
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies suffer from high computational costs in multimedia resource classification, especially since a large number of samples need to be labeled and repeated calculations are required during model training.

Method used

Multimedia resources are classified and predicted using pre-defined discriminative and generative models. By acquiring sample resources labeled with category labels and classification reason labels, the target loss data is calculated and the model is trained together, reducing computational power.

Benefits of technology

It reduces computational costs, improves model training efficiency and application flexibility in multimedia resource classification, and enables the selection of appropriate models for classification based on different needs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115883A_ABST
    Figure CN123115883A_ABST
Patent Text Reader

Abstract

This application discloses a training method, resource classification method, and apparatus for a multimedia resource classification model, which can be applied to various scenarios such as cloud technology, artificial intelligence, intelligent transportation, and the Internet of Vehicles. The method includes: inputting a first sample of multimedia resources into a preset discriminative model for classification prediction to obtain a first sample classification result; inputting a second sample of multimedia resources into a preset generative model for classification prediction to obtain a second sample classification result and a sample classification justification result; training the preset discriminative model and the preset generative model based on the differences between the first sample classification result and the first sample category label, the differences between the second sample classification result and the second sample category label, and the differences between the sample classification justification result and the sample classification justification label, to obtain a multimedia resource classification model. This application reduces the computational power during model training.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a training method, resource classification method and apparatus for a multimedia resource classification model. Background Technology

[0002] In the era of rapid internet development, the upload volume of multimedia resources is growing exponentially. It is necessary to understand the content of these resources before categorizing them. Related technologies typically employ machine learning algorithms and human assistance for content understanding. For example, for multi-level content classification and tagging, different classification tree structures and candidate word lists for different domains are first constructed manually for each specific business scenario. Then, a large number of content samples are manually labeled for each different business scenario. These samples are labeled with corresponding multi-level classifications and tags. These different content samples are then used to train different content understanding models, which are then deployed independently. Currently, the main solutions for content understanding and content quality processing combine machine algorithm capabilities with human-assisted content review and tagging.

[0003] Whether it's content understanding or content quality modeling, the process of building a model requires labeling a large number of samples, and different models process the original content in the inference calculation process, resulting in a lot of repetitive calculations and high computational costs. Summary of the Invention

[0004] This application provides a training method, resource classification method, and apparatus for a multimedia resource classification model, which can reduce the computational power during the model training process.

[0005] On the one hand, this application provides a training method for a multimedia resource classification model, the method comprising:

[0006] Obtain the first sample multimedia resource labeled with the first sample category label, and the second sample multimedia resource labeled with the second sample category label and the sample classification reason label;

[0007] The first sample multimedia resource is input into a preset discriminant model for classification and prediction to obtain the first sample classification result of the first sample multimedia resource.

[0008] The second sample multimedia resource is input into a preset generative model for classification and prediction, and the second sample classification result and the sample classification reason result are obtained.

[0009] Based on the difference between the first sample classification result and the first sample category label, the difference between the second sample classification result and the second sample category label, and the difference between the sample classification reason result and the sample classification reason label, the target loss data is determined;

[0010] The preset discriminative model and the preset generative model are trained based on the target loss data to obtain a multimedia resource classification model.

[0011] On the other hand, a multimedia resource classification method is provided, the method comprising:

[0012] The process involves acquiring target multimedia resources and selecting target classification models that match the target multimedia resources from a multimedia resource classification model. These multimedia resource classification models include discriminative classification models and generative classification models. The multimedia resource classification models are trained using the methods described above.

[0013] When the target classification model is the discriminant classification model, the target multimedia resource is input into the discriminant classification model for classification prediction to obtain the first classification result of the target multimedia resource;

[0014] When the target classification model is the generative classification model, the target multimedia resource is input into the generative classification model for classification prediction to obtain the second classification result of the target multimedia resource and the reason for the target classification.

[0015] On the other hand, a training device for a multimedia resource classification model is provided, the device comprising:

[0016] The sample resource acquisition module is used to acquire the first sample multimedia resource labeled with the first sample category label, and the second sample multimedia resource labeled with the second sample category label and the sample classification reason label;

[0017] The first classification module is used to input the first sample multimedia resource into a preset discriminant model for classification prediction, and obtain the first sample classification result of the first sample multimedia resource.

[0018] The second classification module is used to input the second sample multimedia resource into a preset generative model for classification prediction, and to obtain the second sample classification result and the sample classification reason result of the second sample multimedia resource.

[0019] The target loss determination module is used to determine target loss data based on the difference between the first sample classification result and the first sample category label, the difference between the second sample classification result and the second sample category label, and the difference between the sample classification reason result and the sample classification reason label.

[0020] The model training module is used to train the preset discriminative model and the preset generative model based on the target loss data to obtain a multimedia resource classification model.

[0021] In one exemplary embodiment, the model training module includes:

[0022] The parameter adjustment unit is used to adjust the model parameters of the preset discriminative model and the preset generative model according to the target loss data until the training termination condition is met.

[0023] The model determination unit is used to determine the preset discriminative model at the end of training as a discriminative classification model and the preset generative model at the end of training as a generative classification model.

[0024] The model combination unit is used to combine the discriminative classification model and the generative classification model into a multimedia resource classification model.

[0025] In one exemplary embodiment, the apparatus further includes:

[0026] The prompt text acquisition module is used to acquire the first prompt text corresponding to the classification task of the preset discriminative model and the second prompt text corresponding to the classification task of the preset generative model.

[0027] The first classification module is further configured to input the first sample multimedia resource and the first prompt text into a preset discriminant model for classification prediction, and obtain the first sample classification result of the first sample multimedia resource;

[0028] The second classification module is further configured to input the second sample multimedia resource and the second prompt text into a preset generative model for classification prediction, and obtain the second sample classification result and the sample classification reason result of the second sample multimedia resource.

[0029] In one exemplary embodiment, the first sample multimedia resource includes multimodal data, and the preset discriminative model includes a first image feature extraction network, a first text feature extraction network, and a first classification network. The first classification module includes:

[0030] The first sample acquisition unit is used to acquire a first sample image and a first sample text that are related in the first sample multimedia resources;

[0031] The first image feature extraction unit is used to input the first sample image into the first image feature extraction network for image feature extraction processing to obtain the first sample image features;

[0032] The first sample fusion unit is used to input the first sample text and the first prompt text into the first text feature extraction network for text feature extraction and feature fusion processing to obtain the first sample fused text features.

[0033] The first classification unit is used to input the image features of the first sample and the fused text features of the first sample into the first classification network for classification prediction, so as to obtain the first sample classification result of the multimedia resource of the first sample.

[0034] In one exemplary embodiment, the preset generative model includes a second image feature extraction network, a second text feature extraction network, and a second classification network, wherein the second classification module includes:

[0035] The second sample acquisition unit is used to acquire second sample images and second sample text that are related in the second sample multimedia resources;

[0036] The second image feature extraction unit is used to input the second sample image into the second image feature extraction network for image feature extraction processing to obtain the features of the second sample image.

[0037] The second sample fusion unit is used to input the second sample text and the second prompt text into the second text feature extraction network for text feature extraction and feature fusion processing to obtain the second sample fused text features.

[0038] The second classification unit is used to input the image features of the second sample and the fused text features of the second sample into the second classification network for classification prediction, so as to obtain the second sample classification result and the sample classification reason result of the multimedia resource of the second sample.

[0039] In one exemplary embodiment, the first image feature extraction network and the second image feature extraction network have the same structure, and the first text feature extraction network and the second text feature extraction network have the same structure. The model training module includes:

[0040] The first parameter adjustment unit is used to adjust the network parameters of the first image feature extraction network based on the target loss data to obtain the first network parameters, and to adjust the network parameters of the first text feature extraction network to obtain the second network parameters; and to adjust the network parameters of the first classification network and the second classification network.

[0041] The second parameter adjustment unit is used to adjust the network parameters of the second image feature extraction network to the first network parameters, and to adjust the network parameters of the second text feature extraction network to the second network parameters.

[0042] The model training unit is used to train the preset discriminative model with adjusted parameters based on the first sample multimedia resources, and to train the preset generative model with adjusted parameters based on the second sample multimedia resources, so as to obtain a multimedia resource classification model.

[0043] In one exemplary embodiment, the first sample multimedia resource further includes multiple unimodal data, the multiple unimodal data including sample unimodal text and sample unimodal images, the preset discriminative model further includes a unimodal text extraction network, a unimodal image extraction network, and a third classification network, and the device further includes:

[0044] The single-modal text extraction module is used to input the sample single-modal text into the single-modal text extraction network for text feature extraction to obtain the sample single-modal text features;

[0045] A single-modal image extraction module is used to input the sample single-modal image into the single-modal image extraction network for image feature extraction to obtain the sample single-modal image features;

[0046] The first classification unit includes:

[0047] The first classification prediction subunit is used to input the first sample image features and the first sample fused text features into the first classification network to perform classification prediction and obtain the first classification prediction result.

[0048] The second classification prediction subunit is used to input the sample single-modal text features and the sample single-modal image features into the third classification network for classification prediction to obtain the second classification prediction result;

[0049] The first classification subunit is used to determine the first sample classification result of the first sample multimedia resource based on the first classification prediction result and the second classification prediction result.

[0050] In one exemplary embodiment, the first classification module is further configured to input the first sample multimedia resource into a preset discriminative model for classification prediction to obtain an initial sample classification result of the first sample multimedia resource and a predicted probability of the initial sample classification result; and if the predicted probability is less than a preset probability threshold, input the first sample multimedia resource into the preset generative model for classification prediction to obtain the first sample classification result.

[0051] In one exemplary embodiment, the sample resource acquisition module includes:

[0052] The sample resource acquisition unit is used to acquire multiple sample multimedia resources;

[0053] The resource partitioning unit is used to divide the plurality of sample multimedia resources into the first sample multimedia resource and the second sample multimedia resource according to at least one of the sample resource content length and sample application scenario corresponding to each sample multimedia resource.

[0054] The label determination unit is used to label the first sample multimedia resource with the first sample category label, and to label the second sample multimedia resource with the second sample category label and the sample classification reason label.

[0055] In one exemplary embodiment, the resource partitioning unit includes:

[0056] The sample scenario acquisition subunit is used to acquire the sample resource content length and sample application scenario corresponding to each sample multimedia resource.

[0057] The first resource determination subunit is used to determine the sample multimedia resources that meet at least one of the first and second conditions as the first sample multimedia resources; the first condition is that the length of the sample resource content is less than a preset length threshold, and the second condition is that the sample application scenario is not a resource recommendation scenario and does not require an explanation of the classification reason;

[0058] The second resource determination subunit is used to determine the sample multimedia resources that do not meet the first condition and do not meet the second condition as the second sample multimedia resources.

[0059] On the other hand, a multimedia resource classification device is provided, the device comprising:

[0060] The target resource acquisition module is used to acquire target multimedia resources and filter out target classification models that match the target multimedia resources from the multimedia resource classification models; the multimedia resource classification models include discriminative classification models and generative classification models; the multimedia resource classification models are trained using the methods described above.

[0061] The first prediction module is used to input the target multimedia resource into the discriminant classification model for classification prediction when the target classification model is the discriminant classification model, so as to obtain the first classification result of the target multimedia resource;

[0062] The second prediction module is used to input the target multimedia resource into the generative classification model for classification prediction when the target classification model is the generative classification model, so as to obtain the second classification result of the target multimedia resource and the reason for the target classification.

[0063] In one exemplary embodiment, the target resource acquisition module includes:

[0064] The target scene acquisition unit is used to acquire the target resource content length and target application scene of the target multimedia resource.

[0065] The first matching unit is configured to determine the discriminative classification model as a target classification model that matches the target multimedia resource if the target multimedia resource satisfies at least one of the first target condition and the second target condition; the first target condition is that the length of the target resource content is less than a preset length threshold, and the second target condition is that the target application scenario is not a resource recommendation scenario and does not require an explanation of the classification reason.

[0066] The second matching unit is used to determine the generative classification model as the target classification model that matches the target multimedia resource if the target multimedia resource does not meet the first condition and does not meet the second condition.

[0067] In one exemplary embodiment, the target multimedia resources are at least two, and the target resource acquisition module includes:

[0068] A preset split ratio determination unit is used to determine the preset split ratio of the discriminative classification model and the generative classification model;

[0069] The target resource partitioning unit is used to divide the multiple target multimedia resources into a first target multimedia resource and a second target multimedia resource according to the preset splitting ratio; wherein, the ratio of the total number of the first target multimedia resources to the total number of the second target multimedia resources is equal to the preset splitting ratio;

[0070] The model determination unit is used to determine the target classification model corresponding to the first target multimedia resource as the discriminative classification model, and to determine the target classification model corresponding to the second target multimedia resource as the generative classification model.

[0071] In one exemplary embodiment, the preset split ratio determination unit includes:

[0072] The test resource acquisition subunit is used to acquire test multimedia resources; the test multimedia resources are labeled with test category tags.

[0073] The classification test subunit is used to input the test multimedia resources into the discriminative classification model and the generative classification model respectively for classification testing, and obtain the first test result output by the discriminative classification model and the second test result of the generative classification model.

[0074] The first accuracy determination subunit is used to determine the first accuracy of the discriminative classification model based on the difference between the first test result and the test category label.

[0075] The second accuracy determination subunit is used to determine the second accuracy of the generative classification model based on the difference between the second test result and the test category label.

[0076] The split ratio determination subunit is used to determine the preset split ratio of the discriminative classification model and the generative classification model based on the accuracy ratio of the first accuracy and the second accuracy; the preset split ratio is positively correlated with the accuracy ratio.

[0077] On the other hand, an electronic device is provided, the device including a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the training method of the multimedia resource classification model as described above.

[0078] On the other hand, a computer storage medium is provided, which stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the training method of the multimedia resource classification model as described above.

[0079] On the other hand, a computer program product is provided, including a computer program that is loaded and executed by a processor to implement the training method of the multimedia resource classification model as described above.

[0080] The training method, resource classification method, and apparatus for the multimedia resource classification model provided in this application have the following technical effects:

[0081] This application obtains a first sample multimedia resource labeled with a first sample category tag, and a second sample multimedia resource labeled with a second sample category tag and a sample classification reason tag; then, it inputs the first sample multimedia resource into a preset discriminative model for classification prediction to obtain a first sample classification result for the first sample multimedia resource; it inputs the second sample multimedia resource into a preset generative model for classification prediction to obtain a second sample classification result and a sample classification reason result for the second sample multimedia resource; this achieves the use of different models to process sample multimedia resources with different tags; and based on the difference between the first sample classification result and the first sample category tag, the second sample classification result is compared with the first sample category tag. The target loss data is determined by the difference between the two sample category labels and the difference between the sample classification reason result and the sample classification reason label. Thus, the target loss data can be jointly determined based on the output results of the two models. Then, the preset discriminative model and the preset generative model are trained synchronously based on the target loss data, thereby obtaining a multimedia resource classification model containing two models through joint training. Compared to training the two models separately, this application adjusts the parameters of both models simultaneously based on a single target loss data, reducing the computational power during model training. In application, different models can be selected from the two models for different target multimedia resources, improving the flexibility of model selection. Attached Figure Description

[0082] To more clearly illustrate the technical solutions and advantages in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0083] Figure 1 This is a schematic diagram of the application environment of a training method for a multimedia resource classification model provided in the embodiments of this specification;

[0084] Figure 2 This is a flowchart illustrating a training method for a multimedia resource classification model provided in the embodiments of this specification;

[0085] Figure 3 This is a flowchart illustrating a method provided in this specification for inputting the first sample multimedia resource and the first prompt text into a preset discriminative model for classification and prediction to obtain the first sample classification result of the first sample multimedia resource.

[0086] Figure 4This is a flowchart illustrating a method provided in this specification for inputting the second sample multimedia resource and the second prompt text into a preset generative model for classification prediction, thereby obtaining the second sample classification result and the sample classification reason result of the second sample multimedia resource.

[0087] Figure 5 This is a flowchart illustrating a method for training a preset discriminative model and a preset generative model based on the target loss data to obtain a multimedia resource classification model, as provided in an embodiment of this specification.

[0088] Figure 6 This is a flowchart illustrating a method for training a preset discriminative model and a preset generative model based on the target loss data to obtain a multimedia resource classification model, as provided in an embodiment of this specification.

[0089] Figure 7 This is a schematic diagram of the structure of a training system for a multimedia resource classification model provided in the embodiments of this specification;

[0090] Figure 8 This is a flowchart illustrating a multimedia resource classification method provided in the embodiments of this specification;

[0091] Figure 9 This is a schematic diagram of a comment page for a short video provided in an embodiment of this specification;

[0092] Figure 10 This is a schematic diagram of the structure of a training device for a multimedia resource classification model provided in the embodiments of this specification;

[0093] Figure 11 This is a schematic diagram of the structure of a multimedia resource classification device provided in the embodiments of this specification;

[0094] Figure 12 This is a schematic diagram of the structure of a server provided in an embodiment of this specification. Detailed Implementation

[0095] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0096] It is understood that in the specific embodiments of this application, data related to user information is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0097] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0098] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0099] Please see Figure 1 , Figure 1 This is a schematic diagram of the application environment of a training method for a multimedia resource classification model provided in an embodiment of this application. The application environment may include at least a server 100 and a terminal 200.

[0100] In an optional embodiment, server 100 can be used to train a multimedia resource classification model that includes a discriminative classification model and a generative classification model. Server 100 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0101] In an optional embodiment, terminal 200 can be used to identify the classification results of multimedia resources according to a discriminative classification model or the generative classification model and display them to the user. Specifically, terminal 200 can be, but is not limited to, electronic devices such as smartphones, desktop computers, tablets, laptops, smart speakers, digital assistants, augmented reality (AR) / virtual reality (VR) devices, smart wearable devices, in-vehicle terminals, and smart TVs; it can also be software running on the above-mentioned electronic devices, such as applications and mini-programs. The operating system running on the electronic device in this embodiment can include, but is not limited to, Android, iOS, Linux, and Windows.

[0102] The following describes a training method for a multimedia resource classification model proposed in this application. Figure 2 This is a flowchart illustrating a training method for a multimedia resource classification model provided in the embodiments of this specification. This specification provides the operational steps of the method described in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or server products, the methods shown in the embodiments or drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as shown in the embodiments or drawings... Figure 2 As shown, the method may include:

[0103] S201: Obtain the first sample multimedia resource labeled with the first sample category label, and the second sample multimedia resource labeled with the second sample category label and the sample classification reason label.

[0104] In the embodiments of this specification, the first sample multimedia resource and the second sample multimedia resource can be multimedia resources of the same or different categories, that is, the first sample category label and the second sample category label can be labels of the same or different categories; multimedia resources can include, but are not limited to, video, images, audio, text, etc., and the sample classification reason is the explanatory reason for configuring the second sample multimedia resource with the second sample category label, used to explain the reason for determining that the second sample multimedia resource belongs to the second sample category. The first sample category label and the second sample category label are both sample category labels, and the sample category label can be at least one of the sample resource category label and the sample quality category label; the sample resource category label can be a first-level category label, or it can be a multi-level category label including multiple levels; for example, if the sample multimedia resource is a mobile phone explanation video, then its sample resource category label can include multi-level labels, with the first-level label being technology, the second-level label being smartphone, and the third-level label being a mobile phone of brand xx. For example, the sample multimedia resource can include sample objects, and the sample resource category label can be the sample object category label of the sample object in the sample multimedia resource. The sample quality category label is used to characterize the quality of the sample multimedia resource. For example, if the sample multimedia resource is a video, the sample quality category can be determined based on the video's clarity or its semantic content. That is, the sample quality category can characterize the video's clarity or whether the video content contains metaphorical, vulgar, or satirical information.

[0105] S203: Input the first sample multimedia resource into a preset discriminant model for classification prediction to obtain the first sample classification result of the first sample multimedia resource.

[0106] In the embodiments of this specification, the preset discriminative model directly performs classification or regression tasks by learning the conditional probability distribution P(Y|X) of the data. It focuses on learning the decision boundary between features and labels, and the output result is usually the probability value of a certain label.

[0107] S205: Input the second sample multimedia resource into a preset generative model for classification prediction to obtain the second sample classification result and the sample classification reason result of the second sample multimedia resource.

[0108] In the embodiments of this specification, the pre-defined generative model can be a large-scale language model. It trains the basic language model by predicting the probability of the next token appearing, works by learning the probability distribution of words in the language, and uses this knowledge to generate new text that matches the input context. The pre-defined generative model can generate new text based on a given input, which may include multimodal information such as images and text. It can also extract multimodal information such as images and text from a second sample multimedia resource, and then perform classification prediction through the pre-defined generative model to obtain the second sample classification result and the sample classification reason result of the second sample multimedia resource.

[0109] S207: Determine the target loss data based on the difference between the first sample classification result and the first sample category label, the difference between the second sample classification result and the second sample category label, and the difference between the sample classification reason result and the sample classification reason label.

[0110] In the embodiments of this specification, a loss function can be constructed during the training process of the two models; after the two preset models output prediction results, the target loss data can be calculated based on the loss function and the difference between each prediction result and the corresponding label.

[0111] S209: Train the preset discriminative model and the preset generative model based on the target loss data to obtain a multimedia resource classification model.

[0112] In the embodiments of this specification, during each training process, the model parameters of the preset discriminative model and the preset generative model can be adjusted separately according to the target loss data until the training termination condition is met. The preset discriminative model and the preset generative model at the end of training are then combined to obtain a multimedia resource classification model. Compared to training the two models separately, this application adjusts the parameters of both models simultaneously based on a single target loss data, reducing the computational effort required during model training.

[0113] In some embodiments, obtaining a first sample multimedia resource labeled with a first sample category tag, and a second sample multimedia resource labeled with a second sample category tag and a sample classification reason tag, includes:

[0114] Acquire multimedia resources from multiple samples;

[0115] Based on at least one of the sample resource content length and sample application scenarios corresponding to each sample multimedia resource, the plurality of sample multimedia resources are divided into the first sample multimedia resource and the second sample multimedia resource.

[0116] The first sample multimedia resource is labeled with the first sample category label, and the second sample multimedia resource is labeled with the second sample category label and the sample classification reason label.

[0117] In the embodiments of this specification, the sample resource length can be determined based on the content length of the sample multimedia resource. When the sample multimedia resource includes multimodal information, the combined information length corresponding to the image and text in the multimodal information can be calculated. For example, the content in the sample multimedia resource can be converted into a token, and then the token length can be calculated as the sample resource content length. Multiple sample multimedia resources can be divided into a first sample multimedia resource and a second sample multimedia resource based on at least one of the sample resource length and sample application scenario, thereby determining the preset classification model corresponding to the sample multimedia resource. This achieves the classification of sample multimedia resources based on information such as the sample resource content length and sample application scenario, and assigns different labels to the two types of sample multimedia resources to meet the needs of practical applications.

[0118] In some embodiments, dividing the plurality of sample multimedia resources into first sample multimedia resources and second sample multimedia resources based on at least one of the sample resource content length and sample application scenarios corresponding to each sample multimedia resource includes:

[0119] Obtain the sample resource content length and sample application scenario corresponding to each sample multimedia resource;

[0120] The sample multimedia resources that meet at least one of the first and second conditions are determined as the first sample multimedia resources; the first condition is that the content length of the sample resource is less than a preset length threshold, and the second condition is that the application scenario of the sample is not a resource recommendation scenario and does not require an explanation of the classification reason.

[0121] Multimedia resources that do not meet the first condition and the second condition are identified as the second sample multimedia resources.

[0122] In the embodiments of this specification, the content in the sample multimedia resources is converted into tokens, and then the token length is calculated as the sample resource content length. For multimodal sample multimedia resources, the token lengths corresponding to each of the multiple modal features are calculated, and then summed to obtain the sample resource content length. A preset length threshold can be set in advance, and the sample resource content length is compared with the preset length threshold. The sample multimedia resources are then classified according to the comparison results. Alternatively, sample multimedia resources can be classified according to a first judgment result of whether the sample application scenario is a resource recommendation scenario and a second judgment result of whether an explanation of the classification reason is needed in that sample application scenario. Specifically, sample multimedia resources whose sample resource content length is less than the preset length threshold, or whose sample application scenario is not a resource recommendation scenario and does not require an explanation of the classification reason, can be identified as first sample multimedia resources. Sample multimedia resources that simultaneously satisfy the conditions of having a sample resource content length less than the preset length threshold, a sample application scenario not a resource recommendation scenario, and not requiring an explanation of the classification reason can also be identified as first sample multimedia resources. Sample multimedia resources that satisfy at least one of the following conditions—that the sample resource content length is greater than or equal to the preset length threshold, that the sample application scenario is a resource recommendation scenario, or that requires an explanation of the classification reason—are identified as second sample multimedia resources.

[0123] In the embodiments of this specification, sample multimedia resources with shorter content lengths can be identified as first sample multimedia resources; sample multimedia resources whose application scenarios are not resource recommendation scenarios and do not require explanation of classification reasons can also be identified as first sample multimedia resources; sample multimedia resources that meet at least one of the following conditions—that the content length is greater than or equal to the preset length threshold, the application scenario is a resource recommendation scenario, or the scenario requires explanation of classification reasons—are identified as second sample multimedia resources; and a preset discriminative model is used to process the first sample multimedia resources, while a preset generative model is used to process the second sample multimedia resources. Compared to the preset discriminative model, the preset generative model can handle sample data with higher complexity; thus, different sample multimedia resources are selected for processing based on the characteristics of the two models, improving the convergence efficiency of the model and the prediction accuracy of the classification results of different multimedia resources after training.

[0124] In some embodiments, before inputting the first sample multimedia resource into a preset discriminative model for classification prediction to obtain the first sample classification result of the first sample multimedia resource, the method further includes:

[0125] Obtain the first prompt text corresponding to the classification task of the preset discriminative model and the second prompt text corresponding to the classification task of the preset generative model;

[0126] For example, the step of inputting the first sample multimedia resource into a preset discriminative model for classification prediction to obtain the first sample classification result of the first sample multimedia resource includes:

[0127] The first sample multimedia resource and the first prompt text are input into a preset discriminant model for classification and prediction to obtain the first sample classification result of the first sample multimedia resource.

[0128] The step of inputting the second sample multimedia resource into a preset generative model for classification prediction, and obtaining the second sample classification result and the sample classification reason result of the second sample multimedia resource, includes:

[0129] The second sample multimedia resource and the second prompt text are input into a preset generative model for classification and prediction to obtain the second sample classification result and the sample classification reason result of the second sample multimedia resource.

[0130] In the embodiments of this specification, the first prompt text corresponding to the classification task of the preset discriminative model can be used to control the prediction task of the preset discriminative model; the second prompt text corresponding to the classification task of the preset generative model can be used to control the prediction task of the preset generative model. Since the preset discriminative model only needs to predict the classification result, the first prompt text can be a prompt text indicating the predicted classification result; while the preset generative model is used not only to predict the classification result but also to predict the classification reason corresponding to the classification result, therefore, the second prompt text can be a prompt text indicating both the predicted classification result and the classification reason. The first and second prompt texts serve as prompts for model training; that is, the text content used when posing questions, requests, or describing tasks to the model. Then, the first sample multimedia resource and the first prompt text are input into a preset discriminative model for classification prediction to obtain the first sample classification result of the first sample multimedia resource; the second sample multimedia resource and the second prompt text are input into a preset generative model for classification prediction to obtain the second sample classification result and sample classification reasoning result of the second sample multimedia resource. In this embodiment, prompt texts corresponding to the two models are introduced during the model training process. These can be used to clarify the model's task objectives and limit the model's output range to avoid generating content irrelevant to the task. They can also guide the model to think and reason more deeply when generating output, thereby improving the model's convergence efficiency.

[0131] In some embodiments, the first sample multimedia resource includes multimodal data, and the preset discriminative model includes a first image feature extraction network, a first text feature extraction network, and a first classification network, such as... Figure 3As shown, the step of inputting the first sample multimedia resource and the first prompt text into a preset discriminative model for classification prediction to obtain the first sample classification result of the first sample multimedia resource includes:

[0132] S2031: Obtain the first sample image and the first sample text that have a relationship in the first sample multimedia resources;

[0133] S2033: Input the first sample image into the first image feature extraction network for image feature extraction processing to obtain the first sample image features;

[0134] S2035: Input the first sample text and the first prompt text into the first text feature extraction network to perform text feature extraction and feature fusion processing to obtain the first sample fused text features;

[0135] S2037: Input the first sample image features and the first sample fused text features into the first classification network for classification prediction to obtain the first sample classification result of the first sample multimedia resource.

[0136] In the embodiments of this specification, the first sample multimedia resource may include multimodal data such as images and text. The preset discriminative model includes a first image feature extraction network, a first text feature extraction network, and a first classification network. The first image feature extraction network may include a first initial image feature extraction network and a first image semantic extraction network. First, the first sample image is input into the first initial image feature extraction network to extract the first initial image features. Then, the first image semantic extraction network extracts the image semantic features from the first initial image features to obtain the first sample image features.

[0137] The first initial image feature extraction network can include, but is not limited to, an image encoder trained on a CLIP-ViT network. CLIP-ViT is a multimodal model that combines a Contrastive Language-Image Pretraining (CLIP) framework with a Vision Transformer (ViT), aiming to achieve deep alignment between images and text through large-scale contrastive learning. The core of CLIP-ViT is to map images and text to the same vector space through contrastive learning, enabling the model to understand cross-modal semantic relationships. Its goal is to learn a general cross-modal representation through the comparison of positive samples (matched image-text pairs) and negative samples (mismatched pairs), thereby supporting zero-shot transfer and multi-task applications. The CLIP-ViT network can include an image encoder and a text encoder; in this embodiment, the image encoder in the CLIP-ViT network can be used as the first initial image feature extraction network.

[0138] CLIP aims to map images and text to the same embedding space, aligning relevant samples (positive sample pairs) and separating irrelevant samples (negative sample pairs) through contrastive learning.

[0139] The ViT architecture segments an image into fixed-size patches, which are then input into a Transformer encoder to extract features. The core of the Transformer lies in using a self-attention mechanism to model the relationships between elements in the input sequence.

[0140] The first image semantic extraction network may include, but is not limited to, a network trained on a QFormer network. QFormer (Query Transformer) is an improved version of the Visual Transformer (ViT) architecture, a variant of Transformer specifically designed for efficient cross-modal interaction, primarily used for query-guided feature alignment in multimodal tasks (such as text retrieval and visual question answering). Its core objective is to efficiently retrieve relevant information from another modality (such as images or text) by dynamically generating query vectors, reducing redundant computation and improving model flexibility. Training on a QFormer network can yield both an image semantic extraction module and a text semantic extraction module. In this embodiment, the image semantic extraction module from the QFormer network can be used as the first image semantic extraction network.

[0141] In some embodiments, the first sample text and the first prompt text can be input into the first text feature extraction network for text feature extraction processing, and the fused features obtained by fusing the extracted text features are determined as the first sample fused text features. The first text feature extraction network may include a first text vector transformation network (prompt) and a first text semantic extraction network. The first sample text and the first prompt text are input into the first text vector transformation network to convert them into first sample text vectors corresponding to the two texts. Then, the first sample text vectors corresponding to the two texts are input into the first text semantic extraction network for semantic feature extraction to obtain the first sample fused text features. The first text semantic extraction network may include, but is not limited to, a Large Language Model (LLM), a Bag of Words (BOW) model, TF-IDF (Term Frequency-Inverse Document Frequency), an N-Gram model, a Recurrent Neural Network (RNN), etc. The Bag of Words model represents text as a word frequency vector, without considering the order or context of words, only counting the frequency of each word in the text. TF-IDF is an improvement on the bag-of-words model, assigning different weights to words to measure the importance of each word in a document. Compared to the bag-of-words model, TF-IDF considers not only word frequency but also word prevalence. The N-Gram model is an extension of the bag-of-words model that captures word order information by using word groups as features. A Recurrent Neural Network (RNN) is a type of neural network used to process sequential data and other data with sequential relationships.

[0142] For example, the first classification network may include a cross-attention network and a first feature classification network. Cross-attention is an extension of the attention mechanism, mainly used to handle the interaction between two different sequences (such as text and image, text and speech, etc.), allowing elements in one sequence to dynamically pay attention to related elements in another sequence. It is a core component of multimodal models and cross-modal tasks (such as text-image retrieval and machine translation). The first feature classification network may include, but is not limited to, a network based on the normalized exponential function (Softmax function). In this embodiment, the first sample image features, the first sample text features, and the first sample cue features can be input into the cross-attention network to extract the first multimodal features, and then the first feature classification network can be used to classify and predict the first multimodal features to obtain the first sample classification result of the first sample multimedia resource.

[0143] In the embodiments of this specification, the preset discriminative model can process the first sample multimedia resource and the first prompt text through a first image feature extraction network, a first text feature extraction network, and a first classification network, thereby quickly obtaining the first sample classification result of the first sample multimedia resource.

[0144] In some embodiments, the pre-defined generative model includes a second image feature extraction network, a second text feature extraction network, and a second classification network, such as... Figure 4 As shown, the step of inputting the second sample multimedia resource and the second prompt text into a preset generative model for classification prediction to obtain the second sample classification result and the sample classification rationale result of the second sample multimedia resource includes:

[0145] S2051: Obtain the second sample image and the second sample text that have a relationship in the second sample multimedia resources;

[0146] S2053: Input the second sample image into the second image feature extraction network for image feature extraction processing to obtain the features of the second sample image;

[0147] S2055: Input the second sample text and the second prompt text into the second text feature extraction network to perform text feature extraction and feature fusion processing to obtain the second sample fused text features;

[0148] S2057: Input the second sample image features and the second sample fused text features into the second classification network for classification prediction to obtain the second sample classification result and the sample classification reason result of the second sample multimedia resource.

[0149] In the embodiments of this specification, the second sample multimedia resource may include multimodal data such as images and text. The preset generative model includes a second image feature extraction network, a second text feature extraction network, and a second classification network. The second image feature extraction network and the first image feature extraction network may use the same network structure. The second image feature extraction network includes a second initial image feature extraction network and a second image semantic extraction network. First, the second sample image is input into the second initial image feature extraction network to extract the second initial image features. Then, the image semantic features from the second initial image features are extracted through the second image semantic extraction network to obtain the second sample image features.

[0150] The second initial image feature extraction network may include, but is not limited to, an image encoder trained on a CLIP-ViT network. The second image semantic extraction network may include, but is not limited to, a network trained on a QFormer network. QFormer (Query Transformer) is an improved version of the Visual Transformer (ViT) architecture, a variant of Transformer designed for efficient cross-modal interaction, and is mainly used for query-guided feature alignment in multimodal tasks (such as text retrieval and visual question answering).

[0151] For example, the second text feature extraction network and the first text feature extraction network can use the same network structure.

[0152] In some embodiments, the second text feature extraction network may include, but is not limited to, a second text vector transformation network (prompt), a bag of words (BOW) model, TF-IDF (Term Frequency-Inverse Document Frequency), an N-Gram model, a recurrent neural network (RNN), etc.; for example, the second sample text and the second prompt text can be input into the second text vector transformation network for text feature extraction processing, and the fused features obtained by fusing the text features extracted from the two can be determined as the second sample fused text features.

[0153] The second classification network can output the classification result of the second sample and the corresponding classification reason. The second sample image features, the second sample text features, and the second sample fused text features can first undergo feature fusion processing to obtain the second sample fused features, which are then input into the second classification network for classification result prediction and classification reason prediction, thus obtaining the second sample classification result and sample classification reason result of the second sample multimedia resource. For example, the second classification network can include, but is not limited to, a large language model (LLM); the second sample image features, the second sample text features, and the second sample cue features can be input into the large language model to obtain the second sample classification result and sample classification reason result of the second sample multimedia resource.

[0154] In the embodiments of this specification, the preset generative model can process the second sample multimedia resource and the second prompt text through a second image feature extraction network, a second text feature extraction network, and a second classification network, thereby quickly obtaining the second sample classification result and the sample classification reason result of the second sample multimedia resource.

[0155] In some embodiments, the first image feature extraction network and the second image feature extraction network have the same structure, and the first text feature extraction network and the second text feature extraction network have the same structure, such as... Figure 5 As shown, the step of training the preset discriminative model and the preset generative model based on the target loss data to obtain a multimedia resource classification model includes:

[0156] S501: Adjust the network parameters of the first image feature extraction network based on the target loss data to obtain the first network parameters, and adjust the network parameters of the first text feature extraction network to obtain the second network parameters; and adjust the network parameters of the first classification network and the second classification network.

[0157] S503: Adjust the network parameters of the second image feature extraction network to the first network parameters, and adjust the network parameters of the second text feature extraction network to the second network parameters;

[0158] S505: Train the preset discriminative model with adjusted parameters based on the first sample multimedia resources, and train the preset generative model with adjusted parameters based on the second sample multimedia resources to obtain a multimedia resource classification model.

[0159] In the embodiments of this specification, the first image feature extraction network in the preset discriminative model and the second image feature extraction network in the preset generative model can adopt the same network structure, and the first text feature extraction network in the preset discriminative model and the second text feature extraction network in the preset generative model can adopt the same network structure; during the joint training of the two models, after determining the network parameters of a specific network in one model, the network parameters of the corresponding structure in the other model can be adjusted synchronously.

[0160] For example, the network parameters of the first image feature extraction network can be adjusted based on the target loss data to obtain the first network parameters, and the network parameters of the first text feature extraction network can be adjusted to obtain the second network parameters; then, the network parameters of the second image feature extraction network can be adjusted to the first network parameters, and the network parameters of the second text feature extraction network can be adjusted to the second network parameters; thereby, the preset discriminative model and the preset generative model can be further trained to obtain a multimedia resource classification model. During the parameter adjustment process, the network parameters of the second image feature extraction network and the second text feature extraction network in the preset generative model can also be adjusted first; then, the network parameters of the corresponding networks in the preset discriminative model can be adjusted simultaneously.

[0161] In the embodiments of this specification, during the training process of the discriminative classification model and the generative classification model, network parameters can be shared for networks with the same structure in the two models, reducing the training cost of the models and accelerating the convergence speed of the models. The final models are well-suited to handle content of multiple modalities and modality combinations, fully adapting to the characteristics of various different modalities, so that the two models can be applied to a variety of different application scenarios.

[0162] In some embodiments, the first sample multimedia resource further includes multiple unimodal data, the multiple unimodal data including sample unimodal text and sample unimodal images, the preset discriminative model further includes a unimodal text extraction network, a unimodal image extraction network, and a third classification network, and the method further includes:

[0163] The sample unimodal text is input into the unimodal text extraction network for text feature extraction to obtain the sample unimodal text features;

[0164] The sample single-modal image is input into the single-modal image extraction network for image feature extraction to obtain the sample single-modal image features;

[0165] The step of inputting the first sample image features and the first sample fused text features into the first classification network for classification prediction to obtain the first sample multimedia resource classification result includes:

[0166] The first sample image features and the first sample fused text features are input into the first classification network for classification prediction to obtain the first classification prediction result.

[0167] The sample single-modal text features and the sample single-modal image features are input into the third classification network for classification prediction to obtain the second classification prediction result;

[0168] Based on the first classification prediction result and the second classification prediction result, the first sample classification result of the multimedia resource of the first sample is determined.

[0169] In the embodiments of this specification, multimodal data and single-modal data can be data from the same first sample multimedia resource. For example, if the first sample multimedia resource is a video, then data can be extracted from each video frame. For video frames that include multimodal data, multimodal data can be obtained; for video frames that only include single-modal data, single-modal data can be obtained. In this case, the first sample multimedia resource includes both the multimodal data corresponding to the first video frame and the single-modal data corresponding to the second video frame.

[0170] In some application scenarios, multimodal data and unimodal data can be data from different sample multimedia resources; for example, in short text classification of comments or titles, or image-level classification scenarios such as the clarity and aesthetics of content cover images, the first sample multimedia resource only includes unimodal data, with only text or visual modal input.

[0171] The pre-defined discriminative models also include a unimodal text extraction network, a unimodal image extraction network, and a third-class classification network. The unimodal text extraction network is used to extract text features and may include, but is not limited to, the TinyBERT network. TinyBERT is a lightweight version of the BERT model, compressing the original BERT model through model distillation technology, significantly reducing the number of parameters and computational cost while maintaining high performance. The core of TinyBERT is knowledge distillation, which involves transferring knowledge from a large model (the teacher model, such as BERT) to a smaller model (the student model) by having it mimic the output or intermediate features of the larger model (the teacher model). Its goals are: reducing the number of parameters (reducing memory usage and computational overhead) and accelerating inference (suitable for resource-constrained scenarios such as mobile devices and edge devices). TinyBERT training consists of two stages: pre-training distillation: a distilled general TinyBERT model is first trained on a general corpus (such as WikiText). Task-adapted distillation: TinyBERT is fine-tuned on downstream task data (such as classification and question answering) to adapt it to specific tasks.

[0172] Single-modal image extraction networks are used to extract image features and can include, but are not limited to, Vit networks and ALBEF networks. Vit (Vision Transformer) is an image classification model based on the Transformer architecture. It segments images into fixed-size patches and converts them into sequential input, using a self-attention mechanism to capture global features, overcoming the limitations of local perception in traditional convolutional neural networks. ALBEF (Align before Fuse) is a contrastive learning model for multimodal tasks (vision-language), focusing on cross-modal alignment of images and text, and improving training efficiency through momentum distillation. Its core objective is to learn the joint representation space of vision (image) and language (text) to achieve efficient cross-modal retrieval and generation tasks. ALBEF pre-training tasks are divided into three parts: Image-Text Contrastive Learning, Masked Language Modeling, and Image-Text Matching. In this embodiment, the sub-network used for image feature extraction in the pre-trained ALBEF network can be used as a single-modal image extraction network for image feature extraction.

[0173] The third classification network may have the same or different structure as the first classification network; for example, the first classification network may include, but is not limited to, a network formed based on the normalized exponential function (Softmax function).

[0174] In the embodiments of this specification, the sample unimodal text can be input into the unimodal text extraction network for text feature extraction to obtain sample unimodal text features; then, the sample unimodal image can be input into the unimodal image extraction network for image feature extraction to obtain sample unimodal image features. Subsequently, when determining the first sample classification result of the first sample multimedia resource, the first classification prediction result corresponding to the first classification network and the second classification prediction result corresponding to the third classification network can be obtained; then, the average of the first and second classification prediction results is calculated as the first sample classification result; alternatively, weights can be set for the first and second classification prediction results respectively, and a weighted sum of the first and second classification prediction results can be calculated to obtain the first sample classification result; thereby improving the accuracy of the first sample classification result.

[0175] In some embodiments, the step of inputting the first sample multimedia resource into a preset discriminative model for classification prediction to obtain a first sample classification result of the first sample multimedia resource includes:

[0176] The first sample multimedia resource is input into a preset discriminant model for classification and prediction to obtain the initial sample classification result of the first sample multimedia resource and the predicted probability of the initial sample classification result.

[0177] If the predicted probability is less than a preset probability threshold, the first sample multimedia resource is input into the preset generative model for classification prediction to obtain the classification result of the first sample.

[0178] In the embodiments of this specification, when the preset discriminative model performs classification prediction on the first sample multimedia resource, it can output the initial sample classification result and its corresponding prediction probability; a preset probability threshold can be set according to the actual situation. For example, the output result can be a mobile phone of brand xx with a probability of 80%; if the preset probability threshold is 90%, the first sample multimedia resource can be input into the preset generative model for classification prediction, and the prediction result of the preset generative model can be used as the first sample classification result, thereby improving the accuracy of the first sample classification result and thus improving the convergence efficiency of the model.

[0179] In some embodiments, if the predicted probability is greater than or equal to a preset probability threshold, the initial sample classification result is determined as the first sample classification result.

[0180] In some embodiments, such as Figure 6 As shown, the step of training the preset discriminative model and the preset generative model based on the target loss data to obtain a multimedia resource classification model includes:

[0181] S601: Adjust the model parameters of the preset discriminative model and the preset generative model according to the target loss data until the training termination condition is met;

[0182] S603: Determine the preset discriminative model at the end of training as a discriminative classification model, and determine the preset generative model at the end of training as a generative classification model;

[0183] S605: Combine the discriminative classification model and the generative classification model into a multimedia resource classification model.

[0184] In the embodiments of this specification, during the synchronous training of the two models, target loss data can be calculated based on the output results of each model. Then, the model parameters of the preset discriminative model and the preset generative model are adjusted according to the target loss data until the training termination condition is met. The training termination condition can be determined based on at least one of the target loss data and the number of training iterations. For example, the target loss data being less than a preset loss threshold can be used as the training termination condition. This results in the discriminative classification model and the generative classification model after training, which are then combined to obtain a multimedia resource classification model. This embodiment, by jointly training the two models, can obtain a multimedia resource classification model including both a discriminative classification model and a generative classification model. In application, any branch of the multimedia resource classification model can be selected for application according to the actual situation, saving significant computing resources and reducing costs.

[0185] In some embodiments, such as Figure 7 As shown, Figure 7 This is a schematic diagram of the structure of a training system for a multimedia resource classification model. Sample multimedia resources can come from content published and consumed by content publishers and consumers, including content published and consumed on computers, mobile devices, or other platforms. The system obtains content published and consumed by content publishers and consumers as sample multimedia resources through a content business withdrawal interface service. Then, the content is distributed to human-assisted review through a content flow scheduling service. Human-assisted review includes a content quality and security service group and a content understanding service group. The content quality and security service group can be used for multi-level content classification, while the content understanding service group can be used for content quality classification. Both groups can be accessed through internal... Content processing is performed by a traffic splitting service, followed by content processing using one of two models: a pre-defined discriminative model and a generative classification model. The discriminative classification model includes a first initial image feature extraction network (CLIP-ViT), a first image semantic extraction network (QFormer), a first text vector transformation network (prompt), a first text extraction network (LLM), a cross-attention network, and a first classification network (normalized exponential function). The first initial image feature extraction network is used to input a first sample image; the first text vector transformation network is used to input a first prompt text and the first sample text. The pre-defined discriminative model also includes a unimodal text extraction network (TinyBERT), a unimodal image extraction network (Vit / ALBEF), and a third classification network (normalized exponential function). The unimodal text extraction network is used to input sample unimodal text, and the unimodal image extraction network is used to input sample unimodal images.

[0186] The pre-defined generative model includes a second initial image feature extraction network (CLIP-ViT), a second image semantic extraction network (QFormer), a second text extraction network (prompt), and a second classification network (LLM); wherein, the second initial image feature extraction network is used to input the second sample image; the second text extraction network is used to input the second prompt text and the second sample text.

[0187] In this embodiment, after training a multimedia resource classification model that combines a discriminative classification model and a generative classification model, different models can be selected to process different input multimedia resources during application, thereby obtaining prediction results for each multimedia resource, which are then stored in a distributable content pool for content distribution. Content distribution can be applied to application scenarios such as search, recommendation, and social distribution.

[0188] As can be seen from the technical solutions provided in the embodiments of this specification above, the embodiments of this specification obtain a first sample multimedia resource labeled with a first sample category tag, and a second sample multimedia resource labeled with a second sample category tag and a sample classification reason tag; then, the first sample multimedia resource is input into a preset discriminative model for classification prediction to obtain a first sample classification result of the first sample multimedia resource; the second sample multimedia resource is input into a preset generative model for classification prediction to obtain a second sample classification result and a sample classification reason result of the second sample multimedia resource; this realizes that different models are used to process sample multimedia resources with different tags; and then, based on the difference between the first sample classification result and the first sample category tag, the... The difference between the second sample classification result and the second sample category label, and the difference between the sample classification reason result and the sample classification reason label, determine the target loss data. Thus, the target loss data can be jointly determined based on the output results of the two models. Then, the preset discriminative model and the preset generative model are simultaneously trained based on the target loss data, thereby obtaining a multimedia resource classification model containing two models through joint training. Compared to training the two models separately, this application adjusts the parameters of both models simultaneously based on a single target loss data, reducing the computational power during model training. In application, different models can be selected from the two models for different target multimedia resources, improving the flexibility of model selection.

[0189] like Figure 8 As shown, this embodiment also provides a multimedia resource classification method, the method including:

[0190] S801: Acquire the target multimedia resource and select a target classification model that matches the target multimedia resource from the multimedia resource classification models; the multimedia resource classification model includes a discriminative classification model and a generative classification model; the multimedia resource classification model is trained using the above method.

[0191] S803: When the target classification model is the discriminant classification model, the target multimedia resource is input into the discriminant classification model for classification prediction to obtain the first classification result of the target multimedia resource;

[0192] S805: When the target classification model is the generative classification model, the target multimedia resource is input into the generative classification model for classification prediction to obtain the second classification result of the target multimedia resource and the reason for the target classification.

[0193] In the embodiments of this specification, the target multimedia resource can be unimodal data consisting only of images or text, or multimodal data including both images and text. A target classification model matching the target multimedia resource can be selected from multimedia resource classification models; regardless of whether the target multimedia resource is unimodal or multimodal, either a discriminative classification model or a generative classification model can be chosen for classification. In some embodiments, the target multimedia resource matching it can be determined based on information such as the content complexity and application scenario; for example, for target multimedia resources with low content complexity, a discriminative classification model can be selected as the target classification model; for target multimedia resources with high content complexity, a generative classification model can be selected as the target classification model. For target multimedia resources applied to resource recommendation scenarios or target multimedia resources requiring classification rationale, a generative classification model can be selected as the target classification model; conversely, for target multimedia resources in other application scenarios, a generative classification model can be selected as the target classification model.

[0194] Then, when the target classification model is the discriminative classification model, the target multimedia resource is input into the discriminative classification model for classification prediction, resulting in a first classification result for the target multimedia resource. When the target classification model is the generative classification model, the target multimedia resource is input into the generative classification model for classification prediction, resulting in a second classification result for the target multimedia resource and the reason for the target classification. This achieves the use of different target classification models for classification prediction for different target multimedia resources. When there are many target multimedia resources, different branch models can be used for processing, achieving end-to-end processing, saving a lot of computing resources and reducing costs. It also accelerates inference, improving the overall processing efficiency of the multimedia resource content understanding processing pipeline and reducing processing costs.

[0195] In some embodiments, the step of selecting a target classification model from the multimedia resource classification models that matches the target multimedia resource includes:

[0196] Obtain the target resource content length and target application scenario of the target multimedia resource;

[0197] If the target multimedia resource satisfies at least one of the first target condition and the second target condition, the discriminative classification model is determined as the target classification model that matches the target multimedia resource; the first target condition is that the length of the target resource content is less than a preset length threshold, and the second target condition is that the target application scenario is not a resource recommendation scenario and no explanation of the classification reason is required;

[0198] If the target multimedia resource does not meet the first condition and does not meet the second condition, the generative classification model is determined as the target classification model that matches the target multimedia resource.

[0199] In the embodiments of this specification, the content of the target multimedia resource can be converted into a token, and then the token length can be calculated as the target resource content length. For multimodal target multimedia resources, the token lengths corresponding to each of the multiple modal features are calculated, and then summed to obtain the target resource content length. A preset length threshold can be set in advance, and the target resource content length can be compared with the preset length threshold. The target multimedia resource can be classified according to the comparison result. The target classification model for the target multimedia resource can be determined based on a first judgment result indicating whether the target application scenario is a resource recommendation scenario and a second judgment result indicating whether an explanation of the classification reason is required under the target application scenario. Specifically, if the target multimedia resource meets the first or second target condition, i.e., the length of the target resource content is less than a preset length threshold, or the target application scenario is not a resource recommendation scenario and no explanation of the classification reason is required, the discriminant classification model can be determined as the target classification model matching the target multimedia resource. Alternatively, if the target multimedia resource simultaneously meets both the first and second target conditions, i.e., the length of the target resource content is less than a preset length threshold, and the target application scenario is not a resource recommendation scenario and no explanation of the classification reason is required, the discriminant classification model can be determined as the target classification model matching the target multimedia resource. The situation where the target multimedia resource does not meet the first condition and the second condition may include: the length of the target resource content is greater than or equal to the preset length threshold, the target application scenario is a resource recommendation scenario or a scenario that requires explanation of the classification reason, etc. In this case, the generative classification model is determined as the target classification model that matches the target multimedia resource. This realizes the use of different target classification models for classification prediction for different target multimedia resources, thereby improving the matching degree between the target multimedia resource and the target classification model and improving the classification accuracy.

[0200] In some embodiments, the target multimedia resources are at least two, and the step of selecting a target classification model from the multimedia resource classification models that matches the target multimedia resources includes:

[0201] Determine the preset split ratio for the discriminative classification model and the generative classification model;

[0202] The multiple target multimedia resources are divided into first target multimedia resources and second target multimedia resources according to the preset splitting ratio; wherein, the ratio of the total number of first target multimedia resources to the total number of second target multimedia resources is equal to the preset splitting ratio;

[0203] The target classification model corresponding to the first target multimedia resource is determined as the discriminative classification model, and the target classification model corresponding to the second target multimedia resource is determined as the generative classification model.

[0204] In the embodiments of this specification, when there are at least two target multimedia resources, to avoid overloading either model, traffic can be allocated according to a preset split ratio. A portion of the target multimedia resources can be allocated to the discriminative classification model, and the other portion to the generative classification model. The preset split ratio can be set based on the total number of target multimedia resources and the processing speeds of the discriminative and generative classification models. Typically, the processing speed of the discriminative classification model is greater than that of the generative classification model. If the total number of target multimedia resources is greater than the target number, the preset split ratio between the discriminative and generative classification models can be set to be greater than 1 to ensure faster processing speed. If the total number of target multimedia resources is less than or equal to the target number, the preset split ratio between the discriminative and generative classification models can be set to be less than 1 to ensure richness of prediction results. Furthermore, the preset split ratio can also be determined based on the historical prediction accuracy of the discriminative and generative classification models.

[0205] In some embodiments, a discriminative classification model is used for classification prediction. While obtaining the primary classification result of the target multimedia resource, the predicted probability of the primary classification result is also obtained. If the predicted probability is less than a preset probability threshold, it indicates that the accuracy of the primary classification result is low. In this case, the target multimedia resource can be input again into a generative classification model for secondary classification prediction to obtain a secondary classification result. This secondary classification result can then be used as the primary classification result, thereby improving the prediction accuracy of the classification result. This embodiment is applicable not only to normal usage scenarios but also to traffic splitting scenarios, avoiding the problem of low classification accuracy caused by traffic splitting.

[0206] In some embodiments, determining the preset split ratio of the discriminative classification model and the generative classification model includes:

[0207] Obtain test multimedia resources; the test multimedia resources are labeled with test category tags;

[0208] The test multimedia resources are respectively input into the discriminative classification model and the generative classification model for classification testing, and the first test result output by the discriminative classification model and the second test result of the generative classification model are obtained.

[0209] Based on the difference between the first test result and the test category label, determine the first accuracy of the discriminative classification model;

[0210] The second accuracy of the generative classification model is determined based on the difference between the second test result and the test category label;

[0211] Based on the ratio of the first accuracy rate to the second accuracy rate, a preset split ratio is determined for the discriminative classification model and the generative classification model; the preset split ratio is positively correlated with the accuracy ratio.

[0212] In the embodiments of this specification, before training and applying two models, a discriminative classification model and a generative classification model, the accuracy of the two models can be tested. Then, a preset split ratio for the two models is determined based on their accuracy. The test multimedia resources and the sample multimedia resources can be the same or different. The types of test category labels can be determined based on the sample category labels. A first accuracy of the discriminative classification model is determined based on the difference between the first test result and the test category label. If the difference between the first test result and the test category label is small, the first accuracy is high. A second accuracy of the generative classification model is determined based on the difference between the second test result and the test category label. If the difference between the second test result and the test category label is small, the second accuracy is high. Thus, the accuracy ratio of the first accuracy and the second accuracy can be further calculated to determine the preset split ratio of the discriminative classification model and the generative classification model. The accuracy ratio can be directly determined as the preset split ratio, or the preset split ratio can be obtained by adjusting the accuracy ratio.

[0213] This embodiment allocates a higher traffic volume to models with high accuracy and a lower traffic volume to models with lower accuracy. Especially when there are a large number of target multimedia resources, traffic can be split according to a preset split ratio before classification and prediction. This ensures that both models can quickly process the classification task, and by allocating a larger traffic volume of target multimedia resources to the model with high accuracy, the accuracy of the classification results is further guaranteed.

[0214] For example, the two models in this embodiment can be applied to application scenarios such as search, recommendation, and social distribution. A multimedia resource classification model combining a discriminative classification model and the generative classification model can be used to perform classification prediction processing on each target multimedia resource in the resource database, obtaining the category label information for each target multimedia resource. Furthermore, the generative classification model can predict the classification reason for the target multimedia resource. In a search scenario, the target category of the search resource can be determined based on the resource keywords entered by the user, and then multimedia resources matching the target category can be found in the resource database and displayed as search results. In a resource recommendation scenario, the historical categories of the multimedia resources browsed by the user can be obtained through the discriminative classification model and the generative classification model, and then multimedia resources matching the historical categories can be found in the resource database and recommended to the user.

[0215] For example, the method of this embodiment can be applied to the comment review scenario of short videos. The target multimedia resource is comment information. Low-quality comments can be filtered out from the comments of short videos and highlighted, and then displayed to the reviewing user so that the reviewing user can further review and determine whether to delete them, thereby improving the efficiency of comment quality review. In some embodiments, such as Figure 9 As shown, Figure 9 This is a schematic diagram of a comment page for a short video, which displays some comment information corresponding to the video images of the short video. Low-quality comment information is filtered out through a multimedia resource classification model 11, which can accurately identify low-quality comments containing vulgar metaphors or satire; and is highlighted with a red box to be shown to the reviewing user, thereby improving the efficiency and accuracy of comment quality review.

[0216] This specification also provides a training device for a multimedia resource classification model, such as... Figure 10 As shown, the device includes:

[0217] The sample resource acquisition module 1010 is used to acquire the first sample multimedia resource labeled with the first sample category label, and the second sample multimedia resource labeled with the second sample category label and the sample classification reason label;

[0218] The first classification module 1020 is used to input the first sample multimedia resource into a preset discriminant model for classification prediction, and obtain the first sample classification result of the first sample multimedia resource.

[0219] The second classification module 1030 is used to input the second sample multimedia resource into a preset generative model for classification prediction, and obtain the second sample classification result and the sample classification reason result of the second sample multimedia resource.

[0220] The target loss determination module 1040 is used to determine target loss data based on the difference between the first sample classification result and the first sample category label, the difference between the second sample classification result and the second sample category label, and the difference between the sample classification reason result and the sample classification reason label.

[0221] The model training module 1050 is used to train the preset discriminative model and the preset generative model based on the target loss data to obtain a multimedia resource classification model.

[0222] In some embodiments, the model training module includes:

[0223] The parameter adjustment unit is used to adjust the model parameters of the preset discriminative model and the preset generative model according to the target loss data until the training termination condition is met.

[0224] The model determination unit is used to determine the preset discriminative model at the end of training as a discriminative classification model and the preset generative model at the end of training as a generative classification model.

[0225] The model combination unit is used to combine the discriminative classification model and the generative classification model into a multimedia resource classification model.

[0226] In some embodiments, the apparatus further includes:

[0227] The prompt text acquisition module is used to acquire the first prompt text corresponding to the classification task of the preset discriminative model and the second prompt text corresponding to the classification task of the preset generative model.

[0228] The first classification module is further configured to input the first sample multimedia resource and the first prompt text into a preset discriminant model for classification prediction, and obtain the first sample classification result of the first sample multimedia resource;

[0229] The second classification module is further configured to input the second sample multimedia resource and the second prompt text into a preset generative model for classification prediction, and obtain the second sample classification result and the sample classification reason result of the second sample multimedia resource.

[0230] In some embodiments, the first sample multimedia resource includes multimodal data, the preset discriminative model includes a first image feature extraction network, a first text feature extraction network, and a first classification network, and the first classification module includes:

[0231] The first sample acquisition unit is used to acquire a first sample image and a first sample text that are related in the first sample multimedia resources;

[0232] The first image feature extraction unit is used to input the first sample image into the first image feature extraction network for image feature extraction processing to obtain the first sample image features;

[0233] The first sample fusion unit is used to input the first sample text and the first prompt text into the first text feature extraction network for text feature extraction and feature fusion processing to obtain the first sample fused text features.

[0234] The first classification unit is used to input the image features of the first sample and the fused text features of the first sample into the first classification network for classification prediction, so as to obtain the first sample classification result of the multimedia resource of the first sample.

[0235] In some embodiments, the preset generative model includes a second image feature extraction network, a second text feature extraction network, and a second classification network, wherein the second classification module includes:

[0236] The second sample acquisition unit is used to acquire second sample images and second sample text that are related in the second sample multimedia resources;

[0237] The second image feature extraction unit is used to input the second sample image into the second image feature extraction network for image feature extraction processing to obtain the features of the second sample image.

[0238] The second sample fusion unit is used to input the second sample text and the second prompt text into the second text feature extraction network for text feature extraction and feature fusion processing to obtain the second sample fused text features.

[0239] The second classification unit is used to input the image features of the second sample and the fused text features of the second sample into the second classification network for classification prediction, so as to obtain the second sample classification result and the sample classification reason result of the multimedia resource of the second sample.

[0240] In some embodiments, the first image feature extraction network and the second image feature extraction network have the same structure, and the first text feature extraction network and the second text feature extraction network have the same structure. The model training module includes:

[0241] The first parameter adjustment unit is used to adjust the network parameters of the first image feature extraction network based on the target loss data to obtain the first network parameters, and to adjust the network parameters of the first text feature extraction network to obtain the second network parameters; and to adjust the network parameters of the first classification network and the second classification network.

[0242] The second parameter adjustment unit is used to adjust the network parameters of the second image feature extraction network to the first network parameters, and to adjust the network parameters of the second text feature extraction network to the second network parameters.

[0243] The model training unit is used to train the preset discriminative model with adjusted parameters based on the first sample multimedia resources, and to train the preset generative model with adjusted parameters based on the second sample multimedia resources, so as to obtain a multimedia resource classification model.

[0244] In some embodiments, the first sample multimedia resource further includes multiple monomodal data, the multiple monomodal data including sample monomodal text and sample monomodal images, the preset discriminative model further includes a monomodal text extraction network, a monomodal image extraction network, and a third classification network, and the device further includes:

[0245] The single-modal text extraction module is used to input the sample single-modal text into the single-modal text extraction network for text feature extraction to obtain the sample single-modal text features;

[0246] A single-modal image extraction module is used to input the sample single-modal image into the single-modal image extraction network for image feature extraction to obtain the sample single-modal image features;

[0247] The first classification unit includes:

[0248] The first classification prediction subunit is used to input the first sample image features and the first sample fused text features into the first classification network to perform classification prediction and obtain the first classification prediction result.

[0249] The second classification prediction subunit is used to input the sample single-modal text features and the sample single-modal image features into the third classification network for classification prediction to obtain the second classification prediction result;

[0250] The first classification subunit is used to determine the first sample classification result of the first sample multimedia resource based on the first classification prediction result and the second classification prediction result.

[0251] In some embodiments, the first classification module is further configured to input the first sample multimedia resource into a preset discriminative model for classification prediction to obtain an initial sample classification result of the first sample multimedia resource and a predicted probability of the initial sample classification result; and if the predicted probability is less than a preset probability threshold, input the first sample multimedia resource into the preset generative model for classification prediction to obtain the first sample classification result.

[0252] In some embodiments, the sample resource acquisition module includes:

[0253] The sample resource acquisition unit is used to acquire multiple sample multimedia resources;

[0254] The resource partitioning unit is used to divide the plurality of sample multimedia resources into the first sample multimedia resource and the second sample multimedia resource according to at least one of the sample resource content length and sample application scenario corresponding to each sample multimedia resource.

[0255] The label determination unit is used to label the first sample multimedia resource with the first sample category label, and to label the second sample multimedia resource with the second sample category label and the sample classification reason label.

[0256] In some embodiments, the resource partitioning unit includes:

[0257] The sample scenario acquisition subunit is used to acquire the sample resource content length and sample application scenario corresponding to each sample multimedia resource.

[0258] The first resource determination subunit is used to determine the sample multimedia resources that meet at least one of the first and second conditions as the first sample multimedia resources; the first condition is that the length of the sample resource content is less than a preset length threshold, and the second condition is that the sample application scenario is not a resource recommendation scenario and does not require an explanation of the classification reason;

[0259] The second resource determination subunit is used to determine the sample multimedia resources that do not meet the first condition and do not meet the second condition as the second sample multimedia resources.

[0260] This specification also provides a multimedia resource classification device, such as... Figure 11 As shown, the device includes:

[0261] The target resource acquisition module 1110 is used to acquire target multimedia resources and filter out target classification models that match the target multimedia resources from multimedia resource classification models; the multimedia resource classification models include discriminative classification models and generative classification models; the multimedia resource classification models are trained using the above-described methods.

[0262] The first prediction module 1120 is used to input the target multimedia resource into the discriminant classification model for classification prediction when the target classification model is the discriminant classification model, so as to obtain the first classification result of the target multimedia resource.

[0263] The second prediction module 1130 is used to input the target multimedia resource into the generative classification model for classification prediction when the target classification model is the generative classification model, so as to obtain the second classification result of the target multimedia resource and the reason for the target classification.

[0264] In some embodiments, the target resource acquisition module includes:

[0265] The target scene acquisition unit is used to acquire the target resource content length and target application scene of the target multimedia resource.

[0266] The first matching unit is configured to determine the discriminative classification model as a target classification model that matches the target multimedia resource if the target multimedia resource satisfies at least one of the first target condition and the second target condition; the first target condition is that the length of the target resource content is less than a preset length threshold, and the second target condition is that the target application scenario is not a resource recommendation scenario and does not require an explanation of the classification reason.

[0267] The second matching unit is used to determine the generative classification model as the target classification model that matches the target multimedia resource if the target multimedia resource does not meet the first condition and does not meet the second condition.

[0268] In some embodiments, the target multimedia resources are at least two, and the target resource acquisition module includes:

[0269] A preset split ratio determination unit is used to determine the preset split ratio of the discriminative classification model and the generative classification model;

[0270] The target resource partitioning unit is used to divide the multiple target multimedia resources into a first target multimedia resource and a second target multimedia resource according to the preset splitting ratio; wherein, the ratio of the total number of the first target multimedia resources to the total number of the second target multimedia resources is equal to the preset splitting ratio;

[0271] The model determination unit is used to determine the target classification model corresponding to the first target multimedia resource as the discriminative classification model, and to determine the target classification model corresponding to the second target multimedia resource as the generative classification model.

[0272] In some embodiments, the preset split ratio determination unit includes:

[0273] The test resource acquisition subunit is used to acquire test multimedia resources; the test multimedia resources are labeled with test category tags.

[0274] The classification test subunit is used to input the test multimedia resources into the discriminative classification model and the generative classification model respectively for classification testing, and obtain the first test result output by the discriminative classification model and the second test result of the generative classification model.

[0275] The first accuracy determination subunit is used to determine the first accuracy of the discriminative classification model based on the difference between the first test result and the test category label.

[0276] The second accuracy determination subunit is used to determine the second accuracy of the generative classification model based on the difference between the second test result and the test category label.

[0277] The split ratio determination subunit is used to determine the preset split ratio of the discriminative classification model and the generative classification model based on the accuracy ratio of the first accuracy and the second accuracy; the preset split ratio is positively correlated with the accuracy ratio.

[0278] The apparatus and method embodiments described herein are based on the same inventive concept.

[0279] This specification provides an electronic device including a processor and a memory. The memory stores at least one instruction or at least one program, which is loaded and executed by the processor to implement the training method for the multimedia resource classification model provided in the above method embodiments.

[0280] Embodiments of this application also provide a computer storage medium, which can be disposed in a terminal to store at least one instruction or at least one program related to the training method of a multimedia resource classification model in the method embodiment. The at least one instruction or at least one program is loaded and executed by the processor to implement the training method of the multimedia resource classification model provided in the above method embodiment.

[0281] Embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a training method for the multimedia resource classification model provided in the above-described method embodiments.

[0282] Optionally, in the embodiments of this specification, the storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0283] The memory described in the embodiments of this specification can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for the functions, etc.; the data storage area may store data created according to the use of the device, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory may also include a memory controller to provide the processor with access to the memory.

[0284] The training method for the multimedia resource classification model provided in this specification can be executed on a mobile terminal, computer terminal, server, or similar computing device. Taking running on a server as an example... Figure 12 This is a hardware structure block diagram of a server for a training method of a multimedia resource classification model provided in the embodiments of this specification. For example... Figure 12As shown, the server 1200 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 1210 (CPUs 1210 may include, but are not limited to, microprocessors (MCUs) or programmable logic devices (FPGAs), a memory 1230 for storing data, and one or more storage media 1220 (e.g., one or more mass storage devices) for storing application programs 1223 or data 1222. The memory 1230 and storage media 1220 may be temporary or persistent storage. The program stored in the storage media 1220 may include one or more modules, each module including a series of instruction operations on the server. Furthermore, the CPU 1210 may be configured to communicate with the storage media 1220 and execute the series of instruction operations stored in the storage media 1220 on the server 1200. Server 1200 may also include one or more power supplies 1260, one or more wired or wireless network interfaces 1250, one or more input / output interfaces 1240, and / or one or more operating systems 1221, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0285] The input / output interface 1240 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of server 1200. In one example, the input / output interface 1240 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface 1240 may be a radio frequency (RF) module for wireless communication with the Internet.

[0286] Those skilled in the art will understand that Figure 12 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, server 1200 may also include... Figure 12 The more or fewer components shown, or having the same Figure 12 The different configurations shown.

[0287] As can be seen from the embodiments of the training method, apparatus, device, or storage medium for the multimedia resource classification model provided in this application, this application obtains a first sample multimedia resource labeled with a first sample category label, and a second sample multimedia resource labeled with a second sample category label and a sample classification reason label; then, the first sample multimedia resource is input into a preset discriminative model for classification prediction to obtain a first sample classification result of the first sample multimedia resource; the second sample multimedia resource is input into a preset generative model for classification prediction to obtain a second sample classification result of the second sample multimedia resource and a sample classification reason result; this realizes that different models are used to process sample multimedia resources with different labels; and then, based on the first sample classification result and the first sample category label... The differences between the labels, the differences between the second sample classification result and the second sample category label, and the differences between the sample classification reason result and the sample classification reason label are used to determine the target loss data. Thus, the target loss data can be jointly determined based on the output results of the two models. Then, the preset discriminative model and the preset generative model are trained synchronously based on the target loss data, thereby obtaining a multimedia resource classification model containing two models through joint training. Compared to training the two models separately, this application adjusts the parameters of both models simultaneously based on a single target loss data, reducing the computational power during model training. In application, different models can be selected from the two models for different target multimedia resources, improving the flexibility of model selection.

[0288] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments of this specification have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0289] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0290] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer storage medium, such as a read-only memory, a disk, or an optical disk.

[0291] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for training a multimedia resource classification model, characterized in that, The method includes: Obtain the first sample multimedia resource labeled with the first sample category label, and the second sample multimedia resource labeled with the second sample category label and the sample classification reason label; The first sample multimedia resource is input into a preset discriminant model for classification and prediction to obtain the first sample classification result of the first sample multimedia resource. The second sample multimedia resource is input into a preset generative model for classification and prediction, and the second sample classification result and the sample classification reason result are obtained. Based on the difference between the first sample classification result and the first sample category label, the difference between the second sample classification result and the second sample category label, and the difference between the sample classification reason result and the sample classification reason label, the target loss data is determined; The preset discriminative model and the preset generative model are trained based on the target loss data to obtain a multimedia resource classification model.

2. The method of claim 1, wherein, The step of training the preset discriminative model and the preset generative model based on the target loss data to obtain a multimedia resource classification model includes: Adjust the model parameters of the preset discriminative model and the preset generative model according to the target loss data until the training termination condition is met; The preset discriminative model at the end of training is determined as a discriminative classification model, and the preset generative model at the end of training is determined as a generative classification model. The discriminative classification model and the generative classification model are combined into a multimedia resource classification model.

3. The method of claim 1, wherein, Before inputting the first sample multimedia resource into a preset discriminative model for classification prediction to obtain the first sample classification result of the first sample multimedia resource, the method further includes: Obtain the first prompt text corresponding to the classification task of the preset discriminative model and the second prompt text corresponding to the classification task of the preset generative model; The step of inputting the first sample multimedia resource into a preset discriminative model for classification prediction to obtain the first sample classification result of the first sample multimedia resource includes: The first sample multimedia resource and the first prompt text are input into a preset discriminant model for classification and prediction to obtain the first sample classification result of the first sample multimedia resource. The step of inputting the second sample multimedia resource into a preset generative model for classification prediction, and obtaining the second sample classification result and the sample classification reason result of the second sample multimedia resource, includes: The second sample multimedia resource and the second prompt text are input into a preset generative model for classification and prediction to obtain the second sample classification result and the sample classification reason result of the second sample multimedia resource.

4. The method of claim 3, wherein, The first sample multimedia resource includes multimodal data, and the preset discriminative model includes a first image feature extraction network, a first text feature extraction network, and a first classification network. The step of inputting the first sample multimedia resource and the first prompt text into the preset discriminative model for classification prediction to obtain the first sample classification result of the first sample multimedia resource includes: Obtain the first sample image and the first sample text that are related in the first sample multimedia resources; The first sample image is input into the first image feature extraction network for image feature extraction processing to obtain the features of the first sample image. The first sample text and the first prompt text are input into the first text feature extraction network for text feature extraction and feature fusion processing to obtain the first sample fused text features. The first sample image features and the first sample fused text features are input into the first classification network for classification prediction to obtain the first sample classification result of the first sample multimedia resource.

5. The method of claim 4, wherein, The preset generative model includes a second image feature extraction network, a second text feature extraction network, and a second classification network. The second sample multimedia resource and the second prompt text are input into the preset generative model for classification prediction to obtain the second sample classification result and the sample classification reason result for the second sample multimedia resource, including: Obtain the second sample image and the second sample text that are related from the second sample multimedia resources; The second sample image is input into the second image feature extraction network for image feature extraction processing to obtain the features of the second sample image; The second sample text and the second prompt text are input into the second text feature extraction network for text feature extraction and feature fusion processing to obtain the second sample fused text features. The second sample image features and the second sample fused text features are input into the second classification network for classification prediction, so as to obtain the second sample classification result and the sample classification reason result of the second sample multimedia resource.

6. The method of claim 5, wherein, The first image feature extraction network and the second image feature extraction network have the same structure, and the first text feature extraction network and the second text feature extraction network have the same structure. The step of training the preset discriminative model and the preset generative model based on the target loss data to obtain a multimedia resource classification model includes: Based on the target loss data, the network parameters of the first image feature extraction network are adjusted to obtain the first network parameters, and the network parameters of the first text feature extraction network are adjusted to obtain the second network parameters; and the network parameters of the first classification network and the second classification network are adjusted. The network parameters of the second image feature extraction network are adjusted to the first network parameters, and the network parameters of the second text feature extraction network are adjusted to the second network parameters. The preset discriminative model with adjusted parameters is trained based on the first sample multimedia resources, and the preset generative model with adjusted parameters is trained based on the second sample multimedia resources to obtain a multimedia resource classification model.

7. The method according to claim 4, characterized in that, The first sample multimedia resource further includes multiple unimodal data, which include sample unimodal text and sample unimodal images. The preset discriminative model further includes a unimodal text extraction network, a unimodal image extraction network, and a third classification network. The method further includes: The sample unimodal text is input into the unimodal text extraction network for text feature extraction to obtain the sample unimodal text features; The sample single-modal image is input into the single-modal image extraction network for image feature extraction to obtain the sample single-modal image features; The step of inputting the first sample image features and the first sample fused text features into the first classification network for classification prediction to obtain the first sample multimedia resource classification result includes: The first sample image features and the first sample fused text features are input into the first classification network for classification prediction to obtain the first classification prediction result. The sample single-modal text features and the sample single-modal image features are input into the third classification network for classification prediction to obtain the second classification prediction result; Based on the first classification prediction result and the second classification prediction result, the first sample classification result of the multimedia resource of the first sample is determined.

8. The method according to claim 1, characterized in that, The step of inputting the first sample multimedia resource into a preset discriminative model for classification prediction to obtain the first sample classification result of the first sample multimedia resource includes: The first sample multimedia resource is input into a preset discriminant model for classification and prediction to obtain the initial sample classification result of the first sample multimedia resource and the predicted probability of the initial sample classification result. If the predicted probability is less than a preset probability threshold, the first sample multimedia resource is input into the preset generative model for classification prediction to obtain the classification result of the first sample.

9. The method according to claim 1, characterized in that, The acquisition of the first sample multimedia resource labeled with the first sample category label, and the second sample multimedia resource labeled with the second sample category label and the sample classification reason label, includes: Acquire multimedia resources from multiple samples; Based on at least one of the sample resource content length and sample application scenarios corresponding to each sample multimedia resource, the plurality of sample multimedia resources are divided into the first sample multimedia resource and the second sample multimedia resource. The first sample multimedia resource is labeled with the first sample category label, and the second sample multimedia resource is labeled with the second sample category label and the sample classification reason label.

10. The method according to claim 9, characterized in that, The step of dividing the multiple sample multimedia resources into first sample multimedia resources and second sample multimedia resources based on at least one of the following: sample resource content length and sample application scenario corresponding to each sample multimedia resource, includes: Obtain the sample resource content length and sample application scenario corresponding to each sample multimedia resource; The sample multimedia resources that meet at least one of the first and second conditions are determined as the first sample multimedia resources; the first condition is that the content length of the sample resource is less than a preset length threshold, and the second condition is that the application scenario of the sample is not a resource recommendation scenario and does not require an explanation of the classification reason. Multimedia resources that do not meet the first condition and the second condition are identified as the second sample multimedia resources.

11. A multimedia resource classification method, characterized in that, The method includes: The method involves acquiring target multimedia resources and selecting target classification models that match the target multimedia resources from a multimedia resource classification model; the multimedia resource classification model includes a discriminative classification model and a generative classification model; the multimedia resource classification model is trained using the method described in any one of claims 1-10. When the target classification model is the discriminant classification model, the target multimedia resource is input into the discriminant classification model for classification prediction to obtain the first classification result of the target multimedia resource; When the target classification model is the generative classification model, the target multimedia resource is input into the generative classification model for classification prediction to obtain the second classification result of the target multimedia resource and the reason for the target classification.

12. The method according to claim 11, characterized in that, The step of selecting a target classification model from the multimedia resource classification models that matches the target multimedia resource includes: Obtain the target resource content length and target application scenario of the target multimedia resource; If the target multimedia resource satisfies at least one of the first target condition and the second target condition, the discriminative classification model is determined as the target classification model that matches the target multimedia resource; the first target condition is that the length of the target resource content is less than a preset length threshold, and the second target condition is that the target application scenario is not a resource recommendation scenario and no explanation of the classification reason is required; If the target multimedia resource does not meet the first condition and does not meet the second condition, the generative classification model is determined as the target classification model that matches the target multimedia resource.

13. The method according to claim 11, characterized in that, The target multimedia resources are at least two, and the step of selecting a target classification model from the multimedia resource classification models that matches the target multimedia resources includes: Determine the preset split ratio for the discriminative classification model and the generative classification model; The multiple target multimedia resources are divided into first target multimedia resources and second target multimedia resources according to the preset splitting ratio; wherein, the ratio of the total number of first target multimedia resources to the total number of second target multimedia resources is equal to the preset splitting ratio; The target classification model corresponding to the first target multimedia resource is determined as the discriminative classification model, and the target classification model corresponding to the second target multimedia resource is determined as the generative classification model.

14. The method according to claim 13, characterized in that, Determining the preset split ratio of the discriminative classification model and the generative classification model includes: Obtain test multimedia resources; the test multimedia resources are labeled with test category tags; The test multimedia resources are respectively input into the discriminative classification model and the generative classification model for classification testing, and the first test result output by the discriminative classification model and the second test result of the generative classification model are obtained. Based on the difference between the first test result and the test category label, determine the first accuracy of the discriminative classification model; The second accuracy of the generative classification model is determined based on the difference between the second test result and the test category label; Based on the ratio of the first accuracy rate to the second accuracy rate, a preset split ratio is determined for the discriminative classification model and the generative classification model; the preset split ratio is positively correlated with the accuracy ratio.

15. A training device for a multimedia resource classification model, characterized in that, The device includes: The sample resource acquisition module is used to acquire the first sample multimedia resource labeled with the first sample category label, and the second sample multimedia resource labeled with the second sample category label and the sample classification reason label; The first classification module is used to input the first sample multimedia resource into a preset discriminant model for classification prediction, and obtain the first sample classification result of the first sample multimedia resource. The second classification module is used to input the second sample multimedia resource into a preset generative model for classification prediction, and to obtain the second sample classification result and the sample classification reason result of the second sample multimedia resource. The target loss determination module is used to determine target loss data based on the difference between the first sample classification result and the first sample category label, the difference between the second sample classification result and the second sample category label, and the difference between the sample classification reason result and the sample classification reason label. The model training module is used to train the preset discriminative model and the preset generative model based on the target loss data to obtain a multimedia resource classification model.