Model training method, multi-task joint prediction method, device, equipment and medium

Abstract

The present disclosure provides a model training method, a multi-task joint prediction method, an apparatus, a device and a medium. The training method includes: obtaining multimedia training samples, each multimedia training sample includes a sample multimedia resource, a sample task label and a sample state label; inputting the sample multimedia resource into a feature extraction model for feature extraction to obtain the features of the sample multimedia resource; inputting the features of the sample multimedia resource and the sample task label into a conditional probability model for prediction to obtain the predicted state of the sample multimedia resource; inputting the features of the sample multimedia resource and the sample state label into a posterior probability model for prediction to obtain the predicted classification task result of the sample multimedia resource; determining the loss function of the multi-task joint model based on the sample task label, the sample state label, the predicted classification task result and the predicted state of the sample multimedia resource; adjusting the model parameters of the multi-task joint model based on the loss function.

Description

Technical Field

[0001] The present disclosure generally relates to the field of electronic technology, and more specifically, to a model training method, a multi-task joint prediction method, a device, a equipment and a medium. Background Art

[0002] A variety of different judgment rules can be defined for the same problem. For example, the judgment rules defined by different countries for the same judgment problem are both similar and different. Accordingly, different judgment rules can correspond to different prediction tasks, and each prediction task is used to predict the judgment result of the sample according to the corresponding judgment rule.

[0003] In addition, even if the decision rules defined for the same problem are the same, the distribution of data used for the same problem in different domains (e.g., different countries) is different. Accordingly, different domains can correspond to different prediction tasks, and each prediction task is used to predict the decision results of the sample according to the data distribution of the corresponding domain.

[0004] Figure 1 The figure shows a multi-task joint model of a feature hard sharing mode commonly used in related technologies. A, B, and C are the output sides of three tasks respectively. For example, these three tasks are aimed at the same judgment problem, but the judgment rules corresponding to these three tasks are different or the corresponding domains are different. It can be seen that for the existing multi-task joint model, different tasks are distinguished by different output sides. Therefore, the effect and scalability of the existing multi-task joint model need to be improved. Summary of the invention

[0005] The exemplary embodiments of the present disclosure provide a model training method, a multi-task joint prediction method, an apparatus, a device and a medium, which improve the effect and scalability of the multi-task joint model.

[0006] According to a first aspect of an embodiment of the present disclosure, a training method for a multi-task joint model is provided, wherein the multi-task joint model includes a feature extraction model, a conditional probability model and a posterior probability model, and the training method includes: obtaining multimedia training samples, wherein each of the multimedia training samples includes a sample multimedia resource, a sample task label and a sample state label; the sample task label is used to characterize the actual classification task corresponding to the sample multimedia resource, and the sample state label is used to characterize the actual state of the sample multimedia resource under the actual classification task; inputting the sample multimedia resource into the feature extraction model for feature extraction to obtain the feature of the sample multimedia resource; inputting the feature of the sample multimedia resource and the sample task label into the conditional probability model for prediction to obtain the predicted state of the sample multimedia resource; inputting the feature of the sample multimedia resource and the sample state label into the posterior probability model for prediction to obtain the predicted classification task result of the sample multimedia resource; determining the loss function of the multi-task joint model based on the sample task label, the sample state label, the predicted classification task result and the predicted state of the sample multimedia resource; and adjusting the model parameters of the multi-task joint model based on the loss function to train the multi-task joint model.

[0007] Optionally, the step of inputting the features of the sample multimedia resource and the sample task label into the conditional probability model for prediction to obtain the predicted state of the sample multimedia resource includes: obtaining the features of the classification task corresponding to the sample task label; inputting both the features of the classification task and the features of the sample multimedia resource into a first concatenated layer or a first attention network to obtain a first joint feature; inputting the first joint feature into a first fully connected layer to obtain the predicted state of the sample multimedia resource under the classification task output by the first fully connected layer.

[0008] Optionally, the step of obtaining the features of the classification task corresponding to the sample task label includes: taking the embedded features at the position corresponding to the classification task in the first matrix as the features of the classification task; wherein each row in the first matrix corresponds one-to-one to each classification task; wherein, in the step of adjusting the model parameters of the multi-task joint model based on the loss function, the first matrix is ​​adjusted according to the loss function.

[0009] Optionally, the step of inputting the features of the sample multimedia resource and the sample state label into the posterior probability model for prediction to obtain the prediction classification task result of the sample multimedia resource includes: obtaining the features of the multimedia resource state corresponding to the sample state label; inputting both the features of the multimedia resource state and the features of the sample multimedia resource into a second splicing layer or a second attention network to obtain a second joint feature; inputting the second joint feature into a second fully connected layer to obtain the predicted state of the sample multimedia resource output by the second fully connected layer as the prediction classification task result of the multimedia resource state.

[0010] Optionally, the step of obtaining the characteristics of the multimedia resource state corresponding to the sample state label includes: taking the embedded features at the position corresponding to the multimedia resource state in the second matrix as the characteristics of the multimedia resource state; wherein, each row in the second matrix corresponds one-to-one to various multimedia resource states; wherein, in the step of adjusting the model parameters of the multi-task joint model based on the loss function, the second matrix is ​​adjusted according to the loss function.

[0011] Optionally, the sample multimedia resource is a sample video, wherein the step of inputting the sample multimedia resource into the feature extraction model for feature extraction to obtain the features of the sample multimedia resource includes: acquiring M frames of images based on the sample video, wherein M is a positive integer; inputting the M frames of images into a feature extraction network to obtain feature values ​​of D-dimensional features of each frame of the image, wherein D is a positive integer; and obtaining features for describing the content of the sample video from the time dimension and the space dimension based on the feature values ​​of the D-dimensional features of the M frames of images.

[0012] Optionally, the step of obtaining features for describing the content of the sample video from the time dimension and the space dimension based on the feature values ​​of the D-dimensional features of the M-frame images includes: inputting the feature values ​​of the D-dimensional features of the M-frame images into a cross-modal feature interaction model to learn the inter-frame association relationship of the M-frame images to obtain an M*D-dimensional feature vector; determining the maximum eigenvalue corresponding to each of the D feature dimensions in the M*D-dimensional feature vector, and using the vector composed of the D maximum eigenvalues ​​as the feature of the content of the sample video.

[0013] Optionally, the step of determining the loss function of the multi-task joint model based on the sample task label, sample state label, predicted classification task result and predicted state of the sample multimedia resource includes: determining a first loss function based on the predicted state of the sample multimedia resource and the sample state label; determining a second loss function based on the predicted classification task result of the sample multimedia resource and the sample task label; performing weighted summation of the first loss function and the second loss function to obtain the loss function of the multi-task joint model; wherein the weighted weight of the first loss function is greater than the weighted weight of the second loss function.

[0014] According to a second aspect of an embodiment of the present disclosure, a multi-task joint prediction method is provided, which is executed based on a multi-task joint model, and the multi-task joint model includes a feature extraction model, a conditional probability model and a posterior probability model. The multi-task joint prediction method includes: when a target multimedia resource to be predicted and another input quantity are received, the target multimedia resource is input into the feature extraction model for feature extraction to obtain the characteristics of the target multimedia resource; when the other input quantity is a classification task identifier, the characteristics of the target multimedia resource and the classification task identifier are input into the conditional probability model for prediction to obtain the predicted state of the target multimedia resource; when the other input quantity is a multimedia resource state identifier, the characteristics of the target multimedia resource and the multimedia resource state identifier are input into the posterior probability model for prediction to obtain the predicted classification task result of the target multimedia resource.

[0015] Optionally, the step of inputting the characteristics of the target multimedia resource and the classification task identifier into the conditional probability model for prediction to obtain the predicted state of the target multimedia resource includes: acquiring the characteristics of the classification task corresponding to the classification task identifier; inputting both the characteristics of the classification task and the characteristics of the target multimedia resource into a first concatenated layer or a first attention network to obtain a first joint feature; inputting the first joint feature into a first fully connected layer to obtain the predicted state of the target multimedia resource under the classification task output by the first fully connected layer.

[0016] Optionally, the step of obtaining the characteristics of the classification task corresponding to the classification task identifier includes: using the embedded features at the position corresponding to the classification task in the trained first matrix as the characteristics of the classification task; wherein each row in the first matrix corresponds one-to-one to each classification task.

[0017] Optionally, the step of inputting the characteristics of the target multimedia resource and the multimedia resource state identifier into the posterior probability model for prediction to obtain the prediction classification task result of the target multimedia resource includes: acquiring the characteristics of the multimedia resource state corresponding to the multimedia resource state identifier; inputting both the characteristics of the multimedia resource state and the characteristics of the target multimedia resource into a second splicing layer or a second attention network to obtain a second joint feature; inputting the second joint feature into a second fully connected layer to obtain the predicted state of the target multimedia resource output by the second fully connected layer as the prediction classification task result of the multimedia resource state.

[0018] Optionally, the step of obtaining the characteristics of the multimedia resource state corresponding to the multimedia resource state identifier includes: using the embedded features at the position corresponding to the multimedia resource state in the trained second matrix as the characteristics of the multimedia resource state; wherein, each row in the second matrix corresponds one-to-one to various multimedia resource states.

[0019] Optionally, the target multimedia resource is a target video, wherein the step of inputting the target multimedia resource into the feature extraction model for feature extraction to obtain the features of the target multimedia resource includes: acquiring M frames of images based on the target video, wherein M is a positive integer; inputting the M frames of images into a feature extraction network to obtain feature values ​​of D-dimensional features of each frame of the image, wherein D is a positive integer; and obtaining features for describing the content of the target video from time and space dimensions based on the feature values ​​of the D-dimensional features of the M frames of images.

[0020] Optionally, the step of obtaining features for describing the content of the target video from the time dimension and the space dimension based on the feature values ​​of the D-dimensional features of the M-frame images includes: inputting the feature values ​​of the D-dimensional features of the M-frame images into a cross-modal feature interaction model to learn the inter-frame association relationship of the M-frame images to obtain an M*D-dimensional feature vector; determining the maximum eigenvalue corresponding to each of the D feature dimensions in the M*D-dimensional feature vector, and using the vector composed of the D maximum eigenvalues ​​as the feature of the content of the target video.

[0021] According to a third aspect of an embodiment of the present disclosure, a training device for a multi-task joint model is provided, wherein the multi-task joint model includes a feature extraction model, a conditional probability model and a posterior probability model, and the training device includes: a training sample acquisition unit, configured to acquire multimedia training samples, wherein each of the multimedia training samples includes a sample multimedia resource, a sample task label and a sample status label; the sample task label is used to characterize the actual classification task corresponding to the sample multimedia resource, and the sample status label is used to characterize the actual status of the sample multimedia resource under the actual classification task; a feature extraction unit, configured to input the sample multimedia resource into the feature extraction model for feature extraction, and obtain the features of the sample multimedia resource; A prediction unit is configured to input the characteristics of the sample multimedia resource and the sample task label into the conditional probability model for prediction, so as to obtain the predicted state of the sample multimedia resource; a second prediction unit is configured to input the characteristics of the sample multimedia resource and the sample state label into the posterior probability model for prediction, so as to obtain the predicted classification task result of the sample multimedia resource; a loss function determination unit is configured to determine the loss function of the multi-task joint model based on the sample task label, sample state label, predicted classification task result and predicted state of the sample multimedia resource; a training unit is configured to adjust the model parameters of the multi-task joint model based on the loss function to train the multi-task joint model.

[0022] Optionally, the first prediction unit is configured to: obtain the characteristics of the classification task corresponding to the sample task label; input the characteristics of the classification task and the characteristics of the sample multimedia resource into the first splicing layer or the first attention network to obtain a first joint feature; input the first joint feature into the first fully connected layer to obtain the predicted state of the sample multimedia resource under the classification task output by the first fully connected layer.

[0023] Optionally, the first prediction unit is configured to use the embedded features at the position corresponding to the classification task in the first matrix as the features of the classification task; wherein each row in the first matrix corresponds one-to-one to each classification task; wherein the training unit is configured to adjust the first matrix according to the loss function.

[0024] Optionally, the second prediction unit is configured to: obtain the characteristics of the multimedia resource state corresponding to the sample state label; input the characteristics of the multimedia resource state and the characteristics of the sample multimedia resource into a second splicing layer or a second attention network to obtain a second joint feature; input the second joint feature into a second fully connected layer to obtain the predicted state of the sample multimedia resource output by the second fully connected layer as a prediction classification task result of the multimedia resource state.

[0025] Optionally, the second prediction unit is configured to use the embedded features at the position corresponding to the multimedia resource state in the second matrix as the features of the multimedia resource state; wherein each row in the second matrix corresponds one-to-one to various multimedia resource states; wherein the training unit is configured to adjust the second matrix according to the loss function.

[0026] Optionally, the sample multimedia resource is a sample video, wherein the feature extraction unit is configured to: obtain M frames of images based on the sample video, wherein M is a positive integer; input the M frames of images into a feature extraction network to obtain feature values ​​of D-dimensional features of each frame of image, wherein D is a positive integer; based on the feature values ​​of the D-dimensional features of the M frames of images, obtain features for describing the content of the sample video from a time dimension and a space dimension.

[0027] Optionally, the feature extraction unit is configured to: input the feature values ​​of the D-dimensional features of the M frame images into a cross-modal feature interaction model to learn the inter-frame correlation relationship of the M frame images to obtain an M*D-dimensional feature vector; determine the maximum eigenvalue corresponding to each of the D feature dimensions in the M*D-dimensional feature vector, and use the vector composed of the D maximum eigenvalues ​​as the feature of the content of the sample video.

[0028] Optionally, the loss function determination unit is configured to: determine a first loss function based on the predicted state of the sample multimedia resource and the sample state label; determine a second loss function based on the predicted classification task result of the sample multimedia resource and the sample task label; perform weighted summation of the first loss function and the second loss function to obtain the loss function of the multi-task joint model; wherein the weighted weight of the first loss function is greater than the weighted weight of the second loss function.

[0029] According to a fourth aspect of an embodiment of the present disclosure, a multi-task joint prediction device is provided, which performs operations based on a multi-task joint model, wherein the multi-task joint model includes a feature extraction model, a conditional probability model and a posterior probability model, and the multi-task joint prediction device includes: a feature extraction unit, which is configured to input the target multimedia resource into the feature extraction model for feature extraction when receiving a target multimedia resource to be predicted and another input quantity, so as to obtain the characteristics of the target multimedia resource; a first prediction unit, which is configured to input the characteristics of the target multimedia resource and the classification task identifier into the conditional probability model for prediction when the other input quantity is a classification task identifier, so as to obtain the predicted state of the target multimedia resource; and a second prediction unit, which is configured to input the characteristics of the target multimedia resource and the multimedia resource state identifier into the posterior probability model for prediction when the other input quantity is a multimedia resource state identifier, so as to obtain the predicted classification task result of the target multimedia resource.

[0030] Optionally, the first prediction unit is configured to: obtain the characteristics of the classification task corresponding to the classification task identifier; input the characteristics of the classification task and the characteristics of the target multimedia resource into the first concatenation layer or the first attention network to obtain a first joint feature; input the first joint feature into the first fully connected layer to obtain the predicted state of the target multimedia resource under the classification task output by the first fully connected layer.

[0031] Optionally, the first prediction unit is configured to use the embedded features at the positions corresponding to the classification tasks in the trained first matrix as the features of the classification task; wherein each row in the first matrix corresponds one-to-one to each classification task.

[0032] Optionally, the second prediction unit is configured to: obtain the characteristics of the multimedia resource state corresponding to the multimedia resource state identifier; input the characteristics of the multimedia resource state and the characteristics of the target multimedia resource into a second splicing layer or a second attention network to obtain a second joint feature; input the second joint feature into a second fully connected layer to obtain the predicted state of the target multimedia resource output by the second fully connected layer as a prediction classification task result of the multimedia resource state.

[0033] Optionally, the second prediction unit is configured to use the embedded features at the positions corresponding to the multimedia resource states in the trained second matrix as the features of the multimedia resource states; wherein each row in the second matrix corresponds one-to-one to various multimedia resource states.

[0034] Optionally, the target multimedia resource is a target video, wherein the feature extraction unit is configured to: obtain M frames of images based on the target video, wherein M is a positive integer; input the M frames of images into a feature extraction network to obtain feature values ​​of D-dimensional features of each frame of the image, wherein D is a positive integer; and based on the feature values ​​of the D-dimensional features of the M frames of images, obtain features for describing the content of the target video from a time dimension and a space dimension.

[0035] Optionally, the feature extraction unit is configured to: input the feature values ​​of the D-dimensional features of the M frame images into a cross-modal feature interaction model to learn the inter-frame correlation relationship of the M frame images to obtain an M*D-dimensional feature vector; determine the maximum eigenvalue corresponding to each of the D feature dimensions in the M*D-dimensional feature vector, and use the vector composed of the D maximum eigenvalues ​​as the feature of the content of the target video.

[0036] According to a fifth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the multi-task joint model training method as described above and / or the multi-task joint prediction method as described above.

[0037] According to a sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the training method of the multi-task joint model as described above and / or the multi-task joint prediction method as described above.

[0038] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by at least one processor, implement the training method of the multi-task joint model as described above and / or the multi-task joint prediction method as described above.

[0039] According to the model training method, multi-task joint prediction method, device, equipment and medium of the exemplary embodiments of the present disclosure, multi-task joint optimization is implemented on the input side, thereby improving the effect and scalability of the multi-task joint model.

[0040] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0042] Figure 1 Show examples of existing multi-task joint models;

[0043] Figure 2 A flowchart showing a method for training a multi-task joint model according to an exemplary embodiment of the present disclosure;

[0044] Figure 3 A flowchart showing a method for obtaining a predicted state of a sample multimedia resource according to an exemplary embodiment of the present disclosure;

[0045] Figure 4 A flowchart showing a method for obtaining a prediction classification task result of a sample multimedia resource according to an exemplary embodiment of the present disclosure;

[0046] Figure 5 A flowchart showing a method for extracting content features of a sample video according to an exemplary embodiment of the present disclosure;

[0047] Figure 6 A flowchart showing a multi-task joint prediction method according to an exemplary embodiment of the present disclosure is shown;

[0048] Figure 7 An example of a multi-task joint prediction method according to an exemplary embodiment of the present disclosure is shown;

[0049] Figure 8 A structural block diagram of a training device for a multi-task joint model according to an exemplary embodiment of the present disclosure is shown;

[0050] Fig. 9 A structural block diagram of a multi-task joint prediction device according to an exemplary embodiment of the present disclosure is shown;

[0051] Fig.10 A structural block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0052] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0053] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0054] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0055] Figure 2 A flowchart of a training method for a multi-task joint model according to an exemplary embodiment of the present disclosure is shown. The multi-task joint model for multiple classification tasks includes: a feature extraction model, a conditional probability model, and a posterior probability model. The conditional probability model is used to predict the classification result (i.e., predicted state) of a given sample under a given classification task given a sample and a classification task; the posterior probability model is used to predict under which one or more classification tasks the classification result of a given sample will be a given classification result given a sample and a classification result (i.e., actual state), that is, to obtain a predicted classification task result.

[0056] As an example, the multiple classification tasks may be multiple classification tasks for the same classification problem. For example, if the same classification problem is to determine the status of a multimedia resource (for example, a violation status or a non-violation status, which should be understood to be other types of status), the classification results of the multiple classification tasks are the prediction results of the multimedia resource status. As an example, each of the multiple classification tasks may be a multimedia resource classification task for a classification rule (that is, the classification rule learned corresponding to the classification task), and different classification tasks may correspond to different classification rules. For example, different classification rules may correspond to different countries or regions, that is, different classification rules may be classification rules for multimedia resource status in different countries. As another example, each of the multiple classification tasks may be a multimedia resource classification task for a domain (that is, the classification ability of the classification task is learned based on the data distribution of the corresponding domain), and different classification tasks may correspond to different domains. For example, different domains may correspond to different countries.

[0057] Reference Figure 2 , in step S101, multimedia training samples are obtained.

[0058] Each multimedia training sample includes a sample multimedia resource, a sample task label and a sample status label; the sample task label is used to characterize the actual classification task corresponding to the sample multimedia resource, and the sample status label is used to characterize the actual status of the sample multimedia resource under the actual classification task.

[0059] The actual classification task is one of the multiple classification tasks. The actual state of the sample multimedia resource under the actual classification task is: the state to which the sample multimedia resource actually belongs for the classification rule corresponding to the actual classification task. For example, when the classification rule corresponding to the actual classification task is the classification rule of ** country on whether the video content is illegal, the actual state of the sample multimedia resource under the actual classification task is: the actual classification result of whether the sample multimedia resource belongs to the illegal state or the non-illegal state under the classification rule of ** country on whether the video content is illegal. For example, the actual classification result can be obtained by manual classification.

[0060] As an example, the sample task label may be the identifier of the actual classification task. For example, the identifier of the classification task may be a task id. For example, when the total number of the multiple classification tasks is T, one-hot encoding (0, 1, ..., T-1) may be performed for each classification task to obtain the task ids of T classification tasks.

[0061] As an example, the sample status label may be an identifier of a multimedia resource status. For example, the identifier of a multimedia resource status may be represented by 0 and 1 (for example, 0 represents a non-violation status, and 1 represents a violation status). For example, when the total number of types of multimedia resource status is 2, a one-hot encoding (0, 1) binary encoding may be performed for each multimedia resource status to obtain the identifiers of various multimedia resource statuses.

[0062] In step S102, the sample multimedia resource is input into the feature extraction model to extract features, so as to obtain features of the sample multimedia resource.

[0063] As an example, according to the specific object classified by the classification task, the sample multimedia resources may include but are not limited to one of sample videos, sample images, sample texts, and sample audios. As an example, when the multiple classification tasks are classification tasks for determining the video status, the content features of the sample videos may be extracted. Figure 5 An exemplary embodiment of extracting content features of a sample video is described below.

[0064] In step S103, the features of the sample multimedia resource and the sample task label are input into the conditional probability model for prediction to obtain the predicted state of the sample multimedia resource, that is, the classification result of the sample multimedia resource under the actual classification task is obtained.

[0065] For example, if the actual classification task is a classification task for country A, then the predicted state of the sample multimedia resource in country A is obtained in step S103.

[0066] As an example, the following will combine Figure 3 An exemplary embodiment of step S103 is described below.

[0067] In step S104, the features of the sample multimedia resource and the sample state label are input into the posterior probability model for prediction to obtain a predicted classification task result of the sample multimedia resource. The predicted classification task result is: under which one or more classification tasks among the multiple classification tasks the sample multimedia resource is classified to predict the actual state.

[0068] For example, if the actual state is "state 1", then in step S104, it is predicted that in which classification task or tasks the sample multimedia resource is classified, the classification result of "state 1" is obtained. For example, it is predicted that the classification result of "state 1" is obtained by classifying the sample multimedia resource under the classification task for country B, and the classification result of "state 1" is not obtained by classifying the sample multimedia resource under the classification task for country D, which also indicates that it is predicted that the sample multimedia resource in country B belongs to "state 1" and the sample multimedia resource in country D belongs to "state 0".

[0069] In fact, the present disclosure considers recombining input = (multimedia resource, classification task id), output = multimedia resource state, to form a new input = (multimedia resource, multimedia resource state), output = classification task id; that is, if the state of the multimedia resource is currently known, it can be determined that the multimedia resource is judged to be in this state because of which classification task (that is, in which bucket it appears). That is, the a posteriori form of P(X|R) = C. Among them, X represents the multimedia resource, R represents the a posteriori result, and C represents the condition.

[0070] As an example, the following will combine Figure 4 An exemplary embodiment of step S104 is described below.

[0071] In step S105, a loss function of the multi-task joint model is determined based on the sample task label, the sample state label, the predicted classification task result and the predicted state of the sample multimedia resource.

[0072] As an example, a first loss function may be determined based on the predicted state of the sample multimedia resource and the sample state label; and a second loss function may be determined based on the predicted classification task result of the sample multimedia resource and the sample task label; then, the first loss function Loss1 and the second loss function Loss2 are weightedly summed to obtain the loss function of the multi-task joint model (i.e., a*Loss1+b*Loss2). For example, a is greater than b. It should be understood that the values ​​of a and b may be set according to actual conditions and requirements, for example, a may be 1 and b may be 0.4.

[0073] In step S106, model parameters of the multi-task joint model are adjusted based on the loss function to train the multi-task joint model.

[0074] It should be understood that a plurality of multimedia training samples may be used to train the multi-task joint model. For example, positive samples and negative samples for different classification tasks may be used to train the multi-task joint model until the loss no longer decreases or the number of network iterations is reached.

[0075] As an example, the first loss function and / or the second loss function may be a cross entropy loss function. It should be understood that the first loss function and the second loss function may also be other types of loss functions, and the present disclosure is not limited to this.

[0076] As an example, a feature extraction model, a conditional probability model, and a posterior probability model of the model parameters of the multi-task joint model may be adjusted based on the loss function to train the multi-task joint model.

[0077] Figure 3A flowchart of a method for obtaining a predicted state of a sample multimedia resource according to an exemplary embodiment of the present disclosure is shown.

[0078] Reference Figure 3 In step S201, the features of the classification task corresponding to the sample task label are obtained.

[0079] As an example, the feature of the classification task may be an embedding feature. It should be understood that it may also be other types of features, and the present disclosure is not limited to this.

[0080] As an example, the embedded features at the position corresponding to the classification task in the first matrix can be used as the features of the classification task. Each row in the first matrix (T, D) corresponds to the multiple classification tasks (T classification tasks) one by one, that is, the first matrix has T rows and D columns. For example, the i-th classification task corresponds to the i-th row of the first matrix, and accordingly, the D-dimensional Embedding features of the i-th row can be used as the features of the i-th classification task. As an example, the first matrix can be a dictionary matrix.

[0081] As an example, the first matrix is ​​randomly initialized at the beginning of training, and in the subsequent training process, the first matrix can be adjusted according to the loss function of the multi-task joint model. For example, the partial derivative can be calculated using the chain rule according to the loss function, and the first matrix can be updated in the direction of decreasing loss based on the partial derivative.

[0082] In step S202, the features of the classification task and the features of the sample multimedia resources are both input into a first concatenation layer or a first attention network to obtain a first joint feature.

[0083] As an example, the features of the classification task and the features of the sample multimedia resources may both be input into the first splicing layer, that is, the features of the classification task and the features of the sample multimedia resources may be directly spliced ​​through the first splicing layer to obtain a first joint feature of 2D dimension.

[0084] As another example, both the features of the classification task and the features of the sample multimedia resources can be input into the first attention network. Specifically, the features of the classification task can be used as Q (query) in the attention mechanism, and the features of the sample multimedia resources can be used as K (key) and V (value) in the attention mechanism. As an example, the first attention network can be a multi-head attention network. It should be understood that through the first attention network, the features of different classification tasks can activate different feature dimensions of multimedia resources. In other words, the features of different classification tasks can enhance the various feature dimensions of multimedia resources to different degrees.

[0085] In step S203, the first joint feature is input into a first fully connected layer to obtain a predicted state of the sample multimedia resource under the classification task output by the first fully connected layer.

[0086] As an example, the first fully connected layer may be a multi-layer fc fully connected layer.

[0087] Figure 4 A flowchart of a method for obtaining a prediction classification task result of a sample multimedia resource according to an exemplary embodiment of the present disclosure is shown.

[0088] Reference Figure 4 In step S301, the characteristics of the multimedia resource state corresponding to the sample state label are obtained.

[0089] As an example, the feature of the multimedia resource state may be an embedding feature. It should be understood that it may also be other types of features, and the present disclosure does not limit this.

[0090] As an example, the embedded features at the position corresponding to the multimedia resource state in the second matrix can be used as the features of the multimedia resource state. Each row in the second matrix (O, D) corresponds to various multimedia resource states (O types of multimedia resource states), that is, the second matrix has O rows and D columns. For example, the j-th multimedia resource state corresponds to the j-th row of the second matrix, and accordingly, the D-dimensional Embedding features of the j-th row can be used as the features of the j-th multimedia resource state. As an example, the second matrix can be a dictionary matrix.

[0091] As an example, the second matrix is ​​randomly initialized at the beginning of training, and in the subsequent training process, the second matrix can be adjusted according to the loss function of the multi-task joint model. For example, the partial derivative can be calculated using the chain rule according to the loss function, and the second matrix can be updated in the direction of decreasing loss based on the partial derivative.

[0092] In step S302, the feature of the multimedia resource state and the feature of the sample multimedia resource are both input into a second concatenation layer or a second attention network to obtain a second joint feature.

[0093] As an example, the features of the multimedia resource state and the features of the sample multimedia resource can both be input into the second splicing layer, that is, the features of the multimedia resource state and the features of the sample multimedia resource can be directly spliced ​​through the second splicing layer to obtain a second joint feature of 2D dimension.

[0094] As another example, both the characteristics of the multimedia resource state and the characteristics of the sample multimedia resource can be input into the second attention network. Specifically, the characteristics of the multimedia resource state can be used as Q (query) in the attention mechanism, and the characteristics of the sample multimedia resource can be used as K (key) and V (value) in the attention mechanism. Input into the second attention network. As an example, the second attention network can be a multi-head attention network. It should be understood that through the second attention network, the characteristics of different multimedia resource states can activate different feature dimensions of multimedia resources. In other words, the characteristics of different multimedia resource states can enhance the various feature dimensions of multimedia resources to different degrees.

[0095] In step S303, the second joint feature is input into the second fully connected layer to obtain the prediction result output by the second fully connected layer.

[0096] As an example, the second fully connected layer may be a multi-layer fc fully connected layer.

[0097] Figure 5 A flowchart of a method for extracting content features of a sample video according to an exemplary embodiment of the present disclosure is shown.

[0098] Reference Figure 5 In step S401, M frames of images are acquired based on the sample video, where M is a positive integer.

[0099] As an example, N video frames (i.e., global images) may be sampled from a sample video, and target objects (e.g., human bodies) may be detected on the N global images to obtain local images including the target objects, and then the N global images and the obtained local images may be used as the M images. When the total number of frames of the N global images and the obtained local images is less than M, frames may be supplemented (e.g., cover frames may be supplemented) based on the N global images and the obtained local images to obtain a total of M frames.

[0100] It should be understood that Figure 7 As shown, one or more local images including a target object may be acquired for a global image. For example, each local image may include a target object, and different local images acquired for the same global image may include different target objects.

[0101] In step S402, the M frames of images are input into a feature extraction network to obtain a feature value of a D-dimensional feature of each frame of image, where D is a positive integer.

[0102] As an example, M frames of images may be input into an image feature extraction network to extract features of each frame of image and obtain M*D dimensional image features in total. As an example, the image feature extraction network may include but is not limited to inception-v3 or resent-50d or EfficientNet-B3.

[0103] In step S403, based on the feature values ​​of the D-dimensional features of the M image frames, features for describing the content of the sample video from a temporal dimension and a spatial dimension are obtained.

[0104] As an example, the feature values ​​of the D-dimensional features of the M frame images can be input into the cross-modal feature interaction model to learn the inter-frame correlation relationship of the M frame images to obtain an M*D-dimensional feature vector; and the maximum eigenvalue corresponding to each of the D feature dimensions in the M*D-dimensional feature vector (that is, the maximum eigenvalue in each column of the M*D-dimensional feature vector) is determined, and the vector composed of the D maximum eigenvalues ​​is used as the feature of the content of the sample video.

[0105] As an example, the cross-modal feature interaction model may be a transformer structure. For example, the M*D-dimensional feature vector formed by the eigenvalues ​​of the D-dimensional features of the M-frame images may be input into the transformer structure to learn the interaction relationship between the M-frame images, and the learned M*D-dimensional feature vector is obtained. Then, the learned M*D-dimensional feature vector is subjected to reduce_max processing to obtain the maximum eigenvalue corresponding to each dimension (a total of D dimensions). The maximum eigenvalues ​​corresponding to the D dimensions constitute the content features of the sample video. It should be understood that the cross-modal feature interaction model may also be other appropriate model structures for realizing feature interaction, and the present disclosure does not limit this.

[0106] According to an exemplary embodiment of the present disclosure, global features and local features of a sample video can be extracted, and the interaction of global features and local features can be realized to obtain features for comprehensively describing the content of the sample video from the time dimension and the space dimension.

[0107] Figure 6 A flowchart of a multi-task joint prediction method according to an exemplary embodiment of the present disclosure is shown. The multi-task joint prediction method is performed based on a multi-task joint model, and the multi-task joint model includes a feature extraction model, a conditional probability model, and a posterior probability model. As an example, the multi-task joint model can be trained using the training method described in the above exemplary embodiment.

[0108] Reference Figure 6In step S501, when a target multimedia resource to be predicted and another input quantity are received, the target multimedia resource is input into the feature extraction model for feature extraction to obtain features of the target multimedia resource.

[0109] As an example, the target multimedia resource is a target video, and M frames of images can be obtained based on the target video; the M frames of images are input into a feature extraction network to obtain feature values ​​of D-dimensional features of each frame of the image; then, based on the feature values ​​of the D-dimensional features of the M frames of images, a feature X for describing the content of the target video from the time dimension and the space dimension is obtained. Wherein, M is a positive integer, and D is a positive integer.

[0110] As an example, the feature values ​​of the D-dimensional features of the M frame images can be input into a cross-modal feature interaction model to learn the inter-frame correlation relationship of the M frame images to obtain an M*D-dimensional feature vector; then, the maximum eigenvalue corresponding to each of the D feature dimensions in the M*D-dimensional feature vector is determined, and the vector composed of the D maximum eigenvalues ​​is used as the feature of the content of the target video.

[0111] In step S502, when the other input quantity is a classification task identifier, the characteristics of the target multimedia resource and the classification task identifier are input into the conditional probability model for prediction to obtain a predicted state of the target multimedia resource.

[0112] In other words, when the other input quantity is a classification task identifier, it is necessary to predict the state of the target multimedia resource under the classification task corresponding to the classification task identifier.

[0113] As an example, the features of the classification task corresponding to the classification task identifier can be obtained; the features of the classification task and the features of the target multimedia resource are both input into the first splicing layer or the first attention network to obtain a first joint feature; then, the first joint feature is input into the first fully connected layer to obtain the predicted state of the target multimedia resource under the classification task output by the first fully connected layer.

[0114] As an example, the embedded features at the positions corresponding to the classification task identifiers in the first matrix obtained through training can be used as features of the classification task corresponding to the classification task identifier; wherein each row in the first matrix corresponds one-to-one to each classification task.

[0115] Combination Figure 7, when each classification task is a classification task for determining whether a multimedia resource violates a rule, if the classification task corresponding to the received classification task identifier is a classification task for country A (i.e., condition C), then in step S502, it is predicted whether the target multimedia resource in country A is in a violation state or a non-violation state. Therefore, as an example, it can be determined whether to publish the target multimedia resource on the video platform corresponding to country A according to the predicted state of the target multimedia resource.

[0116] In step S503, when the other input quantity is a multimedia resource state identifier, the characteristics of the target multimedia resource and the multimedia resource state identifier are input into the posterior probability model for prediction to obtain a prediction classification task result of the target multimedia resource.

[0117] In other words, when the other input quantity is a multimedia resource state identifier, it is necessary to predict under which classification task or tasks the target multimedia resource is classified to predict the multimedia resource state corresponding to the multimedia resource state identifier.

[0118] As an example, the characteristics of the multimedia resource state corresponding to the multimedia resource state identifier can be obtained; the characteristics of the multimedia resource state and the characteristics of the target multimedia resource are both input into a second splicing layer or a second attention network to obtain a second joint feature; then, the second joint feature is input into a second fully connected layer to obtain the predicted state of the target multimedia resource output by the second fully connected layer as a prediction classification task result of the multimedia resource state.

[0119] As an example, the embedded features at the positions corresponding to the multimedia resource state identifiers in the trained second matrix can be used as features of the multimedia resource state corresponding to the multimedia resource state identifier; wherein the various multimedia resource states in the second matrix correspond one to one.

[0120] As an example, each classification task may correspond to different distribution scenarios, for example, distribution scenarios corresponding to different countries or regions. As an example, various multimedia resource states may include: a violation state (ie, a distribution not allowed state) and a non-violation state (ie, a distribution allowed state).

[0121] As an example, the multi-task joint prediction method according to the exemplary embodiment of the present disclosure may further include: distributing the target multimedia resource according to the obtained prediction status of the target multimedia resource and the prediction classification task result of the target multimedia resource.

[0122] Combination Figure 7, if the multimedia resource state corresponding to the received multimedia resource state identifier is "violation state" (i.e., the posterior result R), then in step S503, it is predicted that under which classification task or tasks, the classification result of "violation state" will be obtained when predicting the target multimedia resource. For example, it is predicted that the classification result of "violation state" will be obtained when predicting the target multimedia resource under the classification task for country B, and the classification result of "violation state" will not be obtained when predicting the target multimedia resource under the classification task for country D, which also indicates that it is predicted that the target multimedia resource in country B belongs to "violation state" and the target multimedia resource in country D belongs to "non-violation state". Therefore, as an example, the target multimedia resource can be published on the video platform corresponding to country B according to the predicted classification task result.

[0123] According to the exemplary embodiments of the present disclosure, a new multi-task joint optimization technical solution implemented on the input side is proposed, which has better effect and scalability than the current multi-head joint optimization technical solution. Specifically as follows: a technical solution that can realize multi-task joint optimization on the input side is proposed; adaptive learning of personalized embeddings of different tasks to activate different multimedia resource features, and achieve conditional probability: P (multimedia resource feature X | condition C); adaptive learning of personalized embeddings of different multimedia resource states to activate different multimedia resource features, and achieve posterior probability: P (multimedia resource feature X | posterior result R); simple to launch and strong scalability.

[0124] Figure 8 A structural block diagram of a training device for a multi-task joint model according to an exemplary embodiment of the present disclosure is shown. The multi-task joint model for multiple classification tasks includes: a feature extraction model, a conditional probability model, and a posterior probability model.

[0125] like Figure 8 As shown, the training device 10 of the multi-task joint model according to the exemplary embodiment of the present disclosure includes: a training sample acquisition unit 101, a feature extraction unit 102, a first prediction unit 103, a second prediction unit 104, a loss function determination unit 105, and a training unit 106.

[0126] Specifically, the training sample acquisition unit 101 is configured to acquire multimedia training samples, wherein each of the multimedia training samples includes a sample multimedia resource, a sample task label and a sample status label; the sample task label is used to characterize the actual classification task corresponding to the sample multimedia resource, and the sample status label is used to characterize the actual status of the sample multimedia resource under the actual classification task.

[0127] The feature extraction unit 102 is configured to input the sample multimedia resource into the feature extraction model to perform feature extraction to obtain the features of the sample multimedia resource.

[0128] The first prediction unit 103 is configured to input the features of the sample multimedia resource and the sample task label into the conditional probability model for prediction, so as to obtain a predicted state of the sample multimedia resource.

[0129] The second prediction unit 104 is configured to input the features of the sample multimedia resource and the sample state label into the posterior probability model for prediction, and obtain a prediction classification task result of the sample multimedia resource.

[0130] The loss function determination unit 105 is configured to determine the loss function of the multi-task joint model based on the sample task label, the sample state label, the predicted classification task result and the predicted state of the sample multimedia resource.

[0131] The training unit 106 is configured to adjust model parameters of the multi-task joint model based on the loss function to train the multi-task joint model.

[0132] As an example, the first prediction unit 103 can be configured to: obtain the characteristics of the classification task corresponding to the sample task label; input the characteristics of the classification task and the characteristics of the sample multimedia resource into the first splicing layer or the first attention network to obtain a first joint feature; input the first joint feature into the first fully connected layer to obtain the predicted state of the sample multimedia resource under the classification task output by the first fully connected layer.

[0133] As an example, the first prediction unit 103 may be configured to use the embedded features at the position in the first matrix corresponding to the classification task as the features of the classification task; wherein each row in the first matrix corresponds one-to-one to each classification task; wherein the training unit 106 may be configured to adjust the first matrix according to the loss function.

[0134] As an example, the second prediction unit 104 can be configured to: obtain the characteristics of the multimedia resource state corresponding to the sample state label; input both the characteristics of the multimedia resource state and the characteristics of the sample multimedia resource into a second splicing layer or a second attention network to obtain a second joint feature; input the second joint feature into a second fully connected layer to obtain the predicted state of the sample multimedia resource output by the second fully connected layer as a prediction classification task result of the multimedia resource state.

[0135] As an example, the second prediction unit 104 may be configured to use the embedded features at the position in the second matrix corresponding to the multimedia resource state as the features of the multimedia resource state; wherein the rows in the second matrix correspond one-to-one to various multimedia resource states; wherein the training unit 106 may be configured to adjust the second matrix according to the loss function.

[0136] As an example, the sample multimedia resource is a sample video, wherein the feature extraction unit 102 can be configured to: obtain M frames of images based on the sample video, wherein M is a positive integer; input the M frames of images into a feature extraction network to obtain feature values ​​of D-dimensional features of each frame of the image, wherein D is a positive integer; based on the feature values ​​of the D-dimensional features of the M frames of images, obtain features for describing the content of the sample video from a temporal dimension and a spatial dimension.

[0137] As an example, the feature extraction unit 102 can be configured to: input the feature values ​​of the D-dimensional features of the M frame images into the cross-modal feature interaction model to learn the inter-frame correlation relationship of the M frame images to obtain an M*D-dimensional feature vector; determine the maximum eigenvalue corresponding to each of the D feature dimensions in the M*D-dimensional feature vector, and use the vector composed of the D maximum eigenvalues ​​as the feature of the content of the sample video.

[0138] As an example, the loss function determination unit 105 can be configured to: determine a first loss function based on the predicted state of the sample multimedia resource and the sample state label; determine a second loss function based on the predicted classification task result of the sample multimedia resource and the sample task label; perform weighted summation of the first loss function and the second loss function to obtain the loss function of the multi-task joint model; wherein the weighted weight of the first loss function is greater than the weighted weight of the second loss function.

[0139] Fig. 9 The structure block diagram of a multi-task joint prediction device according to an exemplary embodiment of the present disclosure is shown. The multi-task joint prediction device performs operations based on a multi-task joint model, which includes a feature extraction model, a conditional probability model, and a posterior probability model.

[0140] like Fig. 9 As shown, the multi-task joint prediction device 20 according to an exemplary embodiment of the present disclosure includes: a feature extraction unit 201, a first prediction unit 202, and a second prediction unit 203.

[0141] Specifically, when receiving a target multimedia resource to be predicted and another input quantity, the feature extraction unit 201 is configured to input the target multimedia resource into the feature extraction model for feature extraction to obtain the features of the target multimedia resource.

[0142] The first prediction unit 202 is configured to, when the other input amount is a classification task identifier, input the characteristics of the target multimedia resource and the classification task identifier into the conditional probability model for prediction to obtain a predicted state of the target multimedia resource.

[0143] The second prediction unit 203 is configured to input the characteristics of the target multimedia resource and the multimedia resource state identifier into the posterior probability model for prediction when the other input quantity is a multimedia resource state identifier, so as to obtain a prediction classification task result of the target multimedia resource.

[0144] As an example, the first prediction unit 202 can be configured to: obtain the characteristics of the classification task corresponding to the classification task identifier; input the characteristics of the classification task and the characteristics of the target multimedia resource into the first concatenation layer or the first attention network to obtain a first joint feature; input the first joint feature into the first fully connected layer to obtain the predicted state of the target multimedia resource under the classification task output by the first fully connected layer.

[0145] As an example, the first prediction unit 202 may be configured to use the embedded features at the positions corresponding to the classification tasks in the trained first matrix as features of the classification task; wherein each row in the first matrix corresponds one-to-one to each classification task.

[0146] As an example, the second prediction unit 203 can be configured to: obtain the characteristics of the multimedia resource state corresponding to the multimedia resource state identifier; input the characteristics of the multimedia resource state and the characteristics of the target multimedia resource into a second splicing layer or a second attention network to obtain a second joint feature; input the second joint feature into a second fully connected layer to obtain the predicted state of the target multimedia resource output by the second fully connected layer as a prediction classification task result of the multimedia resource state.

[0147] As an example, the second prediction unit 203 can be configured to use the embedded features at the positions corresponding to the multimedia resource states in the trained second matrix as the features of the multimedia resource states; wherein each row in the second matrix corresponds one-to-one to various multimedia resource states.

[0148] As an example, the target multimedia resource is a target video, wherein the feature extraction unit 201 can be configured to: obtain M frames of images based on the target video, wherein M is a positive integer; input the M frames of images into a feature extraction network to obtain feature values ​​of D-dimensional features of each frame of the image, wherein D is a positive integer; based on the feature values ​​of the D-dimensional features of the M frames of images, obtain features for describing the content of the target video from a temporal dimension and a spatial dimension.

[0149] As an example, the feature extraction unit 201 can be configured to: input the feature values ​​of the D-dimensional features of the M frame images into the cross-modal feature interaction model to learn the inter-frame correlation relationship of the M frame images to obtain an M*D-dimensional feature vector; determine the maximum eigenvalue corresponding to each of the D feature dimensions in the M*D-dimensional feature vector, and use the vector composed of the D maximum eigenvalues ​​as the feature of the content of the target video.

[0150] Regarding the multi-task joint model training device 10 and the multi-task joint prediction device 20 in the above-mentioned embodiments, the specific manner in which each unit performs operations has been described in detail in the embodiments of the relevant methods and will not be elaborated here.

[0151] In addition, it should be understood that the various units in the multi-task joint model training device 10 and the multi-task joint prediction device 20 according to the exemplary embodiment of the present disclosure can be implemented as hardware components and / or software components. Those skilled in the art can implement the various units, for example, using a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC) according to the processing performed by the defined various units.

[0152] Fig.10 FIG. 1 is a block diagram showing a structure of an electronic device according to an exemplary embodiment of the present disclosure. Fig.10 The electronic device 30 includes: at least one memory 301 and at least one processor 302, wherein the at least one memory 301 stores a set of computer executable instructions, and when the computer executable instruction set is executed by the at least one processor 302, the training method of the multi-task joint model as described in the above exemplary embodiment and / or the multi-task joint prediction method as described in the above exemplary embodiment is executed.

[0153] As an example, the electronic device 30 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device 30 is not necessarily a single electronic device, but may also be any device or circuit collection capable of executing the above-mentioned instructions (or instruction sets) individually or in combination. The electronic device 30 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0154] In the electronic device 30, the processor 302 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor 302 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0155] The processor 302 may execute instructions or codes stored in the memory 301, wherein the memory 301 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0156] The memory 301 may be integrated with the processor 302, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 301 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 301 and the processor 302 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 302 can read files stored in the memory.

[0157] In addition, the electronic device 30 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 30 may be connected to each other via a bus and / or a network.

[0158] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, the at least one processor is prompted to execute the training method of the multi-task joint model as described in the above exemplary embodiment and / or the multi-task joint prediction method as described in the above exemplary embodiment. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0159] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, the instructions in which may be executed by at least one processor to complete the training method of the multi-task joint model as described in the above exemplary embodiment and / or the multi-task joint prediction method as described in the above exemplary embodiment.

[0160] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0161] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A training method for a multi-task joint model, characterized in that: The multi-task joint model includes a feature extraction model, a conditional probability model and a posterior probability model, and the training method includes: Acquire multimedia training samples, wherein each of the multimedia training samples includes a sample multimedia resource, a sample task label, and a sample state label; the sample task label is used to characterize the actual classification task corresponding to the sample multimedia resource, and the sample state label is used to characterize the actual state of the sample multimedia resource under the actual classification task; Inputting the sample multimedia resource into the feature extraction model to perform feature extraction to obtain features of the sample multimedia resource; Inputting the characteristics of the sample multimedia resource and the sample task label into the conditional probability model for prediction to obtain a predicted state of the sample multimedia resource; Inputting the features of the sample multimedia resource and the sample state label into the posterior probability model for prediction, and obtaining the prediction classification task result of the sample multimedia resource; Determining a loss function of the multi-task joint model based on the sample task label, the sample state label, the predicted classification task result and the predicted state of the sample multimedia resource; Adjusting model parameters of the multi-task joint model based on the loss function to train the multi-task joint model; Wherein, the step of determining the loss function of the multi-task joint model based on the sample task label, sample state label, predicted classification task result and predicted state of the sample multimedia resource includes: Determining a first loss function based on the predicted state of the sample multimedia resource and the sample state label; Determining a second loss function based on the prediction classification task result of the sample multimedia resource and the sample task label; The first loss function and the second loss function are weightedly summed to obtain a loss function of the multi-task joint model.

2. The training method according to claim 1, characterized in that: The step of inputting the characteristics of the sample multimedia resource and the sample task label into the conditional probability model for prediction to obtain the predicted state of the sample multimedia resource comprises: Obtaining the characteristics of the classification task corresponding to the sample task label; Inputting the features of the classification task and the features of the sample multimedia resource into a first concatenation layer or a first attention network to obtain a first joint feature; The first joint feature is input into a first fully connected layer to obtain a predicted state of the sample multimedia resource under the classification task output by the first fully connected layer.

3. The training method according to claim 2, characterized in that: The step of obtaining the features of the classification task corresponding to the sample task label includes: Using the embedded features at the positions corresponding to the classification task in the first matrix as the features of the classification task; Wherein, each row in the first matrix corresponds to each classification task one by one; Wherein, in the step of adjusting the model parameters of the multi-task joint model based on the loss function, the first matrix is ​​adjusted according to the loss function.

4. The training method according to claim 1, characterized in that: The step of inputting the features of the sample multimedia resource and the sample state label into the posterior probability model for prediction to obtain the prediction classification task result of the sample multimedia resource comprises: Acquire the characteristics of the multimedia resource state corresponding to the sample state label; Inputting the feature of the multimedia resource state and the feature of the sample multimedia resource into a second concatenation layer or a second attention network to obtain a second joint feature; The second joint feature is input into a second fully connected layer, and the predicted state of the sample multimedia resource output by the second fully connected layer is obtained as a prediction classification task result of the multimedia resource state.

5. The training method according to claim 4, characterized in that: The step of obtaining the feature of the multimedia resource state corresponding to the sample state label comprises: Using the embedded features at the positions corresponding to the multimedia resource states in the second matrix as the features of the multimedia resource states; Wherein, each row in the second matrix corresponds to various multimedia resource states one by one; Wherein, in the step of adjusting the model parameters of the multi-task joint model based on the loss function, the second matrix is ​​adjusted according to the loss function.

6. The training method according to claim 1, characterized in that: The sample multimedia resource is a sample video, wherein the step of inputting the sample multimedia resource into the feature extraction model to extract features and obtain the features of the sample multimedia resource comprises: Acquire M frames of images based on the sample video, where M is a positive integer; Input the M frames of images into a feature extraction network to obtain a feature value of a D-dimensional feature of each frame of image, where D is a positive integer; Based on the feature values ​​of the D-dimensional features of the M frames of images, features for describing the content of the sample video from a temporal dimension and a spatial dimension are obtained.

7. The training method according to claim 6, characterized in that: The step of obtaining the feature for describing the content of the sample video from the time dimension and the space dimension based on the feature value of the D-dimensional feature of the M frames of image comprises: Inputting the feature values ​​of the D-dimensional features of the M-frame images into a cross-modal feature interaction model to learn the inter-frame correlation relationship of the M-frame images to obtain an M*D-dimensional feature vector; The maximum eigenvalue corresponding to each of the D feature dimensions in the M*D dimensional feature vector is determined, and a vector consisting of the D maximum eigenvalues ​​is used as a feature of the content of the sample video.

8. The training method according to claim 1, characterized in that: The weighted weight of the first loss function is greater than the weighted weight of the second loss function.

9. A multi-task joint prediction method, characterized in that: The multi-task joint prediction method is performed based on a multi-task joint model, the multi-task joint model includes a feature extraction model, a conditional probability model and a posterior probability model, and the multi-task joint prediction method includes: When receiving a target multimedia resource to be predicted and another input quantity, inputting the target multimedia resource into the feature extraction model to extract features, thereby obtaining features of the target multimedia resource; When the other input is a classification task identifier, inputting the characteristics of the target multimedia resource and the classification task identifier into the conditional probability model for prediction to obtain a predicted state of the target multimedia resource; When the other input is a multimedia resource state identifier, inputting the characteristics of the target multimedia resource and the multimedia resource state identifier into the posterior probability model for prediction to obtain a prediction classification task result of the target multimedia resource; The multi-task joint model is trained using the training method described in any one of claims 1 to 8.

10. The multi-task joint prediction method according to claim 9, characterized in that: The step of inputting the characteristics of the target multimedia resource and the classification task identifier into the conditional probability model for prediction to obtain the predicted state of the target multimedia resource comprises: Acquire the characteristics of the classification task corresponding to the classification task identifier; Inputting the features of the classification task and the features of the target multimedia resource into a first concatenation layer or a first attention network to obtain a first joint feature; The first joint feature is input into a first fully connected layer to obtain a predicted state of the target multimedia resource under the classification task output by the first fully connected layer.

11. The multi-task joint prediction method according to claim 10, characterized in that: The step of obtaining the feature of the classification task corresponding to the classification task identifier comprises: Using the embedded features at the positions corresponding to the classification task in the first matrix obtained through training as the features of the classification task; Wherein, each row in the first matrix corresponds to each classification task one by one.

12. The multi-task joint prediction method according to claim 9, characterized in that: The step of inputting the characteristics of the target multimedia resource and the multimedia resource state identifier into the posterior probability model for prediction to obtain the prediction classification task result of the target multimedia resource comprises: Acquire the characteristics of the multimedia resource state corresponding to the multimedia resource state identifier; Inputting the feature of the multimedia resource state and the feature of the target multimedia resource into a second concatenation layer or a second attention network to obtain a second joint feature; The second joint feature is input into a second fully connected layer, and the predicted state of the target multimedia resource output by the second fully connected layer is obtained as a prediction classification task result of the multimedia resource state.

13. The multi-task joint prediction method according to claim 12, characterized in that: The step of obtaining the feature of the multimedia resource state corresponding to the multimedia resource state identifier comprises: Using the embedded features at the positions corresponding to the multimedia resource states in the trained second matrix as the features of the multimedia resource states; Each row in the second matrix corresponds to each multimedia resource status one by one.

14. The multi-task joint prediction method according to claim 9, characterized in that: The target multimedia resource is a target video, wherein the step of inputting the target multimedia resource into the feature extraction model for feature extraction to obtain the features of the target multimedia resource comprises: Acquire M frames of images based on the target video, where M is a positive integer; Input the M frames of images into a feature extraction network to obtain a feature value of a D-dimensional feature of each frame of image, where D is a positive integer; Based on the feature values ​​of the D-dimensional features of the M frames of images, features for describing the content of the target video from a temporal dimension and a spatial dimension are obtained.

15. The multi-task joint prediction method according to claim 14, characterized in that: The step of obtaining the feature for describing the content of the target video from the time dimension and the space dimension based on the feature value of the D-dimensional feature of the M frames of images comprises: Inputting the feature values ​​of the D-dimensional features of the M-frame images into a cross-modal feature interaction model to learn the inter-frame correlation relationship of the M-frame images to obtain an M*D-dimensional feature vector; The maximum eigenvalue corresponding to each of the D feature dimensions in the M*D dimensional feature vector is determined, and a vector consisting of the D maximum eigenvalues ​​is used as a feature of the content of the target video.

16. A training device for a multi-task joint model, characterized in that: The multi-task joint model includes a feature extraction model, a conditional probability model and a posterior probability model, and the training device includes: The training sample acquisition unit is configured to acquire multimedia training samples, wherein each of the multimedia training samples includes a sample multimedia resource, a sample task label and a sample state label; the sample task label is used to characterize the actual classification task corresponding to the sample multimedia resource, and the sample state label is used to characterize the actual state of the sample multimedia resource under the actual classification task; A feature extraction unit is configured to input the sample multimedia resource into the feature extraction model to perform feature extraction, and obtain features of the sample multimedia resource; A first prediction unit is configured to input the characteristics of the sample multimedia resource and the sample task label into the conditional probability model for prediction to obtain a predicted state of the sample multimedia resource; A second prediction unit is configured to input the features of the sample multimedia resource and the sample state label into the posterior probability model for prediction, and obtain a prediction classification task result of the sample multimedia resource; A loss function determination unit configured to determine a loss function of the multi-task joint model based on a sample task label, a sample state label, a predicted classification task result, and a predicted state of the sample multimedia resource; A training unit, configured to adjust model parameters of the multi-task joint model based on the loss function to train the multi-task joint model; Among them, the loss function determination unit is configured to: determine a first loss function based on the predicted state of the sample multimedia resource and the sample state label; determine a second loss function based on the predicted classification task result of the sample multimedia resource and the sample task label; and perform weighted summation of the first loss function and the second loss function to obtain the loss function of the multi-task joint model.

17. The training device according to claim 16, characterized in that The first prediction unit is configured to: obtain the characteristics of the classification task corresponding to the sample task label; input the characteristics of the classification task and the characteristics of the sample multimedia resource into the first splicing layer or the first attention network to obtain a first joint feature; input the first joint feature into the first fully connected layer to obtain the predicted state of the sample multimedia resource under the classification task output by the first fully connected layer.

18. The training device according to claim 17, characterized in that The first prediction unit is configured to use the embedded features at the positions corresponding to the classification tasks in the first matrix as the features of the classification tasks; Wherein, each row in the first matrix corresponds to each classification task one by one; The training unit is configured to adjust the first matrix according to the loss function.

19. The training device according to claim 16, characterized in that The second prediction unit is configured to: obtain the characteristics of the multimedia resource state corresponding to the sample state label; input the characteristics of the multimedia resource state and the characteristics of the sample multimedia resource into the second splicing layer or the second attention network to obtain a second joint feature; input the second joint feature into the second fully connected layer to obtain the predicted state of the sample multimedia resource output by the second fully connected layer as the prediction classification task result of the multimedia resource state.

20. The training device according to claim 19, characterized in that The second prediction unit is configured to use the embedded features at the positions corresponding to the multimedia resource states in the second matrix as the features of the multimedia resource states; Wherein, each row in the second matrix corresponds to each multimedia resource status one by one; The training unit is configured to adjust the second matrix according to the loss function.

21. The training device according to claim 16, characterized in that The sample multimedia resource is a sample video, wherein the feature extraction unit is configured to: obtain M frames of images based on the sample video, wherein M is a positive integer; input the M frames of images into a feature extraction network to obtain feature values ​​of D-dimensional features of each frame of the image, wherein D is a positive integer; and obtain features for describing the content of the sample video from a time dimension and a space dimension based on the feature values ​​of the D-dimensional features of the M frames of images.

22. The training device according to claim 21, characterized in that The feature extraction unit is configured to: input the feature values ​​of the D-dimensional features of the M frame images into a cross-modal feature interaction model to learn the inter-frame correlation relationship of the M frame images to obtain an M*D-dimensional feature vector; determine the maximum eigenvalue corresponding to each of the D feature dimensions in the M*D-dimensional feature vector, and use the vector composed of the D maximum eigenvalues ​​as the feature of the content of the sample video.

23. The training device according to claim 16, characterized in that The weighted weight of the first loss function is greater than the weighted weight of the second loss function.

24. A multi-task joint prediction device, characterized in that: The multi-task joint prediction device performs operations based on a multi-task joint model, the multi-task joint model includes a feature extraction model, a conditional probability model and a posterior probability model, and the multi-task joint prediction device includes: A feature extraction unit is configured to, when receiving a target multimedia resource to be predicted and another input quantity, input the target multimedia resource into the feature extraction model to perform feature extraction, and obtain features of the target multimedia resource; A first prediction unit is configured to, when the other input is a classification task identifier, input the characteristics of the target multimedia resource and the classification task identifier into the conditional probability model for prediction to obtain a predicted state of the target multimedia resource; A second prediction unit is configured to, when the other input quantity is a multimedia resource state identifier, input the characteristics of the target multimedia resource and the multimedia resource state identifier into the posterior probability model for prediction to obtain a prediction classification task result of the target multimedia resource; The multi-task joint model is trained using the training method described in any one of claims 1 to 8.

25. The multi-task joint prediction device according to claim 24, characterized in that: The first prediction unit is configured to: obtain the characteristics of the classification task corresponding to the classification task identifier; input the characteristics of the classification task and the characteristics of the target multimedia resource into the first splicing layer or the first attention network to obtain a first joint feature; input the first joint feature into the first fully connected layer to obtain the predicted state of the target multimedia resource under the classification task output by the first fully connected layer.

26. The multi-task joint prediction device according to claim 25, characterized in that: The first prediction unit is configured to use the embedded features at the positions corresponding to the classification task in the trained first matrix as the features of the classification task; Wherein, each row in the first matrix corresponds to each classification task one by one.

27. The multi-task joint prediction device according to claim 24, characterized in that: The second prediction unit is configured to: obtain the characteristics of the multimedia resource state corresponding to the multimedia resource state identifier; input the characteristics of the multimedia resource state and the characteristics of the target multimedia resource into the second splicing layer or the second attention network to obtain a second joint feature; input the second joint feature into the second fully connected layer to obtain the predicted state of the target multimedia resource output by the second fully connected layer as the prediction classification task result of the multimedia resource state.

28. The multi-task joint prediction device according to claim 27, characterized in that: The second prediction unit is configured to use the embedded features at the positions corresponding to the multimedia resource states in the trained second matrix as the features of the multimedia resource states; Each row in the second matrix corresponds to each multimedia resource status one by one.

29. The multi-task joint prediction device according to claim 24, characterized in that: The target multimedia resource is a target video, wherein the feature extraction unit is configured to: obtain M frames of images based on the target video, wherein M is a positive integer; input the M frames of images into a feature extraction network to obtain feature values ​​of D-dimensional features of each frame of the image, wherein D is a positive integer; and obtain features for describing the content of the target video from a time dimension and a space dimension based on the feature values ​​of the D-dimensional features of the M frames of images.

30. The multi-task joint prediction device according to claim 29, characterized in that: The feature extraction unit is configured to: input the feature values ​​of the D-dimensional features of the M frame images into a cross-modal feature interaction model to learn the inter-frame correlation relationship of the M frame images to obtain an M*D-dimensional feature vector; determine the maximum eigenvalue corresponding to each of the D feature dimensions in the M*D-dimensional feature vector, and use the vector composed of the D maximum eigenvalues ​​as the feature of the content of the target video.

31. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the training method of the multi-task joint model as described in any one of claims 1 to 8 and / or the multi-task joint prediction method as described in any one of claims 9 to 15.

32. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the training method of the multi-task joint model as described in any one of claims 1 to 8 and / or the multi-task joint prediction method as described in any one of claims 9 to 15.

33. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by at least one processor, the training method of the multi-task joint model as described in any one of claims 1 to 8 and / or the multi-task joint prediction method as described in any one of claims 9 to 15 are implemented.

Images