Method for training multimedia feature representation model and method for generating feature representation

Abstract

The present disclosure relates to a multimedia feature representation model training method, a multimedia feature representation generation method, an apparatus, an electronic device, and a computer readable storage medium. The model training method obtains multimedia sample data, extracts features of the multimedia sample data using an initial feature extraction network to obtain corresponding multimedia sample feature representations, and obtains an account similarity between a first account and a second account. Then, the initial feature extraction network is trained according to the account similarity, the first multimedia sample feature representation, and the second multimedia sample feature representation, thereby obtaining a multimedia feature representation model. Since the model is trained using sample data of different users with the same behavior and the similarity between users, the model can learn the content features and user behavior features of the sample data, and the dimension of the feature representation output by the trained model is richer and more accurate.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of Internet, and in particular, to a multimedia feature representation model training method, a multimedia feature representation generation method, an apparatus, an electronic device, a computer readable storage medium, and a computer program product. BACKGROUND

[0002] With the development of Internet technology, various push services have been popularized. For the push of multimedia, the feature representation of multimedia is crucial. For example, for the push of short videos, the feature representation of short videos can be used for clustering, and the clustering result can be directly used in a recommendation system for diversity recommendation; in addition, the feature representation of short videos can also be directly input as a feature into the recommendation system for similarity search, so as to obtain more similar candidate videos for recommendation.

[0003] In related technologies, the feature representation of multimedia is generally generated based on content. For example, it is generated based on the classification category to which the multimedia content belongs, or based on the topic label (hashtag) of the multimedia content, so that the multimedia of the same category has similar feature representation, and the multimedia of different categories has dissimilar feature representation.

[0004] However, in related technologies, the feature representation of multimedia is only generated by content, which has a single dimension, resulting in low recommendation accuracy when used for recommendation. SUMMARY

[0005] The present disclosure provides a multimedia feature representation generation method, a model training method, an apparatus, an electronic device, a computer readable storage medium, and a computer program product to at least solve the problem of single dimension of multimedia feature representation generated by content only in related technologies. The technical solutions of the present disclosure are as follows:

[0006] According to a first aspect of an embodiment of the present disclosure, a multimedia feature representation model training method is provided, and the method comprises:

[0007] Obtaining multimedia sample data, the multimedia sample data comprising a plurality of first multimedia samples of a first account under a same behavior, and a plurality of second multimedia samples of a second account under the same behavior;

[0008] Performing feature extraction on the first multimedia samples by using an initial feature extraction network to obtain first multimedia sample feature representations;

[0009] Performing feature extraction on the second multimedia samples by using the initial feature extraction network to obtain second multimedia sample feature representations;

[0010] obtaining an account similarity between the first account and the second account;

[0011] training the initial feature extraction network according to the account similarity, the first multimedia sample feature representation and the second multimedia sample feature representation to obtain a multimedia feature representation model.

[0012] In one of the embodiments, the training of the initial feature extraction network according to the account similarity, the first multimedia sample feature representation and the second multimedia sample feature representation to obtain a multimedia feature representation model comprises: determining a loss function for model training according to the account similarity, the first multimedia sample feature representation and the second multimedia sample feature representation; adjusting network parameters of the initial feature extraction network according to the loss function; and determining the adjusted initial feature extraction network as the multimedia feature representation model when the adjusted initial feature extraction network meets a convergence condition.

[0013] In one of the embodiments, the determining of the loss function for model training according to the account similarity, the first multimedia sample feature representation and the second multimedia sample feature representation comprises: when the account similarity is greater than or equal to a set similarity threshold, determining a first loss function according to the first multimedia sample feature representation and the second multimedia sample feature representation, and determining the first loss function as the loss function for model training; or when the account similarity is less than the set similarity threshold, determining a second loss function according to the first loss function, the first multimedia sample feature representation and the second multimedia sample feature representation, and determining the second loss function as the loss function for model training.

[0014] In one of the embodiments, the first multimedia sample and the second multimedia sample each have a corresponding content label; and the determining of the first loss function according to the first multimedia sample feature representation and the second multimedia sample feature representation comprises: obtaining a first predicted content by using a content generation network according to the first multimedia sample feature representation of the first multimedia sample; obtaining a second predicted content by using the content generation network according to the second multimedia sample feature representation of the second multimedia sample; obtaining a first cross-entropy loss between the content label and the first predicted content of each of the first multimedia samples according to the content label and the first predicted content of each of the first multimedia samples; obtaining a second cross-entropy loss between the content label and the second predicted content of each of the second multimedia samples according to the content label and the second predicted content of each of the second multimedia samples; and determining the first loss function according to the first cross-entropy loss and the second cross-entropy loss.

[0015] In one of the embodiments, the determining the second loss function according to the first loss function, the first multimedia sample feature representation and the second multimedia sample feature representation comprises: performing feature decoding processing according to the first multimedia sample feature representation of each of the first multimedia samples to obtain a first decoded feature representation of each of the first multimedia samples; performing feature decoding processing according to the second multimedia sample feature representation of each of the second multimedia samples to obtain a second decoded feature representation of each of the second multimedia samples; obtaining a first mean square error according to the sample quantity of the first multimedia samples and the second multimedia samples, the first multimedia sample feature representation, the first decoded feature representation, the second multimedia sample feature representation, the second decoded feature representation and a preset first target value; obtaining a second mean square error according to the sample quantity, the first multimedia sample feature representation, the first decoded feature representation, the second multimedia sample feature representation, the second decoded feature representation and a preset second target value; and determining the second loss function according to the first mean square error, the second mean square error and the first loss function.

[0016] In one of the embodiments, the obtaining the first mean square error according to the sample quantity of the first multimedia samples and the second multimedia samples, the first multimedia sample feature representation, the first decoded feature representation, the second multimedia sample feature representation, the second decoded feature representation and a preset first target value comprises: determining a first similarity between the first multimedia sample feature representation and the first decoded feature representation of each of the first multimedia samples; determining a second similarity between the second multimedia sample feature representation and the second decoded feature representation of each of the second multimedia samples; obtaining a first sum of square differences between each of the first similarity and the second similarity and the first target value; and determining a quotient of the first sum of square differences and the sample quantity as the first mean square error.

[0017] In one of the embodiments, the obtaining the second mean square error according to the sample quantity, the first multimedia sample feature representation, the first decoded feature representation, the second multimedia sample feature representation, the second decoded feature representation, and the preset second target value comprises: determining a third similarity between the second decoded feature representation of each of the second multimedia samples and the first multimedia sample feature of each of the first multimedia samples; determining a fourth similarity between the second multimedia sample feature representation of each of the second multimedia samples and the first decoded feature representation of each of the first multimedia samples; obtaining a second sum of squared differences between each of the third similarity and the fourth similarity and the second target value; and determining a quotient of the second sum of squared differences and the sample quantity as the second mean square error.

[0018] In one of the embodiments, the determining the second loss function according to the first mean square error, the second mean square error, and the first loss function comprises: obtaining a sum of the first mean square error, the second mean square error, and the first loss function; and determining the sum as the second loss function.

[0019] In one of the embodiments, the obtaining the account similarity between the first account and the second account comprises: determining an account feature of the first account according to the first multimedia sample feature representation of the plurality of first multimedia samples of the first account; determining an account feature of the second account according to the second multimedia sample feature representation of the plurality of second multimedia samples of the second account; and calculating the account similarity between the first account and the second account according to the account feature of the first account and the account feature of the second account.

[0020] According to a second aspect of the embodiments of the present disclosure, a method for generating a multimedia feature representation is provided, and the method comprises:

[0021] obtaining multimedia data;

[0022] inputting the multimedia data into the multimedia feature representation model obtained by the method according to the first aspect to obtain a multimedia feature representation of the multimedia data, wherein the multimedia feature representation at least comprises an encoded feature having a content dimension and a user behavior dimension.

[0023] According to a third aspect of the embodiments of the present disclosure, a training device for a multimedia feature representation model is provided, and the device comprises:

[0024] a sample data obtaining module configured to perform obtaining multimedia sample data, wherein the multimedia sample data comprises a plurality of first multimedia samples of a first account under a same behavior and a plurality of second multimedia samples of a second account under the same behavior;

[0025] a feature extraction module configured to perform feature extraction on the first multimedia sample using an initial feature extraction network to obtain a first multimedia sample feature representation, and perform feature extraction on the second multimedia sample using the initial feature extraction network to obtain a second multimedia sample feature representation;

[0026] a similarity obtaining module configured to perform obtaining an account similarity between the first account and the second account;

[0027] a model training module configured to perform training the initial feature extraction network according to the account similarity, the first multimedia sample feature representation and the second multimedia sample feature representation to obtain a multimedia feature representation model.

[0028] In one of the embodiments, the model training module is configured to perform determining a loss function for model training according to the account similarity, the first multimedia sample feature representation and the second multimedia sample feature representation, adjusting network parameters of the initial feature extraction network according to the loss function, and determining the adjusted initial feature extraction network as the multimedia feature representation model when the adjusted initial feature extraction network satisfies a convergence condition.

[0029] In one of the embodiments, the model training module is further configured to perform, when the account similarity is greater than or equal to a set similarity threshold, determining a first loss function according to the first multimedia sample feature representation and the second multimedia sample feature representation, and determining the first loss function as the loss function for model training, or when the account similarity is less than the set similarity threshold, determining a second loss function according to the first loss function, the first multimedia sample feature representation and the second multimedia sample feature representation, and determining the second loss function as the loss function for model training.

[0030] In one of the embodiments, the first multimedia sample and the second multimedia sample each has a corresponding content label; the model training module is further configured to perform: obtaining, according to the first multimedia sample feature representation of the first multimedia sample, a first predicted content by using a content generation network; obtaining, according to the second multimedia sample feature representation of the second multimedia sample, a second predicted content by using the content generation network; obtaining, according to the content label and the first predicted content of each of the first multimedia sample, a first cross-entropy loss between the content label and the first predicted content of each of the first multimedia sample; obtaining, according to the content label and the second predicted content of each of the second multimedia sample, a second cross-entropy loss between the content label and the second predicted content of each of the second multimedia sample; and determining the first loss function according to the first cross-entropy loss and the second cross-entropy loss.

[0031] In one of the embodiments, the model training module is further configured to perform: obtaining, according to the first multimedia sample feature representation of each of the first multimedia sample, a first decoded feature representation of each of the first multimedia sample by performing feature decoding processing; obtaining, according to the second multimedia sample feature representation of each of the second multimedia sample, a second decoded feature representation of each of the second multimedia sample by performing feature decoding processing; obtaining a first mean square error according to the sample quantity of the first multimedia sample and the second multimedia sample, the first multimedia sample feature representation, the first decoded feature representation, the second multimedia sample feature representation, the second decoded feature representation, and a preset first target value; obtaining a second mean square error according to the sample quantity, the first multimedia sample feature representation, the first decoded feature representation, the second multimedia sample feature representation, the second decoded feature representation, and a preset second target value; and determining a second loss function according to the first mean square error, the second mean square error, and the first loss function.

[0032] In one of the embodiments, the model training module is further configured to perform: determining a first similarity between the first multimedia sample feature representation and the first decoded feature representation of each of the first multimedia sample; determining a second similarity between the second multimedia sample feature representation and the second decoded feature representation of each of the second multimedia sample; obtaining a first sum of squared differences between each of the first similarity and the second similarity and the first target value; and determining the first mean square error as a quotient of the first sum of squared differences and the sample quantity.

[0033] In one of the embodiments, the model training module is further configured to perform: determining a third similarity between the second decoded feature representation of each of the second multimedia samples and the first multimedia sample feature of each of the first multimedia samples; determining a fourth similarity between the second multimedia sample feature representation of each of the second multimedia samples and the first decoded feature representation of each of the first multimedia samples; obtaining a second sum of squared differences between each of the third similarity and the fourth similarity and the second target value, respectively; and determining a quotient of the second sum of squared differences and the sample quantity as a second mean square error.

[0034] In one of the embodiments, the model training module is further configured to perform: obtaining a sum of the first mean square error, the second mean square error and the first loss function, and determining the sum as a second loss function.

[0035] In one of the embodiments, the similarity obtaining module is further configured to perform: determining an account feature of the first account according to the first multimedia sample feature representation of the plurality of first multimedia samples of the first account; determining an account feature of the second account according to the second multimedia sample feature representation of the plurality of second multimedia samples of the second account; and calculating an account similarity between the first account and the second account according to the account feature of the first account and the account feature of the second account.

[0036] According to a fourth aspect of embodiments of the present disclosure, a multimedia feature representation generation apparatus is provided, and the apparatus comprises:

[0037] a data obtaining module configured to perform obtaining multimedia data;

[0038] a feature generation module configured to perform inputting the multimedia data into a multimedia feature representation model obtained by the method according to the first aspect, to obtain a multimedia feature representation of the multimedia data, the multimedia feature representation comprising at least an encoded feature with content dimension and user behavior dimension.

[0039] According to a fifth aspect of embodiments of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the steps of the method according to any one of the first aspect or the second aspect.

[0040] According to a sixth aspect of embodiments of the present disclosure, a computer readable storage medium is provided, when instructions in the computer readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the steps of the method according to any one of the first aspect or the second aspect.

[0041] According to a seventh aspect of the embodiments of the present disclosure, a computer program product is provided, which comprises instructions executed by a processor of an electronic device, so that the electronic device can perform the steps of the method according to any one of the first aspect or the second aspect.

[0042] The embodiments of the present disclosure provide at least the following beneficial effects: by obtaining multimedia sample data, performing feature extraction on the multimedia sample data by using an initial feature extraction network to obtain corresponding multimedia sample feature representation, and obtaining the account similarity between the first account and the second account, and then training the initial feature extraction network according to the account similarity, the first multimedia sample feature representation and the second multimedia sample feature representation, a multimedia feature representation model is obtained. Since the model is trained by using sample data of different users under the same behavior and the similarity between users, the model can learn the content features and user behavior features of the sample data, and the dimension of the feature representation output by the trained model is richer and more accurate.

[0043] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0044] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure, and do not constitute an undue limitation on the present disclosure.

[0045] Figure 1 is a flowchart of a method for training a multimedia feature representation model according to an exemplary embodiment.

[0046] Figure 2 is a schematic diagram of the model training step according to an exemplary embodiment.

[0047] Figure 3 is a schematic diagram of the step of determining the first loss function according to an exemplary embodiment.

[0048] Figure 4 is a schematic diagram of the initial feature extraction network framework structure according to an exemplary embodiment.

[0049] Figure 5 is a schematic diagram of the step of determining the second loss function according to an exemplary embodiment.

[0050] Figure 6 is a schematic diagram of the step of determining the first mean square error according to an exemplary embodiment.

[0051] Figure 7 is a schematic diagram of a step of determining a second mean square error according to an example embodiment.

[0052] Figure 8 is a schematic diagram of a step of obtaining account similarity according to an example embodiment.

[0053] Figure 9 is a schematic diagram of a model training framework according to an example embodiment.

[0054] Figure 10 is a flow chart of a method of generating a multimedia feature representation according to an example embodiment.

[0055] Figure 11 is a block diagram of a training device of a multimedia feature representation model according to an example embodiment.

[0056] Figure 12 is a block diagram of a generating device of a multimedia feature representation according to an example embodiment.

[0057] Figure 13 is a block diagram of an electronic device according to an example embodiment. DETAILED DESCRIPTION

[0058] In order for those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings.

[0059] It should be noted that the terms "first", "second", and the like in the specification and claims of the present disclosure and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all the embodiments consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0060] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, analyzed data, etc.) involved in the present disclosure are all information and data authorized by the user or authorized by all parties.

[0061] In one embodiment, as shown in Figure 1 A method for training a multimedia feature representation model is provided, which is applied to a computer device as an example for illustration, and specifically includes the following steps:

[0062] In step S110, multimedia sample data is acquired.

[0063] The multimedia sample data is training data for model training. Specifically, the multimedia is a combination of multiple media, generally including text, sound, image and other media forms.

[0064] In this embodiment, the multimedia sample data includes multiple first multimedia samples of a first account under the same behavior and multiple second multimedia samples of a second account under the same behavior. The first account and the second account are respectively used to represent different user accounts. The behavior can refer to user behavior, such as a user's like behavior, a collection behavior, a long-playing behavior, etc., wherein the long-playing behavior refers to a behavior of playing a video for a certain length of time. Specifically, the multimedia sample data of this embodiment includes multimedia samples of different users under the same behavior. In this embodiment, taking a video sample as the multimedia sample and a long-playing behavior as the user behavior as an example, the multimedia sample data includes a first long-playing video list of the first account (i.e., multiple first multimedia samples under the long-playing behavior of the first account) and a second long-playing video list of the second account (i.e., multiple second multimedia samples under the long-playing behavior of the second account).

[0065] Specifically, before training the model, the computer device first needs to acquire corresponding training data. That is, to acquire multiple first multimedia samples of a first account under the same behavior and multiple second multimedia samples of a second account. Specifically, the training data can be pre-stored in the computer device, or can be stored in various types of memories accessible by the computer device, and the present embodiment does not limit this.

[0066] In step S120, the initial feature extraction network is used to perform feature extraction on the multimedia sample data to obtain multimedia sample feature representations.

[0067] The initial feature extraction network can be various deep learning models or machine learning models. For each multimedia sample, the initial feature extraction network can be used to perform feature extraction to obtain a multimedia sample feature representation of the corresponding content. Specifically, the feature representation can be represented as a feature vector of the multimedia sample content.

[0068] In this embodiment, the computer device can use the initial feature extraction network to perform feature extraction on the multiple first multimedia samples respectively, thereby obtaining corresponding multiple first multimedia sample feature representations. The computer device can also use the initial feature extraction network to perform feature extraction on the multiple second multimedia samples respectively, thereby obtaining corresponding multiple second multimedia sample feature representations.

[0069] In step S130, an account similarity between the first account and the second account is obtained.

[0070] The account similarity is used to represent a similarity between the accounts. Specifically, the computer device can obtain the account similarity between the first account and the second account based on the account features of the respective accounts, in a manner of Euclidean distance, cosine similarity, etc.

[0071] In step S140, the initial feature extraction network is trained according to the account similarity, the first multimedia sample feature representation, and the second multimedia sample feature representation, to obtain a multimedia feature representation model.

[0072] The trained multimedia feature representation model can be used to generate a feature representation of a multimedia, which includes at least encoded features with content dimensions and user behavior dimensions.

[0073] In this embodiment, the computer device trains the initial feature extraction network according to the obtained account similarity, the first multimedia sample feature representation, and the second multimedia sample feature representation, so that a trained multimedia feature representation model capable of outputting multi-dimensional features can be obtained.

[0074] In the training method of the multimedia feature representation model, the multimedia sample data is obtained, the initial feature extraction network is used to extract features of the multimedia sample data, to obtain corresponding multimedia sample feature representations, the account similarity between the first account and the second account is obtained, and then the initial feature extraction network is trained according to the account similarity, the first multimedia sample feature representation, and the second multimedia sample feature representation, to obtain the multimedia feature representation model. Since the same behavior sample data of different users and the similarity between users are used to train the model in this embodiment, the model can learn the content features and user behavior features of the sample data, so that the dimension of the feature representation output by the trained model is richer and more accurate.

[0075] In an exemplary embodiment, as shown in FIG. 2, in step S140, the initial feature extraction network is trained according to the account similarity, the first multimedia sample feature representation, and the second multimedia sample feature representation, to obtain the multimedia feature representation model, which can specifically include: Figure 2

[0076] In step S210, a loss function for model training is determined according to the account similarity, the first multimedia sample feature representation, and the second multimedia sample feature representation.

[0077] ​Since the account similarity means that the consumption content of two accounts under the same behavior is likely to be similar, and the account similarity means that the consumption content of two accounts under the same behavior is likely to be different. Based on this, in the embodiment, the computer device can determine the loss function for model training and optimization according to the size of the account similarity. Wherein, the loss function is used to represent the difference between the predicted data and the actual data, which can be used to measure the prediction of the initial feature extraction network. Specifically, the loss function can be determined by the first multimedia sample feature representation and the second multimedia sample feature representation.

[0078] In step S220, the network parameters of the initial feature extraction network are adjusted according to the loss function.

[0079] Since the loss function can be used to measure the prediction of the initial feature extraction network, the computer device can adjust the network parameters of the initial feature extraction network by the loss function determined above, so as to optimize the initial feature extraction network and make it have better prediction performance.

[0080] In step S230, when the adjusted initial feature extraction network meets the convergence condition, the adjusted initial feature extraction network is determined as the multimedia feature representation model.

[0081] Wherein, the convergence condition can be a pre-set trigger condition for stopping training. For example, it can be a set maximum iteration number, then when the iteration exceeds the maximum number, the training is stopped; it can also be a set weight value change threshold between two iterations, then when the weight value between two iterations is less than the threshold, the training is stopped; it can also be a pre-set smaller error threshold, then when the error between the predicted data and the actual data is less than the threshold, the training is stopped.

[0082] Specifically, when the adjusted initial feature extraction network meets the convergence condition, the adjusted initial feature extraction network can be determined as the multimedia feature representation model, that is, the trained model is obtained.

[0083] In the above embodiment, the loss function for model training is determined by the first multimedia sample feature representation and the second multimedia sample feature representation according to the size of the account similarity, and the network parameters of the initial feature extraction network are adjusted according to the loss function. When the adjusted initial feature extraction network meets the convergence condition, the adjusted initial feature extraction network is determined as the multimedia feature representation model. Since the loss function for model training in the embodiment is determined by the first multimedia sample feature representation and the second multimedia sample feature representation according to the size of the account similarity, the model can learn the content features and user behavior features of the sample data, so that the dimension of the feature representation output by the trained model is more abundant and the accuracy is higher.

[0084] In an example embodiment, in step S210, the loss function for model training is determined according to the account similarity, the first multimedia sample feature representation, and the second multimedia sample feature representation, which can specifically include: when the account similarity is greater than or equal to a set similarity threshold, the first loss function is determined according to the first multimedia sample feature representation and the second multimedia sample feature representation, and the first loss function is determined as the loss function for model training.

[0085] The similarity threshold can be a pre-set index for measuring the similarity between users. Specifically, when the account similarity is greater than or equal to the similarity threshold, it indicates that the first account and the second account have a high similarity. For example, the similarity threshold can be 0.5. The first loss function is the loss function for training the model when the account similarity is greater than or equal to the set similarity threshold. In this embodiment, when the computer device determines that the account similarity is greater than or equal to the set similarity threshold, the first loss function can be determined according to the first multimedia sample feature representation and the second multimedia sample feature representation, i.e., the loss function for model training is determined.

[0086] In one scenario, when the account similarity is less than the set similarity threshold, the second loss function is determined according to the first loss function, the first multimedia sample feature representation, and the second multimedia sample feature representation, and the second loss function is determined as the loss function for model training. The second loss function is the loss function for training the model when the account similarity is less than the set similarity threshold. Specifically, when the account similarity is less than the set similarity threshold, it indicates that the first account and the second account are not similar. In this embodiment, when the computer device determines that the account similarity is less than the set similarity threshold, the second loss function can be determined according to the first loss function, the first multimedia sample feature representation, and the second multimedia sample feature representation, i.e., the loss function for model training is determined.

[0087] Since the loss function for model training in this embodiment is determined according to different parameters according to the size of the account similarity, the model can not only learn the content features of the sample data, but also learn the user behavior features, thereby facilitating the improvement of the accuracy of the model output features.

[0088] In an example embodiment, the first multimedia sample and the second multimedia sample each have a corresponding content label. The content label can be a label representing the specific content of the first multimedia sample or a label representing the specific content of the second multimedia sample. For example, the content label can be a topic label (hashtag) of the first multimedia sample or a topic label of the second multimedia sample.

[0089] Then, as shown in Figure 3 determining a first loss function according to the first multimedia sample feature representation and the second multimedia sample feature representation, which can specifically include:

[0090] In step S310, a first predicted content is obtained by using the content generation network according to the first multimedia sample feature representation of the first multimedia sample.

[0091] The content generation network is a network used for content generation or content prediction. Specifically, the computer device can use the content generation network to perform prediction processing on the first multimedia sample feature representation of the first multimedia sample, so as to obtain the predicted content of the first multimedia sample, i.e., the first predicted content. For example, it can be the predicted topic label of the first multimedia sample.

[0092] In step S320, a second predicted content is obtained by using the content generation network according to the second multimedia sample feature representation of the second multimedia sample. Specifically, the computer device can also use the content generation network to perform prediction processing on the second multimedia sample feature representation of the second multimedia sample, so as to obtain the predicted content of the second multimedia sample, i.e., the second predicted content. For example, it can be the predicted topic label of the second multimedia sample.

[0093] In step S330, a first cross-entropy loss between the first predicted content and the content label of each first multimedia sample is obtained according to the content label and the first predicted content of each first multimedia sample.

[0094] The cross-entropy loss (Cross entropy loss function, abbreviated as CE) is used to represent the distance between two probability distributions. In this embodiment, the first cross-entropy loss is used to represent the loss of the first multimedia sample in the content understanding dimension.

[0095] Specifically, the computer device can obtain the first cross-entropy loss between the first predicted content and the content label of each first multimedia sample according to the content label and the first predicted content corresponding to each first multimedia sample. Since each topic label is composed of N characters, each character is a classification, and this embodiment can calculate the corresponding first cross-entropy loss based on the classification of each character in the topic label of the first multimedia sample.

[0096] For example, taking the content label as the topic label as an example, if the topic label corresponding to the first multimedia sample is P1 and the predicted topic label is T1, the first cross-entropy loss CE Loss1 of the first multimedia sample can be calculated based on the following formula, specifically CE Loss1 = CE(P1, T1).

[0097] In step S340, a second cross-entropy loss between the second predicted content and the content label of each second multimedia sample is obtained according to the content label and the second predicted content of each second multimedia sample.

[0098] Similarly, the second cross-entropy loss is used to represent the loss of the second multimedia sample in the content understanding dimension. Specifically, the computer device obtains the second cross-entropy loss between the second predicted content and the content label of each second multimedia sample according to the content label and the second predicted content corresponding to each second multimedia sample.

[0099] For example, taking the content label as a topic label, if the topic label corresponding to the second multimedia sample is P2 and the predicted topic label is T2, then the second cross-entropy loss of each second multimedia sample is CE Loss2 = CE(P2, T2).

[0100] In step S350, a first loss function is determined according to the first cross-entropy loss and the second cross-entropy loss.

[0101] Since the first loss function is a loss function used for training the model when the account similarity is greater than or equal to the set similarity threshold. The training data used for model training includes a plurality of first multimedia samples of the first account and a plurality of second multimedia samples of the second account under the same behavior, and therefore, by summing the first cross-entropy loss corresponding to the first multimedia sample and the second cross-entropy loss corresponding to the second multimedia sample, the corresponding first loss function can be obtained.

[0102] In the above embodiment, when the account similarity is greater than or equal to the set similarity threshold, the first cross-entropy loss between the first predicted content and the content label of each first multimedia sample is obtained according to the content label and the first predicted content corresponding to each first multimedia sample, and the second cross-entropy loss between the second predicted content and the content label of each second multimedia sample is obtained according to the content label and the second predicted content corresponding to each second multimedia sample, and then the first loss function is determined according to the first cross-entropy loss and the second cross-entropy loss. Thus, when the account similarity is high, the model is trained only by the loss in the content understanding dimension to improve the accuracy of the model output features.

[0103] In this embodiment, as shown in FIG. 6, a second loss function is determined according to the first loss function, the first multimedia sample feature representation, and the second multimedia sample feature representation, which can specifically include: Figure 4

[0104] ​In step S410, feature decoding processing is performed based on the first multimedia sample feature representation of each first multimedia sample to obtain the first decoded feature representation of each first multimedia sample.

[0105] In step S420, feature decoding processing is performed based on the feature representation of each second multimedia sample to obtain the second decoded feature representation of each second multimedia sample.

[0106] The decoding process can be a feature extraction process based on an initial feature extraction network. In this embodiment, the initial feature extraction network is used as an example. Figure 5 Taking the Encoder-Decoder framework structure shown as an example, this embodiment further illustrates the extraction process of the first multimedia sample feature representation, the first decoded feature representation, the second multimedia sample feature representation, and the second decoded feature representation. Here, the Encoder is the video encoding feature extraction network (i.e., the first multimedia sample feature representation and the second multimedia sample feature representation) used in this embodiment for illustration, and the Decoder is the network corresponding to the decoding feature extraction (i.e., the first decoded feature representation and the second decoded feature representation). In practical scenarios, other encoding feature extraction networks or decoding feature extraction networks can also be used, and this embodiment does not limit this.

[0107] Specifically, the Encoder's input consists of image and text features from the multimedia samples. Image features can be extracted using image feature extraction networks such as SWIN, or networks like ResNet (Residual Neural Network) and ViT (Vision Transformer). The input image feature extraction network extracts images from the multimedia samples (including the first multimedia sample sequence user1 for the first account and the second multimedia sample sequence user2 for the second account). For example, when the multimedia sample is a video, the input image could be the corresponding video cover image. Figure 5 As shown, after the input image is processed by SWIN (or other image feature extraction networks such as ResNet), corresponding image features can be obtained, such as multiple 512-dimensional (or other number of dimensions) image features.

[0108] The text features can be converted into word vectors by the embedding layer of a text feature extraction network (such as a T5 text converter, which is a short form of Transfer Text-to-Text Transformer) from discrete characters. Taking a video as an example, the input for text feature extraction can include video description, video title, video caption, recognition results obtained by performing OCR (Optical Character Recognition) on video frames, and recognition results obtained by performing ASR (Automatic Speech Recognition) on audio, etc. Specifically, the above five types of text of the video can be spliced and the first m characters can be input into the embedding layer of T5 to obtain m text features text tokens with a dimension of 512 (which can be other number of dimensions, but needs to be consistent with the dimension of the image feature).

[0109] Further, the image feature Image and the text feature text tokens of each video, i.e. multimedia sample, obtained above are jointly input into the Encoder, and after the Encoder, each video will generate a 512-dimensional feature, i.e. an encoding feature. If the first account has n multimedia samples, the encoding feature sequence (i.e. n*512 features) of the first account, i.e. the sequence of corresponding first multimedia sample feature representations, can be obtained after the Encoder. Similarly, for the second multimedia samples of the second account, the encoding feature sequence of the corresponding second account, i.e. the sequence of corresponding second multimedia sample feature representations, can also be obtained.

[0110] The input of the Decoder is the encoded feature sequence of each account output by the Encoder (i.e., n*512 features), and the output of the Decoder is also an n*512 feature sequence. The first feature in the sequence comes from the output of the Encoder, and the second feature to the last feature are all decoding results of the encoded features. For example, if the first account has 5 multimedia samples, the encoded feature sequence of the first account (i.e., 5*512 features) can be obtained after the Encoder, and the decoding feature sequence output by the Decoder is obtained after the encoded feature sequence is input into the Decoder, i.e., the sequence of the first decoding feature representation is obtained. The first element of the decoding feature sequence comes from the first element of the encoded feature sequence, the second element of the decoding feature sequence is the decoding result of the first element of the encoded feature sequence, the third element of the decoding feature sequence is the decoding result of the first and second elements of the encoded feature sequence, and so on. The n-th element of the decoding feature sequence is the decoding result of the first n-1 elements of the encoded feature sequence. Since the last decoded decoding feature fuses the features of the entire encoded sequence, and the sequence is related to the user, the last decoded decoding feature can also be used as the embedding vector output of the user. Similarly, the second decoding feature representation of the second multimedia sample can also be obtained.

[0111] In step S430, the first mean square error is obtained according to the sample quantity, the first multimedia sample feature representation, the first decoding feature representation, the second multimedia sample feature representation, the second decoding feature representation, and the preset first target value.

[0112] The sample quantity refers to the number of multimedia sample data, i.e., the number of the first multimedia sample feature representation and the second multimedia sample feature representation. The mean square error (MSE loss) is used to represent the mean value of the square sum of the corresponding point errors between the predicted data and the original data. Since the encoded features and the decoded features obtained based on the multimedia samples of the same user should have high similarity, the first mean square error in this embodiment is the mean value of the square sum of the errors between the similarity (i.e., the predicted value) between the encoded features and the decoded features of each multimedia sample of the user and the maximum similarity 1 (i.e., the target value). The first target value is the maximum similarity 1, which is also the target value for calculating the mean square error.

[0113] For example, the computer device can calculate the similarity between each first multimedia sample feature representation of the first user and the first decoded feature representation, and the similarity between each second multimedia sample feature representation of the second user and the second decoded feature representation, calculate the sum of squares of differences between each similarity and the maximum similarity 1 (i.e. the first target value), and divide the sum of squares of differences by the number of samples to obtain the first mean square error. The purpose is to narrow the distance of the same user behavior and make them similar.

[0114] In step S440, according to the number of samples, the first multimedia sample feature representation, the first decoded feature representation, the second multimedia sample feature and the second decoded feature representation, and the preset second target value, the second mean square error is obtained.

[0115] The second mean square error in the embodiment is the mean value of the sum of squares of differences between the similarity (i.e. the predicted value) between the encoded features of the multimedia samples of one user and the decoded features of the multimedia samples of another user and the second target value 0 (i.e. the target value), and the purpose is to make the features of different users not similar.

[0116] Similarly, the computer device can also calculate the similarity between each first multimedia sample feature representation of the first user and each second decoded feature representation of the second user, and the similarity between each second multimedia sample feature representation of the second user and each first decoded feature representation of the first user, calculate the sum of squares of differences between each similarity and the second target value 0 respectively, and divide the sum of squares of differences by the number of samples to obtain the second mean square error.

[0117] In step S450, the second loss function is determined according to the first mean square error, the second mean square error and the first loss function.

[0118] Specifically, the comprehensive value of the first mean square error, the second mean square error and the first loss function can be determined as the second loss function. That is, the sum of the first mean square error, the second square error and the first loss function is calculated to obtain the sum as the second loss function.

[0119] In the above embodiments, based on the number of samples, the encoding and decoding features corresponding to the first multimedia sample feature representation, the encoding and decoding features corresponding to the second multimedia sample feature representation, the first target value, and the second target value, the corresponding first mean squared error and second mean squared error are obtained. Then, based on the first mean squared error, the second mean squared error, and the first loss function, a second loss function is determined. Since the second loss function is the loss function used for model training when the account similarity is less than a set similarity threshold, and this second loss function is obtained by combining multiple different losses, it can improve the model's learning effect and make the model's output more accurate.

[0120] In one embodiment, such as Figure 6 As shown, in step S430, based on the number of samples, a multimedia sample feature representation, a first decoded feature representation, a second multimedia sample feature representation, and a second decoded feature representation, as well as a preset first target value, a first mean square error is obtained, which may specifically include:

[0121] In step S610, a first similarity is determined between the first multimedia sample feature representation and the first decoded feature representation of each first multimedia sample.

[0122] For example, a computer device can treat multiple first multimedia samples of a first account as a sample sequence. This sample sequence, after being encoded by an encoder, yields a corresponding encoded feature sequence, i.e., a sequence of first multimedia sample feature representations. Subsequently, after being decoded by a decoder, a decoded feature sequence, i.e., a sequence of first decoded feature representations, is obtained. For each sample in the sample sequence, the computer device can calculate the first similarity between its encoded features and its corresponding decoded features. That is, it calculates the first similarity between the first multimedia sample feature representation and the first decoded feature representation of each first multimedia sample.

[0123] In step S620, a second similarity is determined between the second multimedia sample feature representation and the second decoded feature representation of each second multimedia sample.

[0124] Similarly, for multiple second multimedia samples of a second account, the computer device can calculate the second similarity between the second multimedia sample feature representation and the second decoded feature representation of each second multimedia sample.

[0125] In step S630, the sum of squared differences between each first similarity and second similarity and the first target value is obtained.

[0126] In step S640, the quotient of the sum of squares of the first difference and the number of samples is determined as the first mean square error.

[0127] Specifically, the computer device can calculate the corresponding first mean squared error based on the formula for calculating the mean squared error, and according to the number of samples, the first similarity, the second similarity, and the first target value.

[0128] For example, the mean square error can be calculated using the following formula:

[0129] Where f(x) is the predicted value, which is the similarity between the encoded and decoded features of each multimedia sample, y is the target value, which is the preset first target value, and n is the number of samples. Then, the first similarity between the first multimedia sample feature representation and the first decoded feature representation of each first multimedia sample of the first user can be calculated, as well as the second similarity between the second multimedia sample feature representation and the second decoded feature representation of each second multimedia sample. The sum of squared differences between each first similarity and each second similarity and the maximum similarity 1 (i.e., the first target value) is calculated, and the result of dividing the sum of squared differences by the number of samples is taken as the first mean square error.

[0130] In the above embodiments, a first similarity is determined between the decoded and encoded features of each first multimedia sample, and a second similarity is determined between the decoded and encoded features of each second multimedia sample. The sum of squared differences between each first and second similarity and a first target value is then obtained. Finally, a first mean squared error is determined based on the number of samples and the sum of squared differences. Since the first mean squared error can narrow the distance between similar user behaviors, making them similar, training the model with it enables the model to learn the features of the same user, thereby improving the accuracy of the model's content prediction.

[0131] In one embodiment, such as Figure 7 As shown, in step S440, based on the number of samples, the first multimedia sample feature representation, the first decoding feature representation, the second multimedia sample feature and the second decoding feature representation, and a preset second target value, a second mean square error is obtained, which may specifically include:

[0132] In step S710, a third similarity is determined between the second decoded feature representation of each second multimedia sample and the first multimedia sample feature of each first multimedia sample.

[0133] In step S720, a fourth similarity is determined between the second multimedia sample feature representation of each second multimedia sample and the first decoded feature representation of each first multimedia sample.

[0134] The third similarity and the fourth similarity refer to the similarity between the decoding feature of the multimedia sample of a user and the encoding feature of the multimedia sample of another user. For example, the similarity can be between the first multimedia sample feature representation of the first account and the second decoding feature representation of the second account, or the similarity can be between the second multimedia sample feature representation of the second account and the first decoding feature representation of the first account.

[0135] In this embodiment, the computer device can determine the third similarity between the second decoding feature representation of each second multimedia sample and the first multimedia sample feature representation of each first multimedia sample, and determine the fourth similarity between the second multimedia sample feature representation of each second multimedia sample and the first decoding feature representation of each first multimedia sample, according to the corresponding encoding feature and decoding feature of each first multimedia sample feature representation and each second multimedia sample feature representation.

[0136] In step S730, the second sum of square differences between each third similarity and fourth similarity and the second target value is obtained.

[0137] In step S740, the quotient of the second sum of square differences and the number of samples is determined as the second mean square error.

[0138] Specifically, the computer device can also determine the third similarity between the decoding feature of each second multimedia sample feature representation and the encoding feature of each first multimedia sample feature representation, and determine the fourth similarity between the encoding feature of each second multimedia sample and the corresponding decoding feature of each first multimedia sample feature representation, and calculate the second sum of square differences between each third similarity and fourth similarity and the second target value 0, and divide the result by the number of samples to obtain the second mean square error. The purpose is to pull away the distance between different users. Thus, the model can learn the characteristics of different user behaviors, thereby improving the accuracy of the model output.

[0139] In one embodiment, the first multimedia sample feature representation and the second multimedia sample feature representation each include a corresponding encoding feature. The encoding feature is the corresponding feature obtained after the multimedia sample is subjected to feature extraction by an encoding feature extraction network (such as an Encoder). Then, as shown in FIG. 1B, in step S130, the account similarity between the first account and the second account is obtained, which can specifically include: Figure 8

[0140] In step S810, the account feature of the first account is determined according to the first multimedia sample feature representation of the plurality of first multimedia samples of the first account. ​

[0141] Specifically, the account feature of the first account is determined by calculating the average value of the encoding features corresponding to the plurality of first multimedia samples (i.e., the first multimedia sample feature representation), i.e., summing the encoding features corresponding to the plurality of first multimedia samples and calculating the average value thereof, and determining the obtained average value as the account feature of the first account.

[0142] In step S820, the account feature of the second account is determined according to the second multimedia sample feature representation of the plurality of second multimedia samples of the second account.

[0143] Similarly, the account feature of the second account is determined by summing the encoding features corresponding to the plurality of second multimedia samples of the second account (i.e., the second multimedia sample feature representation) and calculating the average value thereof, and determining the obtained average value as the account feature of the second account.

[0144] In step S830, the account similarity between the first account and the second account is calculated according to the account feature of the first account and the account feature of the second account.

[0145] The account similarity can be calculated in the manner of cosine similarity or distance similarity, etc. Specifically, the computer device can obtain the account similarity between the first account and the second account by calculation according to the account feature of the first account and the account feature of the second account obtained above.

[0146] In the above embodiment, the account feature of the first account is determined according to the encoding features corresponding to the plurality of first multimedia samples of the first account, the account feature of the second account is determined according to the encoding features corresponding to the plurality of second multimedia samples of the second account, and the account similarity between the first account and the second account is calculated according to the account feature of the first account and the account feature of the second account. Since the account similarity of the embodiment is calculated based on the encoding features of the multimedia samples of different accounts, the characteristics of whether the different accounts are similar in content consumption can be accurately reflected, and then the loss function for training the model can be determined based on the characteristics, thereby improving the training effect of the model.

[0147] In one embodiment, as shown in FIG. 8, the training method of the above multimedia feature representation model is further explained as follows. Figure 9 Figure 9 ​As shown, the multimedia sample data includes a first multimedia sample sequence U1 (u1, u2, u3, …, un) of a first account user1 under the same behavior, and a second multimedia sample sequence K1 (k1, k2, k3, …, kn) of a second account user2. After the first multimedia sample sequence U1 and the second multimedia sample sequence K1 pass through the embedding layer (such as embedding) of the image feature extraction network such as swin and the text feature extraction network, the image feature Image and the text feature text tokens of each sample are obtained. Then, the image feature Image and the text feature text tokens of each sample are processed by the multimodal model (which can also be other encoding feature extraction networks such as Encoder), so as to obtain the corresponding encoding feature, that is, the first encoding feature sequence U2 corresponding to the first multimedia sample sequence U1, and the second encoding feature sequence K2 corresponding to the second multimedia sample sequence K1.

[0148] The first encoding feature sequence U2 and the second encoding feature sequence K2 are decoded by the decoding feature extraction network such as Decoder-photo, that is, the corresponding decoding result first decoding feature sequence U3 and the second decoding feature sequence K3 are obtained, and each element in the sequence is the decoding feature of the corresponding sample. At the same time, the first encoding feature sequence U2 and the second encoding feature sequence K2 can be decoded and predicted by the content generation network such as Decoder-Gen, so as to obtain the corresponding first decoding prediction sequence U4 and the second decoding prediction sequence K4, and each element in the sequence is the predicted content of the corresponding sample, for example, the predicted topic label of the sample.

[0149] Further, the corresponding loss can be calculated based on the above extracted features. For example, according to the content label of each sample in the first multimedia sample sequence U1 and the second multimedia sample sequence K1, and the predicted content of each sample in the first decoding prediction sequence U4 and the second decoding prediction sequence K4, the loss in the corresponding content dimension, that is, the first loss function, can be calculated. For details, refer to the method shown in Figure 3 The embodiment will not be described here.

[0150] The first mean square error between the encoding feature and the decoding feature of each multimedia sample of the same user can also be calculated according to the encoding feature of each sample in the first encoding feature sequence U2 and the second encoding feature sequence K2, and the decoding feature of each sample in the first decoding feature sequence U3 and the second decoding feature sequence K3. For details, refer to the method shown in Figure 6 The embodiment will not be described here.

[0151] The second mean square error between the encoding features and the decoding features of different users can also be calculated according to the encoding features of each sample in the first encoding feature sequence U2 and the second encoding feature sequence K2, and the decoding features of each sample in the first decoding feature sequence U3 and the second decoding feature sequence K3. The calculation can be performed according to the method shown in Figure 7 , and the present embodiment will not be described here.

[0152] The account similarity between the first account and the second account can be calculated according to the first encoding feature sequence U2 and the second encoding feature sequence K2. The calculation can be performed according to the method shown in Figure 8 , and the present embodiment will not be described here. Then, the loss function for training the model is determined according to the size of the account similarity. For example, when the account similarity is greater than or equal to 0.5, the first loss function calculated above is directly used as the loss function for training the model. When the account similarity is less than 0.5, the sum of the first loss function, the first mean square error and the second mean square error calculated above is used as the loss function for training the model. Since the loss function for training the model in the present embodiment is determined according to different parameters according to the size of the account similarity, the model can not only learn the content features of the sample data, but also learn the user behavior features, thereby improving the accuracy of the output features of the model.

[0153] In one embodiment, as shown in Figure 10 , a method for generating a multimedia feature representation is also provided, which can specifically include the following steps:

[0154] In step S1010, multimedia data is obtained.

[0155] The multimedia data can be data in various media forms such as text, sound and image. For example, it can be video data. Specifically, the multimedia data can be multimedia data used for pushing in a recommendation service, or multimedia data used for classification in a classification service, etc.

[0156] In step S1020, the multimedia data is input into a multimedia feature representation model to obtain a multimedia feature representation of the multimedia data.

[0157] The multimedia feature representation model is a model trained based on the method described above Figures 1 to 9 . The multimedia feature representation can at least include encoding features with content dimensions and user behavior dimensions.

[0158] Specifically, by inputting multimedia data into a multimedia feature representation model, a multimedia feature representation of the multimedia data can be obtained. This multimedia feature representation can then be used to recommend or classify the multimedia data. Since the multimedia feature representation in this embodiment possesses not only content-level encoded features but also user behavior-level encoded features, it enables more accurate recommendations or classifications.

[0159] It should be understood that, although Figures 1-10 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figures 1-10 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0160] It is understood that the same / similar parts between the various embodiments of the methods described above in this specification can be referred to each other. Each embodiment focuses on the differences from other embodiments, and relevant parts can be referred to the description of other method embodiments.

[0161] Figure 11 This is a block diagram of a training apparatus for a multimedia feature representation model according to an exemplary embodiment. (Refer to...) Figure 11 The device includes a sample data acquisition module 1102, a feature extraction module 1104, a similarity acquisition module 1106, and a model training module 1108.

[0162] The sample data acquisition module 1102 is configured to acquire multimedia sample data, which includes multiple first multimedia samples of a first account under the same behavior and multiple second multimedia samples of a second account under the same behavior.

[0163] The feature extraction module 1104 is configured to perform feature extraction on the first multimedia sample using an initial feature extraction network to obtain a feature representation of the first multimedia sample; and to perform feature extraction on the second multimedia sample using the initial feature extraction network to obtain a feature representation of the second multimedia sample.

[0164] The similarity acquisition module 1106 is configured to acquire the account similarity between the first account and the second account;

[0165] The model training module 1108 is configured to train the initial feature extraction network according to the account similarity, the first multimedia sample feature representation, and the second multimedia sample feature representation to obtain a multimedia feature representation model.

[0166] In an example embodiment, the model training module is configured to perform: determining a loss function for model training according to the account similarity, the first multimedia sample feature representation, and the second multimedia sample feature representation; adjusting network parameters of the initial feature extraction network according to the loss function; and determining the adjusted initial feature extraction network as the multimedia feature representation model when the adjusted initial feature extraction network satisfies a convergence condition.

[0167] In an example embodiment, the model training module is further configured to perform: when the account similarity is greater than or equal to a set similarity threshold, determining a first loss function according to the first multimedia sample feature representation and the second multimedia sample feature representation, and determining the first loss function as the loss function for model training; or when the account similarity is less than the set similarity threshold, determining a second loss function according to the first loss function, the first multimedia sample feature representation, and the second multimedia sample feature representation, and determining the second loss function as the loss function for model training.

[0168] In an example embodiment, the first multimedia sample and the second multimedia sample each have a corresponding content label; and the model training module is further configured to perform: obtaining a first predicted content using a content generation network according to the first multimedia sample feature representation of the first multimedia sample; obtaining a second predicted content using the content generation network according to the second multimedia sample feature representation of the second multimedia sample; obtaining a first cross-entropy loss between the content label and the first predicted content of each of the first multimedia samples according to the content label and the first predicted content of each of the first multimedia samples; obtaining a second cross-entropy loss between the content label and the second predicted content of each of the second multimedia samples according to the content label and the second predicted content of each of the second multimedia samples; and determining the first loss function according to the first cross-entropy loss and the second cross-entropy loss.

[0169] In an example embodiment, the model training module is further configured to perform: performing feature decoding processing according to the first multimedia sample feature representation of each of the first multimedia samples to obtain a first decoded feature representation of each of the first multimedia samples; performing feature decoding processing according to the second multimedia sample feature representation of each of the second multimedia samples to obtain a second decoded feature representation of each of the second multimedia samples; obtaining a first mean square error according to a sample number of the first multimedia samples and the second multimedia samples, the first multimedia sample feature representation, the first decoded feature representation, the second multimedia sample feature representation, the second decoded feature representation, and a preset first target value; obtaining a second mean square error according to the sample number, the first multimedia sample feature representation, the first decoded feature representation, the second multimedia sample feature representation, the second decoded feature representation, and a preset second target value; and determining a second loss function according to the first mean square error, the second mean square error, and the first loss function.

[0170] In an example embodiment, the model training module is further configured to perform: determining a first similarity between the first multimedia sample feature representation and the first decoded feature representation of each of the first multimedia samples; determining a second similarity between the second multimedia sample feature representation and the second decoded feature representation of each of the second multimedia samples; obtaining a first sum of squared differences between each of the first similarity and the second similarity and the first target value; and determining a quotient of the first sum of squared differences and the sample number as the first mean square error.

[0171] In an example embodiment, the model training module is further configured to perform: determining a third similarity between the second decoded feature representation of each of the second multimedia samples and the first multimedia sample feature of each of the first multimedia samples; determining a fourth similarity between the second multimedia sample feature representation of each of the second multimedia samples and the first decoded feature representation of each of the first multimedia samples; obtaining a second sum of squared differences between each of the third similarity and the fourth similarity and the second target value; and determining a quotient of the second sum of squared differences and the sample number as the second mean square error.

[0172] In an example embodiment, the model training module is further configured to perform: obtaining a sum of the first mean square error, the second mean square error, and the first loss function; and determining the sum as the second loss function.

[0173] In an example embodiment, the similarity obtaining module is further configured to perform: determining an account feature of the first account according to the first multimedia sample feature representations of the plurality of first multimedia samples of the first account; determining an account feature of the second account according to the second multimedia sample feature representations of the plurality of second multimedia samples of the second account; and calculating an account similarity between the first account and the second account according to the account feature of the first account and the account feature of the second account.

[0174] Figure 12 is a block diagram of an apparatus for generating a multimedia feature representation according to an example embodiment. Referring to Figure 12 , the apparatus includes a data obtaining module 1202 and a feature generating module 1204. Wherein,

[0175] The data obtaining module 1202 is configured to perform obtaining multimedia data.

[0176] The feature generating module 1204 is configured to perform inputting the multimedia data into a multimedia feature representation model to obtain a multimedia feature representation of the multimedia data. Wherein, the multimedia feature representation model is a model trained based on the above Figures 1 to 9 mentioned method. The multimedia feature representation can at least include an encoded feature with content dimension and user behavior dimension.

[0177] As to the apparatus in the above embodiments, the specific ways in which the respective modules perform operations have been described in detail in the embodiments of the method, and will not be described in detail here.

[0178] Figure 13 is a block diagram of an electronic device S00 for a training method of a multimedia feature representation model or a generation method of a multimedia feature representation according to an example embodiment. For example, the electronic device S00 can be a computer, a terminal or a server, etc. Referring to Figure 13 , the electronic device S00 includes a processing component S20, which further includes one or more processors, and a memory resource represented by a memory S22, for storing instructions executable by the processing component S20, such as an application program. The application program stored in the memory S22 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component S20 is configured to execute the instructions to perform the above method.

[0179] The electronic device S00 can further include a power supply component S24 configured to perform power management of the electronic device S00, a wired or wireless network interface S26 configured to connect the electronic device S00 to a network, and an input output (I / O) interface S28. The electronic device S00 can operate based on an operating system stored in the memory S22, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, or the like.

[0180] In exemplary embodiments, there is also provided a computer readable storage medium including instructions, such as the memory S22 including instructions, which can be executed by the processor of the electronic device S00 to complete the above method. The storage medium can be a computer readable storage medium, such as a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, etc.

[0181] In exemplary embodiments, there is also provided a computer program product including instructions, which can be executed by the processor of the electronic device S00 to complete the above method.

[0182] It should be noted that the above-described apparatus, electronic device, computer readable storage medium, computer program product, etc. according to the description of the method embodiments can also include other implementations, and the specific implementation can be referred to the description of the related method embodiments, which will not be described here.

[0183] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features of the present disclosure disclosed herein. It is intended that the present disclosure cover any and all variations of the present disclosure including those variations contained within the spirit and scope of the features of the present disclosure. It is intended that the specification and examples be considered exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0184] It should be understood that the present disclosure is not limited to the precise structures herein described and illustrated in the drawings and that various modifications and changes can be made therein without departing from the scope thereof. The scope of the present disclosure is indicated by the appended claims.

Claims

1. A method for training a multimedia feature representation model, characterized in that, The method comprises: obtaining multimedia sample data, the multimedia sample data comprising a plurality of first multimedia samples of a first account under the same behavior and a plurality of second multimedia samples of a second account under the same behavior; the multimedia sample data is data comprising text, sound and image; extracting features of the first multimedia samples by using an initial feature extraction network to obtain first multimedia sample feature representations; extracting features of the second multimedia samples by using the initial feature extraction network to obtain second multimedia sample feature representations; obtaining an account similarity between the first account and the second account; training the initial feature extraction network according to the account similarity, the first multimedia sample feature representations and the second multimedia sample feature representations to obtain a multimedia feature representation model; the training of the initial feature extraction network according to the account similarity, the first multimedia sample feature representations and the second multimedia sample feature representations to obtain a multimedia feature representation model comprises: when the account similarity is greater than or equal to a set similarity threshold, determining a first loss function according to the first multimedia sample feature representations and the second multimedia sample feature representations, and determining the first loss function as a loss function used for model training; when the account similarity is less than the set similarity threshold, determining a second loss function according to the first loss function, the first multimedia sample feature representations and the second multimedia sample feature representations, and determining the second loss function as the loss function used for model training; adjusting network parameters of the initial feature extraction network according to the loss function; when the adjusted initial feature extraction network meets a convergence condition, determining the adjusted initial feature extraction network as the multimedia feature representation model.

2. The method of claim 1, wherein, The first multimedia samples and the second multimedia samples each have a corresponding content label; the determination of the first loss function according to the first multimedia sample feature representations and the second multimedia sample feature representations comprises: obtaining first predicted content by using a content generation network according to the first multimedia sample feature representations of the first multimedia samples; obtaining second predicted content by using the content generation network according to the second multimedia sample feature representations of the second multimedia samples; obtaining a first cross-entropy loss between the first predicted content and the content label of each of the first multimedia samples according to the content label and the first predicted content of each of the first multimedia samples; obtaining a second cross-entropy loss between the second predicted content and the content label of each of the second multimedia samples according to the content label and the second predicted content of each of the second multimedia samples; determining the first loss function according to the first cross-entropy loss and the second cross-entropy loss.

3. The method according to claim 1 or 2, characterized in that, the determination of the second loss function according to the first loss function, the first multimedia sample feature representations and the second multimedia sample feature representations comprises: According to the first multimedia sample feature representation of each of the first multimedia samples, a feature decoding process is performed to obtain a first decoded feature representation of each of the first multimedia samples; According to the second multimedia sample feature representation of each of the second multimedia samples, a feature decoding process is performed to obtain a second decoded feature representation of each of the second multimedia samples; According to the sample quantity of the first multimedia samples and the second multimedia samples, the first multimedia sample feature representation, the first decoded feature representation, the second multimedia sample feature representation, the second decoded feature representation, and a preset first target value, a first mean square error is obtained; According to the sample quantity, the first multimedia sample feature representation, the first decoded feature representation, the second multimedia sample feature representation, the second decoded feature representation, and a preset second target value, a second mean square error is obtained; According to the first mean square error, the second mean square error, and the first loss function, a second loss function is determined.

4. The method of claim 3, wherein, The first mean square error is obtained according to the sample quantity of the first multimedia samples and the second multimedia samples, the first multimedia sample feature representation, the first decoded feature representation, the second multimedia sample feature representation, the second decoded feature representation, and a preset first target value, including: A first similarity between the first multimedia sample feature representation and the first decoded feature representation of each of the first multimedia samples is determined; A second similarity between the second multimedia sample feature representation and the second decoded feature representation of each of the second multimedia samples is determined; A first sum of square differences between each of the first similarity and the second similarity and the first target value is obtained; The first sum of square differences divided by the sample quantity is determined as the first mean square error.

5. The method of claim 3, wherein, The second mean square error is obtained according to the sample quantity, the first multimedia sample feature representation, the first decoded feature representation, the second multimedia sample feature representation, the second decoded feature representation, and a preset second target value, including: A third similarity between the second decoded feature representation of each of the second multimedia samples and the first multimedia sample feature of each of the first multimedia samples is determined; A fourth similarity between the second multimedia sample feature representation of each of the second multimedia samples and the first decoded feature representation of each of the first multimedia samples is determined; A second sum of square differences between each of the third similarity and the fourth similarity and the second target value is obtained; The second sum of square differences divided by the sample quantity is determined as the second mean square error.

6. The method of claim 3, wherein, The second loss function is determined according to the first mean square error, the second mean square error, and the first loss function, including: The sum of the first mean square error, the second mean square error, and the first loss function is obtained, and the sum is determined as the second loss function.

7. The method according to claim 1 or 2, characterized in that, The account similarity between the first account and the second account is obtained by: determining an account feature of the first account according to the first multimedia sample feature representation of the plurality of first multimedia samples of the first account; determining an account feature of the second account according to the second multimedia sample feature representation of the plurality of second multimedia samples of the second account; calculating the account similarity between the first account and the second account according to the account feature of the first account and the account feature of the second account.

8. A method of generating a multimedia feature representation, characterized by, The method comprises: obtaining multimedia data; inputting the multimedia data into the multimedia feature representation model obtained by the method of any one of claims 1 to 7 to obtain a multimedia feature representation of the multimedia data, the multimedia feature representation comprising at least an encoded feature with content dimensions and user behavior dimensions.

9. A training device for a multimedia feature representation model, characterized in that, The device is applied to the training method of the multimedia feature representation model of any one of claims 1 to 7, and the device comprises: a sample data obtaining module configured to obtain multimedia sample data, the multimedia sample data comprising a plurality of first multimedia samples of a first account under the same behavior and a plurality of second multimedia samples of a second account under the same behavior; a feature extraction module configured to perform feature extraction on the first multimedia samples by using an initial feature extraction network to obtain first multimedia sample feature representation, and perform feature extraction on the second multimedia samples by using the initial feature extraction network to obtain second multimedia sample feature representation; a similarity obtaining module configured to obtain the account similarity between the first account and the second account; a model training module configured to train the initial feature extraction network according to the account similarity, the first multimedia sample feature representation and the second multimedia sample feature representation to obtain a multimedia feature representation model.

10. An apparatus for generating a multimedia feature representation, characterized by The device comprises: a data obtaining module configured to obtain multimedia data; a feature generation module configured to input the multimedia data into the multimedia feature representation model obtained by the method of any one of claims 1 to 7 to obtain a multimedia feature representation of the multimedia data, the multimedia feature representation comprising at least an encoded feature with content dimensions and user behavior dimensions.

11. An electronic device, comprising: comprise: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the method of any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the method of any one of claims 1 to 8.

Images