Multimedia data recommendation method, related device, equipment and storage medium

Abstract

The invention discloses a multimedia data recommendation method, a related device, equipment and a storage medium. In the method provided by the invention, a first model can perform feature extraction on first object data according to a plurality of first virtual kernel variables to obtain a plurality of second embedded vectors, then the second embedded vectors are connected to generate a first embedded vector, and the plurality of first virtual kernel variables are generated according to training of target object side embedded vector fusion guided by a multimedia side embedded vector. The first model performs knowledge distillation using a teacher network that is trained and generated using target object data and multimedia data and cross data. According to the similarity between the first embedded vector and a third embedded vector, data to be recommended are determined and output from the multiple pieces of target multimedia data, and the third embedded vector is generated by connecting an embedded vector obtained by conducting feature extraction on the target multimedia data with the multiple first virtual kernel variables. Through the mode, the feature fitting degree of the multimedia side and the target object side is improved, and the model recommendation effect is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a multimedia data recommendation method, related devices, equipment and storage medium. Background Art

[0002] Recommendation systems usually include four stages: recall, rough sorting, fine sorting, and re-ranking. Each stage is like a funnel, filtering out items that users are more interested in from a large set of items. Recommendations require "thousands of things with thousands of faces": the same user is interested in different types of advertisements for different reasons. From the perspective of representation learning, it is insufficient to use a single expression to describe the multi-faceted characteristics of users. In the rough sorting stage, because hundreds or thousands of items need to be sorted, too complex models cannot be used. In the rough sorting system, the double tower model is usually used.

[0003] The current processing method of the dual-tower model is to encode the user-side features and item features separately, and then send the representation to the last layer for interaction to obtain the final score. However, in the ordinary dual-tower structure model, only the top layer interacts, and the vector reasoning of the two towers is not aware of each other. The learning of the target is usually in an underfitting state, which will reduce the effect of the model. Summary of the invention

[0004] The embodiments of the present application provide a multimedia data recommendation method, related devices, equipment and storage medium, which are used to use a teacher network to perform knowledge distillation to update the parameters of a first model, so that the features extracted by the parameters of the first model are more in line with the features on the multimedia side. During the training process, multiple virtual kernel variables are guided by the embedding vector on the multimedia side, and then the embedding vector on the target object side is generated, and the embedding vector on the target object side is matched with the embedding vector on the multimedia side. Not only the highest-level features interact, but also the fit of the features on the multimedia side and the target object side is further improved, thereby improving the recommendation effect of the model.

[0005] In view of this, the present application provides a multimedia data recommendation method, comprising:

[0006] Obtaining a recommendation request, the recommendation request including first object data of a first object;

[0007] Performing feature extraction on the first object data according to the first model to obtain a first embedding vector of the first object, the first embedding vector is generated by connecting multiple second embedding vectors, the multiple second embedding vectors are generated by the first object data through multiple first virtual kernel variables in the first model, the multiple first virtual kernel variables are generated by training the multimedia side embedding vector and the target object side embedding vector, the target object side embedding vector is generated by weighted fusion based on the guidance of the multimedia side embedding vector, the first model is obtained by performing knowledge distillation using a teacher network, the training data of the teacher network is the target object data, the multimedia data and the cross data, and the cross data is the data after the target object data and the multimedia data features are combined;

[0008] Obtaining a third embedding vector corresponding to the target multimedia data, where the third embedding vector is generated by connecting an embedding vector obtained by performing feature extraction on the target multimedia data with a plurality of first virtual kernel variables;

[0009] Determining to-be-recommended data from a plurality of target multimedia data according to the first embedding vector and the third embedding vector;

[0010] Output the data to be recommended.

[0011] Another aspect of the present application provides a multimedia data recommendation device, comprising:

[0012] An acquiring unit, configured to acquire a recommendation request, wherein the recommendation request includes first object data of a first object;

[0013] an extraction unit, configured to perform feature extraction on first object data according to a first model to obtain a first embedding vector of the first object, wherein the first embedding vector is generated by connecting a plurality of second embedding vectors, the plurality of second embedding vectors are generated by the first object data through a plurality of first virtual kernel variables in the first model, the plurality of first virtual kernel variables are generated by training of multimedia-side embedding vectors and target-object-side embedding vectors, the target-object-side embedding vector is generated by weighted fusion based on guidance of the multimedia-side embedding vector, the first model is obtained by performing knowledge distillation using a teacher network, the training data of the teacher network are target object data, multimedia data and cross data, and the cross data is data after the features of the target object data and the multimedia data are combined;

[0014] The acquisition unit is further used to acquire a third embedding vector corresponding to the target multimedia data, where the third embedding vector is generated by connecting an embedding vector obtained by performing feature extraction on the target multimedia data with a plurality of first virtual kernel variables;

[0015] A determination unit, configured to determine the data to be recommended from the plurality of target multimedia data according to the first embedding vector and the third embedding vector;

[0016] The output unit is used to output the data to be recommended.

[0017] In a possible design, in another implementation of another aspect of the embodiment of the present application, the acquisition unit is further used to:

[0018] Acquire target object data and multimedia data;

[0019] The extraction unit is further used to perform feature extraction on the target object data according to the second model to obtain a plurality of fourth embedding vectors; perform feature extraction on the multimedia data according to the second model to obtain a fifth embedding vector;

[0020] The device also includes a weighting unit, which is specifically used to perform weighted fusion on the plurality of fourth embedding vectors according to a first weight to obtain a sixth embedding vector, wherein the first weight is indicated by the fifth embedding vector;

[0021] The device also includes a training unit, which is specifically used to train the second model according to the sixth embedding vector and the fifth embedding vector to obtain the first model;

[0022] The acquisition unit is also used to acquire cross data;

[0023] The training unit is also used to train a teacher network based on target object data, multimedia data and cross data; and update parameters of the teacher network and the first model based on the loss of the teacher network, the loss of the first model and the distillation loss.

[0024] In a possible design, in another implementation of another aspect of the embodiment of the present application, the extraction unit is specifically used for:

[0025] Extract features from the target object data through the first embedding layer of the second model to obtain a seventh embedding vector;

[0026] The seventh embedding vector is input into a plurality of first expert networks of the second model to output a plurality of fourth embedding vectors.

[0027] In a possible design, in another implementation of another aspect of the embodiment of the present application, the weighting unit is further configured to:

[0028] Matrix multiplication and normalization are performed on the fifth embedding vector according to the second virtual kernel variable of each first expert network in the multiple first expert networks to obtain a first weight.

[0029] In a possible design, in another implementation of another aspect of the embodiment of the present application, the acquiring unit is specifically configured to:

[0030] Based on the multimedia selection of the target object, obtaining target object data of the target object;

[0031] The multimedia selection data of the target object within a preset time range is obtained, where the multimedia data includes the multimedia selection data.

[0032] In a possible design, in another implementation of another aspect of the embodiment of the present application, the acquiring unit is specifically configured to:

[0033] Use multimedia selection data as positive samples;

[0034] Get random multimedia targets in the database as negative samples, and the multimedia data includes positive samples and negative samples.

[0035] In a possible design, in another implementation of another aspect of the embodiment of the present application, the determining unit is further configured to:

[0036] The loss of the first model is determined according to a loss function, wherein a temperature hyperparameter of the loss function is lower than a preset value.

[0037] In a possible design, in another implementation of another aspect of the embodiment of the present application, the extraction unit is specifically used for:

[0038] Dividing the multimedia data into a plurality of data sets, the plurality of data sets including a training set;

[0039] Feature extraction is performed on the training set according to the second model to obtain a fifth embedding vector.

[0040] In one possible design, in another implementation of another aspect of the embodiment of the present application, the multiple data sets further include a test set, and the apparatus further includes a verification unit, and the verification unit is specifically used to:

[0041] The first model is verified using the test set.

[0042] In a possible design, in another implementation of another aspect of the embodiment of the present application, the verification unit is further used to:

[0043] One of the multiple data sets is determined as a test set in turn, and the data sets other than the test set are used as training sets.

[0044] In a possible design, in another implementation of another aspect of the embodiment of the present application, the determining unit is specifically configured to:

[0045] Determine a characteristic distance between a first embedding vector corresponding to each target multimedia data and a third embedding vector of the target multimedia data;

[0046] The target multimedia data corresponding to the feature distance less than the preset threshold is used as the data to be recommended.

[0047] Another aspect of the present application provides a computer device, comprising:

[0048] memories, transceivers, processors, and bus systems;

[0049] Wherein, the memory is used to store programs;

[0050] The processor is used to execute the program in the memory, including executing the above-mentioned methods;

[0051] The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

[0052] Another aspect of the present application provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer is enabled to execute the above-mentioned methods.

[0053] Another aspect of the present application provides a computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided by the above aspects.

[0054] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0055] In an embodiment of the present application, after receiving a recommendation request, the first object data of the first object to be recommended is input into the first model to output the first embedding vector of the first object, wherein the first model can extract features from the first object data based on a plurality of first virtual kernel variables to obtain a plurality of second embedding vectors, and then connect the plurality of second embedding vectors to generate a first embedding vector, wherein the plurality of first virtual kernel variables are generated by training to guide the fusion of the target object side embedding vector based on the multimedia side embedding vector. Among them, the first model uses a teacher network for knowledge distillation, and the teacher network is generated by training using target object data, multimedia data, and cross data. For multiple target multimedia data in the database, according to the similarity between the first embedding vector and the third embedding vector, the data to be recommended can be determined from the multiple target multimedia data, and the data to be recommended can be output, and the third embedding vector is generated by connecting the embedding vector obtained by feature extraction of the target multimedia data with a plurality of first virtual kernel variables. Through the above method, the teacher network is used for knowledge distillation to update the parameters of the first model, so that the features extracted by the parameters of the first model are more in line with the features on the multimedia side. Multiple virtual kernel variables are guided by the embedding vectors on the multimedia side during the training process, and then the embedding vectors on the target object side are generated, and the embedding vectors on the target object side are matched with the embedding vectors on the multimedia side. Not only the highest-level features interact, but also the fit of the features on the multimedia side and the target object side is further improved, thereby improving the recommendation effect of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a schematic diagram of an architecture diagram of a double-tower model in an embodiment of the present application;

[0057] Figure 2 A flowchart of a multimedia data recommendation method in an embodiment of the present application;

[0058] Figure 3 A schematic diagram of the structure of a first model and a teacher network in an embodiment of the present application;

[0059] Figure 4 This is a schematic diagram of the structure of a first expert network in an embodiment of the present application;

[0060] Figure 5 This is a VKG weighting schematic diagram in an embodiment of the present application;

[0061] Figure 6 A schematic diagram of the structure of another first model and a teacher network in an embodiment of the present application;

[0062] Figure 7 This is a schematic diagram of the online part in the embodiment of this application;

[0063] Figure 8This is a schematic diagram of the structure of a multimedia data recommendation device in an embodiment of the present application;

[0064] Fig. 9 It is a structural schematic diagram of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0065] The embodiments of the present application provide a multimedia data recommendation method, related devices, equipment and storage medium, which are used to distill the loss of the learning teacher network to update the parameters of the first model, so that the features extracted by the parameters of the first model are more in line with the multimedia side features. Multiple virtual kernel variables are guided by the embedding vector on the multimedia side during the training process, and then the embedding vector on the target object side is generated, and the embedding vector on the target object side is matched with the embedding vector on the multimedia side. Not only the highest-level features interact, but also the fit of the features on the multimedia side and the target object side is further improved, thereby improving the recommendation effect of the model.

[0066] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "corresponding to" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0067] This application specifically relates to the twin tower model in artificial neural networks in machine learning, see Figure 1 , Figure 1 This is a double tower model architecture provided by the embodiment of the present application. Figure 1The full name of the dual-tower model is deep structured semantic models (DSS). It is widely used in recommendation systems because of its good effect and its friendly to massive data processing in the industry. The dual-tower model is divided into an object tower and an Item tower. Since the dual-tower model in this application is applied to multimedia data recommendation scenarios, the Item tower can be called a multimedia data tower. The object tower extracts the object feature UserEmbedding based on the object information of the object (for example, basic information of the object, group statistical attributes, and multimedia data that have interacted, etc.), and the multimedia data tower extracts the multimedia feature Item Embedding based on the attribute information of the multimedia data (for example, basic information of multimedia data, attribute information, etc.). The object model in the object tower (or the multimedia model in the multimedia data tower) can be composed of a deep neural network (DNN) + a multilayer perceptron (MLP). Finally, the distance between the object features output by the object tower and the multimedia features output by the multimedia data tower is used to determine whether the multimedia data is the multimedia data to be recommended for the object.

[0068] During the training of the dual-tower model, the inner product or cosine similarity calculation is performed on the object feature User Embedding and the multimedia feature Item Embedding, so that the multimedia data of the object and the positive sample are closer in the feature space and the distance from the multimedia data of the negative sample is increased in the feature space.

[0069] From the above, we can see that the dual-tower model has the characteristic of "separation", that is, in the bottom layer of the model, the object information and the attribute information of the multimedia data do not depend on each other, so a large number of multimedia feature vectors can be generated offline in batches periodically, which greatly reduces the pressure on the online server. However, this structural feature also brings some problems. The information interaction is weak. The interaction between objects and multimedia data is a cross calculation at the top layer of the dual tower. At this time, the vectors involved in the cross calculation are already high-dimensional vectors, and the information is highly concentrated. The vector reasoning of the dual towers is not aware of each other, and the learning of the target is usually in an underfitting state, which will reduce the effect of the model.

[0070] To solve the above problems, an embodiment of the present application provides a multimedia data recommendation method, which guides the weighted fusion of multiple second embedding vectors on the target object side through a third embedding vector on the multimedia side, thereby improving the fit of the features on the multimedia side and the target object side and improving the recommendation effect of the model. The method is described as follows.

[0071] The object can be a user, and the object data is the user data. The object data can include fixed-length discrete features, non-fixed-length discrete features, and continuous features. Fixed-length discrete features refer to non-sequential features with discrete values ​​and fixed length. The fixed-length discrete features of object information mainly include: user gender (male / female), user age group (youth / middle-aged / elderly, etc.), user activity level (low activity / medium activity / high activity), user job category (lawyer / programmer / teacher, etc.), user current city type (first-tier city / second-tier city / third-tier city, etc.), user mobile phone system (Android / IOS), user ID, etc. Non-fixed-length discrete features refer to sequence features with discrete values ​​and indefinite length. The non-fixed-length discrete features of object information mainly include: multimedia data sequence exposed by users, multimedia data sequence clicked by users, multimedia data sequence liked by users, and main sequence of posting numbers followed by users. Continuous features refer to features that take continuous values. The continuity features of object information mainly include: the user's age, the statistical values ​​of the user's current gender group for multimedia data types (exposures / clicks / click-through rate), the statistical values ​​of the user's current age group for multimedia data types (exposures / clicks / click-through rate), and the statistical values ​​of the user's historical one-month multimedia data types (exposures / clicks / click-through rate).

[0072] Multimedia data can be any type of data such as video, text, image, and audio, and mainly includes fixed-length discrete features and continuous features. Fixed-length discrete features mainly include: multimedia data ID, multimedia data type (entertainment / sports / economy, etc.), multimedia data release number master ID, etc. The continuous features of attribute information mainly include: statistics of multimedia data (exposure number / click number / click rate / share number / forward number / multimedia data duration, etc.), statistics of the classification of multimedia data (exposure number / click number / click rate / share number / forward number, etc.), etc. In the embodiment of the present application, multimedia data takes advertising data as an example.

[0073] See also Figure 2 ,like Figure 2 FIG. 1 is a flow chart of a multimedia data recommendation method provided in an embodiment of the present application, the method comprising:

[0074] Step 201: Obtain a recommendation request, where the recommendation request includes first object data of a first object.

[0075] In one or more embodiments, the recommendation request is a recommendation request for multimedia data, and the request may be actively initiated by the first object through the client, for example, the first object performs a refresh operation on the client, and the client determines the first object data of the first object based on the refresh operation, and generates a recommendation request and uploads it to the server. The request may also be automatically generated by the server, for example, the server generates the recommendation request for the first object regularly or irregularly.

[0076] Step 202. Perform feature extraction on the first object data according to the first model to obtain a first embedding vector of the first object, the first embedding vector is generated by connecting multiple second embedding vectors, the multiple second embedding vectors are generated by the first object data through multiple first virtual kernel variables in the first model, the multiple first virtual kernel variables are generated by training the multimedia side embedding vector and the target object side embedding vector, the target object side embedding vector is generated by weighted fusion based on the guidance of the multimedia side embedding vector, the first model is obtained by using a teacher network for knowledge distillation, the training data of the teacher network is the target object data, multimedia data and cross data, and the cross data is data after the target object data and multimedia data features are combined.

[0077] In one or more embodiments, the first model is a dual-tower model, and the first object data needs to be processed by the object tower in the first model, wherein the object tower can extract the data field in the first object data and map it into multiple second embedding vectors, and the multiple second embedding vectors are multiple different user representation vectors generated based on different first virtual kernel variables, and these representation vectors each focus on different aspects of user preference learning, and these second embedding vectors can be connected into a vector, which can be called a first embedding vector. In the embodiment of the present application, the embedding vector extracted from the target multimedia data to be compared can be combined with the multiple first virtual kernel variables to obtain a third embedding vector (such as ad embedding), that is, when requesting a recommendation, the first embedding vector is equivalent to being weighted with multiple first virtual kernel variables of the third embedding vector, and the multiple first virtual kernel variables are generated by training to guide the fusion of the target object side embedding vector based on the multimedia side embedding vector, that is, during the training process of the first model, the weights of the multiple embedding vectors on the target object side generated based on different first virtual kernel variables are affected by the multimedia side embedding vector during weighted fusion, so that the focus of the generated first embedding vector is closer to the multimedia side, and the virtual kernel variable of the first model generated by training can be called the first virtual kernel variable. Before generating the first model, the present application also trains a teacher network. In the end, the teacher network is not online. The teacher network can be trained using pure target object data and multimedia data, and can also be trained in combination with cross data of target object data and multimedia data. The cross data can be in the form of data after feature combination of some features of the target object data and some features of the multimedia data. The cross data can be used to train the teacher network. Due to the use of cross data, the features on the multimedia side of the teacher network have a high degree of fit with the features on the target object side. The first model can learn the fit of the teacher network by means of knowledge distillation. Among them, knowledge distillation is a training method of the student network model guided by the teacher model by constructing a teacher model and a student model, and transferring the learning behavior of the teacher model with a complex network structure, a large number of parameters, and a strong learning ability to the student model with a simple network structure, a small number of parameters, and a weak learning ability. The first model in the embodiment of the present application is a student model.

[0078] Exemplarily, the first model and the teacher network can refer to Figure 3As shown, the user tower of the first model includes virtual kernel variables 1 and virtual kernel variables 2, and can generate embedding vectors of different attention directions based on virtual kernel variables 1 and virtual kernel variables 2 (taking two virtual kernel variables as an example) for user input, and then perform weighted fusion on the embedding vectors of different attention directions based on the weighted fusion function, wherein the multimedia tower of the first model can generate the embedding vector of the multimedia side (such as the multimedia input) based on the multimedia input (such as the advertisement input ad-input) through a neural network (such as a deep neural network algorithm (DNN)). Figure 3 The ad embedding in the ad tower can participate in the weighted fusion of the user tower, that is, it guides the generation of the first embedding vector (user embedding). Then the first model can calculate the loss based on the user embedding and the ad embedding. The embodiment of the present application also provides a teacher network, the input of which is user-input, ad-input, and the cross-features of user-advertisement. By training the teacher network with user-input, ad-input, and the cross-features of user-advertisement, a teacher network with higher accuracy than the first model can be obtained, and then the loss of the teacher network is calculated by the logit model. Finally, the parameters of the teacher network and the first model can be updated based on the loss of the teacher network, the loss of the first model, and the distillation loss.

[0079] Step 203: Obtain a third embedding vector corresponding to the target multimedia data, where the third embedding vector is generated by connecting an embedding vector obtained by performing feature extraction on the target multimedia data with a plurality of first virtual kernel variables.

[0080] In one or more embodiments, the target multimedia data is input when training the first model, and the database has stored the ad embeddings corresponding to the target multimedia data. The present application can combine multiple first virtual kernel variables and the ad embeddings to connect them into a vector, which can be called a third embedding vector. Exemplarily, assuming that the multiple first virtual kernel variables are [q1, q2, q3] and the ad embedding is D, the third embedding vector is (q1D, q2D, q3D).

[0081] Step 204: Determine the data to be recommended from the plurality of target multimedia data according to the first embedding vector and the third embedding vector.

[0082] In one or more embodiments, the embedding vector generated for each target multimedia data in the Item tower can be connected with multiple first virtual core variables to generate a third embedding vector. The embodiment of the present application can connect the first embedding vector with the third embedding vector corresponding to each target multimedia data in the multiple target multimedia data. That is, for multiple target multimedia data in the database, the first object data needs to be input into the first model with each target multimedia data in the multiple target multimedia data to output the first embedding vector and the third embedding vector, and then determine the difference between the first embedding vector corresponding to each target multimedia data and the third embedding vector of the target multimedia data, and select the target multimedia data that meets the conditions from the multiple target multimedia data as the data to be recommended.

[0083] Step 205: Output the data to be recommended.

[0084] In one or more embodiments, after the data to be recommended is determined, the data to be recommended can be output and displayed to the user. When the multimedia data is a video, the data to be recommended can be a video thumbnail or link on the page, when the multimedia data is text, the data to be recommended can be a text theme link on the page, and when the multimedia data is an advertisement, the data to be recommended can be an advertisement link on the page. The data to be recommended can be pushed directly to the first object, or the data to be recommended can be filtered and sorted before being pushed to the first object. Among them, when the number of data to be recommended is multiple, the first object is selected from the multiple data to be recommended as the one that has been contacted.

[0085] In an embodiment of the present application, after receiving a recommendation request, the first object data of the first object to be recommended is input into the first model to output the first embedding vector of the first object, wherein the first model can extract features from the first object data based on a plurality of first virtual kernel variables to obtain a plurality of second embedding vectors, and then connect the plurality of second embedding vectors to generate a first embedding vector, wherein the plurality of first virtual kernel variables are generated by training to guide the fusion of the target object side embedding vector based on the multimedia side embedding vector. Among them, the first model uses a teacher network for knowledge distillation learning, and the teacher network is generated by training using target object data, multimedia data, and cross data. For multiple target multimedia data in the database, according to the similarity between the first embedding vector and the third embedding vector, the data to be recommended can be determined from the multiple target multimedia data, and the data to be recommended can be output, and the third embedding vector is generated by connecting the embedding vector obtained by feature extraction of the target multimedia data with a plurality of first virtual kernel variables. Through the above method, the teacher network is used to perform knowledge distillation to update the parameters of the first model, so that the features extracted by the parameters of the first model are more in line with the multimedia side features. The multiple virtual kernel variables generated by the training of the target object side embedding vector fusion guidance based on the multimedia side embedding vector affect the generation of the target object side embedding vector, as well as the matching process between the target object side embedding vector and the multimedia side embedding vector, further improving the fit of the features on the multimedia side and the target object side, and improving the recommendation effect of the model.

[0086] Optionally, in the above Figure 2 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiment of the present application, before extracting features from the first object data according to the first model, the method further includes:

[0087] Acquire target object data and multimedia data;

[0088] Extract features from the target object data according to the second model to obtain a plurality of fourth embedding vectors;

[0089] Extract features from the multimedia data according to the second model to obtain a fifth embedding vector;

[0090] weightedly fuse the plurality of fourth embedding vectors according to a first weight to obtain a sixth embedding vector, wherein the first weight is indicated by the fifth embedding vector;

[0091] Training the second model according to the sixth embedding vector and the fifth embedding vector to obtain a first model;

[0092] Get cross-data;

[0093] Train the teacher network based on target object data, multimedia data, and cross-data;

[0094] The parameters of the teacher network and the first model are updated based on the loss of the teacher network, the loss of the first model, and the distillation loss.

[0095] In one or more embodiments, before using the first model to recommend target multimedia data for the first object, it is necessary to train and generate the first model. Sample data can be determined first, and the sample data includes target object data and multimedia data. The second model can be used to extract the features of the target object data and map them into vectors, and a plurality of fourth embedding vectors with different expressions generated based on different variables can be extracted by the second model and mapped into a fifth embedding vector. The fifth embedding vector can then be used to guide the weighted fusion of the plurality of fourth embedding vectors, that is, the weight of each fourth embedding vector in the plurality of fourth embedding vectors can be determined based on the fifth embedding vector, that is, the first weight of the plurality of fourth embedding vectors can be determined, so that the sixth embedding vector generated by weighted fusion of the plurality of fourth embedding vectors according to the second weight can be fitted with the features of the fifth embedding vector, and then the second model can be trained based on the fifth embedding vector and the sixth embedding vector, so as to obtain the first model. In the embodiment of the present application, the teacher network is also trained synchronously when training the first model, the cross data of the target object data and the multimedia data is obtained, and the teacher network is trained in combination with the target object data and the multimedia data, wherein the form of the cross data can be an interval cross or a non-interval cross, etc., which is not limited here. We use the distillation learning method to let the prediction results of the first model mentioned above learn the prediction results of the teacher network. The loss of this learning operation is the distillation loss. Then the parameters of the teacher network and the first model can be updated based on the loss of the teacher network, the loss of the first model and the distillation loss. Exemplarily, assuming that the kl divergence is used to calculate the distillation loss, this part of the loss can be recorded as kl_loss, and the loss between the prediction results of the teacher network and the sample is recorded as teacher_loss. The loss of the first model is student_loss, and the final loss of the model is m*student_loss+(1-m)*(kl_loss+teacher_loss), where m is a model hyperparameter between 0 and 1, and multiple attempts are required to select the appropriate parameters.

[0096] In an embodiment of the present application, a first model training method is provided. Through the above method, the weighted fusion of the embedding vector of the user side data is guided according to the embedding vector of the multimedia side data, thereby improving the fit between the embedding vector of the user side and the embedding vector of the multimedia side, and then the second model is trained according to the similarity between the embedding vector of the user side and the embedding vector of the multimedia side, so that the user side feature extraction performance of the first model is close to the multimedia side, thereby improving the effect of the first model.

[0097] Optionally, in the above Figure 2On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiment of the present application, feature extraction is performed on the target object data according to the second model to obtain a plurality of fourth embedding vectors, including:

[0098] Extract features from the target object data through the first embedding layer of the second model to obtain a seventh embedding vector;

[0099] The seventh embedding vector is input into a plurality of first expert networks of the second model to output a plurality of fourth embedding vectors.

[0100] In one or more embodiments, the object tower is provided with a first embedding layer, which can map the data field of the id type of the feature of the target object data into a vector, and the embedding vector can be 64-dimensional or 32-dimensional, etc. For continuous features, such as the amount of consumption, the discretization is performed by bucketing to obtain a discrete id feature and map it to a 64-dimensional embedding vector. The present application can connect the target object embedding vectors together in a certain order to obtain a feature with a larger dimension. For example, assuming that feature A is represented as [0, 0, 1] and feature B is represented as [0, 1, 1], then in the order of feature A first and feature B later, the two features are spliced ​​to obtain the spliced ​​feature C: [0, 0, 1, 0, 1, 1]. After connecting the target object embedding vectors, a long vector of n*64 dimensions can be obtained, where n is the number of features of the target object data. The n*64-dimensional long vector is the seventh embedding vector. The embodiment of the present application is configured with multiple first expert networks, such as multiple virtual-kernel experts (VKE). A VKE only needs to focus on modeling a certain aspect of user preferences and accurately describe the user representation in this aspect. All VKEs share the same feature input layer, that is, after the seventh embedding vector is input into multiple VKEs, multiple fourth embedding vectors describing different aspects can be output.

[0101] For example, see Figure 4 The structural diagram of the first expert network shown in the figure shows that a learnable variable is provided inside each VKE: Virtual Kernel (also referred to as Virtual Kernel variable in the embodiment of the present application), and an attention calculation layer is designed to determine the combination of features. Key and Value are both user feature embeddings, and Query is a virtual kernel vector. The parameters calculated based on Query and Key (matrix multiplication (MatMul) and normalization (SoftMax)) are used for weighted concatenation of Value, and are input into the model network f after weighting. u(·) Different importance weights are assigned to different feature inputs for fusion, so that the expert network can focus on modeling user preferences in a certain sub-field.

[0102] Correspondingly, after extracting features from multimedia data and mapping them into embedding vectors corresponding to each feature, the embedding vectors of each feature can be concatenated together to obtain a long vector of m*64 dimensions, where m is the number of features of multimedia data. The m*64-dimensional long vector is passed through the expert network of Item to generate the fifth embedding vector.

[0103] For example, for an online application of a trained first model, the first model may generate a third embedding vector ad embedding E offline. t And the attention of k Virtual-Kernels after training [ω1, ω2, …, ω K 】, based on the users you want to recommend, you can output the corresponding K user vectors based on the k Virtual-Kernels Then the first embedding vector is The third embedding vector is E t The connection with k Virtual-Kernels, i.e. the third embedding vector but That is, we can offline Write the vector engine and put the user part network online. The user part network can obtain the user side vector based on the user side features. Then according to this Recall the topN Ad vectors.

[0104] In an embodiment of the present application, a method for generating an embedding vector is provided. Through the above method, the seventh embedding vector of the same input is processed according to multiple first expert networks, and multiple fourth embedding vectors are generated accordingly, based on multiple aspects of user preference expression, to improve the diversity of the characteristics of the target object data.

[0105] Optionally, in the above Figure 2 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiment of the present application, before weighted fusion of multiple fourth embedding vectors according to the first weight to obtain the sixth embedding vector, the method further includes:

[0106] Matrix multiplication and normalization are performed on the fifth embedding vector according to the second virtual kernel variable of each first expert network in the multiple first expert networks to obtain a first weight.

[0107] In one or more embodiments, multiple first expert networks have virtual kernel variables, and each first expert network has different virtual kernel variables. Before weighted fusion of multiple fourth embedding vectors, multiple second virtual kernel variables and fifth embedding vectors may be matrix multiplied and normalized to obtain first weights. Among them, a virtual-kernel gate (VKG) may be used to weight multiple fourth embedding vectors. VKG is a weighted threshold network based on an attention mechanism, which is used to combine multi-faceted user expressions into a final complete user expression. This weighting is completed by selecting VKG under the guidance of the fifth embedding vector.

[0108] For example, see Figure 5 As shown in the VKG weighted schematic diagram, VKG can selectively weightedly fuse the outputs of multiple VKEs according to the input of the multimedia tower to form an appropriate representation vector of the target object under the multimedia feature. This is also calculated based on the attention mechanism. Key is the virtual core of all first expert networks, Query is the representation vector output by the multimedia tower, and Value is the output representation vector of all VKE. Query and Key are first matrix multiplied and then normalized to generate the first weight to perform weighted summation of Value to generate the final target object representation vector under the current multimedia data.

[0109] Based on VKE and VKG, Figure 3 The architecture of the first model and the teacher network can be referred to Figure 6 As shown, the user tower of the first model can generate a user-side feature embedding vector user features embedding according to the user input user-input, and the user features embedding can be processed by VKE1 and VKE2, wherein VKE1 uses virtual-kernel1 and VKE2 uses virtual-kernel2. The advertising tower generates an advertising feature embedding vector ad features embedding according to the advertising input ad-input, and the ad features embedding is processed by a neural network (expertg t(·)) is processed to generate ad embedding. The outputs of VKE1 and VKE2 can be weighted fused in VKG, where adembedding guides the weighted weights of the outputs of VKE1 and VKE2. VKG finally outputs user embedding and calculates loss with adembedding. The input of the teacher network is the cross-features of user-input, ad-input and user-advertisement. By training the teacher network with the cross-features of user-input, ad-input and user-advertisement, a teacher network with higher accuracy than the first model can be obtained. The loss of the teacher network is then calculated through the logit model. Finally, the parameters of the teacher network and the first model can be updated based on the loss of the teacher network, the loss of the first model and the distillation loss.

[0110] The part of the last online real-time prediction in the embodiment of this application can refer to Figure 7 As shown, in Figure 6 After the training is completed, Figure 7 The part in the thick dashed box is the real-time prediction part used for recommendation.

[0111] In the embodiment of the present application, a weight calculation method is provided. Through the above method, the fifth embedding vector on the multimedia side is used to guide the weighted sum of multiple fourth embedding vectors on the target object side, thereby improving the fit of the features on the multimedia side and the target object side and improving the recommendation effect of the model.

[0112] Optionally, in the above Figure 2 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiment of the present application, acquiring the target object data and the multimedia data includes:

[0113] Based on the multimedia selection of the target object, obtaining target object data of the target object;

[0114] The multimedia selection data of the target object within a preset time range is obtained, where the multimedia data includes the multimedia selection data.

[0115] In one or more embodiments, when each target object determines a multimedia selection, based on the multimedia selection, the target object data of the target object that selects the multimedia object will be extracted. The target object data may include user gender, user age, user activity level, user ID, etc. The embodiment of the present application may extract the multimedia selection data operated by the target object within a preset time range based on the user ID corresponding to the multimedia selection and the target object corresponding to the user ID, such as extracting the user's article reading records, advertisement click records, advertisement conversion records, and other behavioral characteristics in the past 14 days. In addition to the multimedia data selected by the user, the multimedia data may also include the multimedia data not selected by the user.

[0116] In an embodiment of the present application, a sample collection method is provided. Through the above method, multimedia selection data of the user within a preset time range is collected based on the user's multimedia selection, and multimedia data corresponding to a single user is directly obtained, without extracting user identification classification from multimedia data in the database, thereby speeding up sample collection.

[0117] Optionally, in the above Figure 2 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiment of the present application, after obtaining the multimedia selection data of the target object within a preset time range, the method further includes:

[0118] Use multimedia selection data as positive samples;

[0119] Get random multimedia targets in the database as negative samples, and the multimedia data includes positive samples and negative samples.

[0120] In one or more embodiments, a positive sample refers to sample data in a business scenario that can be determined to belong to a preset positive category, and a negative sample refers to sample data used as background. If the multimedia selection data is the multimedia data actually selected by the user, the multimedia selection data can be regarded as a positive sample, and a negative sample can be a multimedia target randomly selected from the database.

[0121] In the embodiment of the present application, a sample selection method is provided. Through the above method, positive samples and negative samples are used at the same time, so that these background samples that are not selected as targets can also participate in the model training process. Accordingly, during the model training process, the trained model can learn the key information contained in such background samples, thereby improving the model performance of the trained model, so that the model can more comprehensively and accurately identify various input data.

[0122] Optionally, in the above Figure 2On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiment of the present application, before updating the parameters of the teacher network and the first model based on the loss of the teacher network, the loss of the first model and the distillation loss, the method further includes:

[0123] The loss of the first model is determined according to a loss function, wherein a temperature hyperparameter of the loss function is lower than a preset value.

[0124] In one or more embodiments, when calculating the loss of the first model, the effect of the model can be affected by adjusting the temperature hyperparameter of the loss function. If the temperature hyperparameter is set too large, the model will treat all negative samples as equal, resulting in no distinction between the important and the unimportant in model learning. If the temperature hyperparameter is set too small, the model will pay special attention to difficult negative samples. Since difficult negative samples may also be potential positive samples, this will make it difficult for the model to converge or have poor generalization ability. Therefore, if the present application hopes to make the neural network pay more attention to difficult negative samples, a preset value can be set to adjust the temperature hyperparameter of the loss function below the preset value. In order to avoid poor generalization ability of the model, a minimum value can also be set, and the temperature hyperparameter cannot be lower than the minimum value. The loss function can use infoNCE. infoNCE believes that if the problem is only regarded as a binary classification, such as only data samples and noise samples, it may be unfriendly to model learning, because many noise samples may not be a class at all. Therefore, it is more reasonable to regard it as a multi-classification problem.

[0125] In an embodiment of the present application, a method for calculating model loss is provided. By lowering the temperature hyperparameter of the loss function, the model pays more attention to difficult negative samples, thereby improving the recommendation effect of the model.

[0126] Optionally, in the above Figure 2 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiment of the present application, feature extraction is performed on the multimedia data according to the second model to obtain a fifth embedding vector, which includes:

[0127] Dividing the multimedia data into a plurality of data sets, the plurality of data sets including a training set;

[0128] Feature extraction is performed on the training set according to the second model to obtain a fifth embedding vector.

[0129] In one or more embodiments, the set of target object data and multimedia data in the sample is divided into multiple data sets, and some or all of the multiple data sets can be selected as training sets for training, and only feature extraction is required for the data in the training sets. It can be understood that the sample includes target object data and multimedia data, and each of the divided multiple data sets includes target object data and multimedia data.

[0130] In the embodiment of the present application, a sample training method is provided. By using the above method, only part of the data is selected to participate in the model training, which can save computing resources.

[0131] Optionally, in the above Figure 2 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiment of the present application, the multiple data sets also include a test set, and the method further includes:

[0132] The first model is verified using the test set.

[0133] In one or more embodiments, when a part of the data sets are selected from multiple data sets as training sets, data sets other than the training sets may also be selected as test sets, and the model may be tested using the test sets to verify the accuracy, recall, F1 value, and other indicators of the model. The ratio of the training set to the test set in the multiple data sets may be 7:3, 8:2, or 9:1, etc., which is not limited here.

[0134] In the embodiment of the present application, a method for testing a model is provided. Through the above implementation, the model is verified by dividing the test set from the sample data. Since the test set and the training set are different data sets from the same source, the verification accuracy can be improved.

[0135] Optionally, in the above Figure 2 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiment of the present application, the method further includes:

[0136] One of the multiple data sets is determined as a test set in turn, and the data sets other than the test set are used as training sets.

[0137] In one or more embodiments, a cross-validation algorithm may be used to evaluate the model to ensure the robustness of the model. For multiple data sets, each data set may be used as a test set in turn, and the remaining data sets may be used as training sets for model training, and then the test set may be used to verify the model. Assuming that the multiple data sets are k data sets, the average value of the k verification results may be calculated as the final model accuracy, recall rate, F1 value and other indicators.

[0138] In the embodiment of the present application, a method for testing a model is provided. Through the above implementation, the accuracy of the verification can be improved through a cross-validation algorithm.

[0139] Optionally, in the above Figure 2 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiment of the present application, determining the data to be recommended from the plurality of target multimedia data according to the first embedding vector and the third embedding vector includes:

[0140] Determine a characteristic distance between a first embedding vector corresponding to each target multimedia data and a third embedding vector of the target multimedia data;

[0141] The target multimedia data corresponding to the feature distance less than the preset threshold is used as the data to be recommended.

[0142] In one or more embodiments, for multiple target multimedia data in the database, the feature distance is the difference between the first embedding vector generated according to each target multimedia data and the third embedding vector of the target multimedia data. The third embedding vectors of different target multimedia data guide the model to weight different first embedding vectors based on the same object data. It is necessary to calculate the feature distance for the first embedding vector and the third embedding vector corresponding to a single target multimedia data. The larger the feature distance, the greater the difference between the target multimedia data and the object's selection habits. The smaller the feature distance, the smaller the difference between the target multimedia data and the object's selection habits. Then, the embodiment of the present application can set a preset threshold, which is the minimum requirement for satisfying the object's selection habits. The target multimedia data corresponding to the feature distance greater than the preset threshold is the multimedia data that does not satisfy the object's selection habits, and the target multimedia data corresponding to the feature distance less than the preset threshold is the multimedia data that satisfies the object's selection habits. The present application can select all or part of the target multimedia data corresponding to the feature distance less than the preset threshold as the data to be recommended to the first object. The selected target multimedia data can also be sorted to generate a multimedia data recommendation table, and the generated multimedia data recommendation table is pushed to the first object.

[0143] In the embodiment of the present application, a multimedia data recommendation method is provided. Through the above implementation, the feature distance between the multimedia side feature and the target object side feature is calculated, and the difference between the multimedia data and the user's selection habits is indicated by the feature distance, so that the target multimedia data that meets the user's selection habits can be intuitively and accurately screened, thereby accelerating the speed and accuracy of multimedia data recommendation.

[0144] The multimedia data recommendation device in this application is described in detail below. Figure 8 , Figure 8 This is a schematic diagram of an embodiment of a multimedia data recommendation device in an embodiment of the present application. The multimedia data recommendation device 80 includes:

[0145] An acquiring unit 801 is configured to acquire a recommendation request, where the recommendation request includes first object data of a first object;

[0146] An extraction unit 802 is used to perform feature extraction on the first object data according to the first model to obtain a first embedding vector of the first object, where the first embedding vector is generated by connecting multiple second embedding vectors, the multiple second embedding vectors are generated by the first object data through multiple first virtual kernel variables in the first model, the multiple first virtual kernel variables are generated by training the multimedia side embedding vector and the target object side embedding vector, the target object side embedding vector is generated by weighted fusion based on the guidance of the multimedia side embedding vector, the first model is obtained by performing knowledge distillation using a teacher network, the training data of the teacher network is the target object data, the multimedia data and the cross data, and the cross data is the data after the target object data and the multimedia data features are combined;

[0147] The acquisition unit 801 is further used to acquire a third embedding vector corresponding to the target multimedia data, where the third embedding vector is generated by connecting an embedding vector obtained by performing feature extraction on the target multimedia data with a plurality of first virtual kernel variables;

[0148] A determining unit 803, configured to determine data to be recommended from a plurality of target multimedia data according to the first embedding vector and the third embedding vector;

[0149] The output unit 804 is used to output the data to be recommended.

[0150] In an embodiment of the present application, a multimedia data recommendation device is provided. The above device is adopted, and the teacher network is used for knowledge distillation to update the parameters of the first model, so that the features extracted by the parameters of the first model are more in line with the multimedia side features, and multiple virtual kernel variables are guided by the embedded vectors on the multimedia side during the training process, and then the generation of the embedded vectors on the target object side, and the matching process of the embedded vectors on the target object side and the embedded vectors on the multimedia side, not only the highest-level features interact, but also further improve the fit of the features on the multimedia side and the target object side, and improve the recommendation effect of the model.

[0151] Optionally, in the above Figure 8 On the basis of the corresponding embodiment, in another embodiment of the multimedia data recommendation device 80 provided in the embodiment of the present application, the acquisition unit 801 is further used for:

[0152] Acquire target object data and multimedia data;

[0153] The extraction unit 802 is further configured to perform feature extraction on the target object data according to the second model to obtain a plurality of fourth embedding vectors; perform feature extraction on the multimedia data according to the second model to obtain a fifth embedding vector;

[0154] The device 80 further includes a weighting unit 805, which is specifically configured to perform weighted fusion on the plurality of fourth embedding vectors according to a first weight to obtain a sixth embedding vector, wherein the first weight is indicated by the fifth embedding vector;

[0155] The device 80 further includes a training unit 806, which is specifically configured to train the second model according to the sixth embedding vector and the fifth embedding vector to obtain the first model;

[0156] The acquisition unit 801 is further used to acquire cross data;

[0157] The training unit 806 is also used to train the teacher network according to the target object data, the multimedia data and the cross data; and update the parameters of the teacher network and the first model based on the loss of the teacher network, the loss of the first model and the distillation loss.

[0158] In an embodiment of the present application, a multimedia data recommendation device is provided. By using the above device, the weighted fusion of the embedding vector of the user side data is guided according to the embedding vector of the multimedia side data, thereby improving the fit between the embedding vector of the user side and the embedding vector of the multimedia side, and then training the second model according to the similarity between the embedding vector of the user side and the embedding vector of the multimedia side, so that the user side feature extraction performance of the first model is close to the multimedia side, thereby improving the effect of the first model.

[0159] Optionally, in the above Figure 8 On the basis of the corresponding embodiment, in another embodiment of the multimedia data recommendation device 80 provided in the embodiment of the present application, the extraction unit 802 is specifically used for:

[0160] Extract features from the target object data through the first embedding layer of the second model to obtain a seventh embedding vector;

[0161] The seventh embedding vector is input into a plurality of first expert networks of the second model to output a plurality of fourth embedding vectors.

[0162] In an embodiment of the present application, a multimedia data recommendation device is provided. Using the above device, a plurality of first expert networks are used to process the seventh embedding vector of the same input, and a plurality of fourth embedding vectors are generated accordingly, so as to improve the diversity of the characteristics of the target object data based on the user preference expressions in various aspects.

[0163] Optionally, in the above Figure 8 On the basis of the corresponding embodiment, in another embodiment of the multimedia data recommendation device 80 provided in the embodiment of the present application, the weighting unit 805 is further used for:

[0164] Matrix multiplication and normalization are performed on the fifth embedding vector according to the second virtual kernel variable of each first expert network in the multiple first expert networks to obtain a first weight.

[0165] In an embodiment of the present application, a multimedia data recommendation device is provided. By using the above device, the fifth embedding vector on the multimedia side is used to guide the weighted sum of multiple fourth embedding vectors on the target object side, thereby improving the fit of the features on the multimedia side and the target object side and improving the recommendation effect of the model.

[0166] Optionally, in the above Figure 8 On the basis of the corresponding embodiment, in another embodiment of the multimedia data recommendation device 80 provided in the embodiment of the present application, the acquisition unit 801 is specifically used for:

[0167] Based on the multimedia selection of the target object, obtaining target object data of the target object;

[0168] The multimedia selection data of the target object within a preset time range is obtained, where the multimedia data includes the multimedia selection data.

[0169] In an embodiment of the present application, a multimedia data recommendation device is provided. The above device is used to collect multimedia selection data of the user within a preset time range based on the user's multimedia selection, directly obtain multimedia data corresponding to a single user, and there is no need to extract user identification classification from multimedia data in the database, thereby speeding up sample collection.

[0170] Optionally, in the above Figure 8 On the basis of the corresponding embodiment, in another embodiment of the multimedia data recommendation device 80 provided in the embodiment of the present application, the acquisition unit 801 is specifically used for:

[0171] Use multimedia selection data as positive samples;

[0172] Get random multimedia targets in the database as negative samples, and the multimedia data includes positive samples and negative samples.

[0173] In an embodiment of the present application, a multimedia data recommendation device is provided. With the above device, positive samples and negative samples are used at the same time, so that these background samples that are not selected as targets can also participate in the model training process. Accordingly, during the model training process, the trained model can learn the key information contained in such background samples, thereby improving the model performance of the trained model, so that the model can more comprehensively and accurately identify various input data.

[0174] Optionally, in the above Figure 8 On the basis of the corresponding embodiment, in another embodiment of the multimedia data recommendation device 80 provided in the embodiment of the present application, the determining unit 803 is further used for:

[0175] The loss of the first model is determined according to a loss function, wherein a temperature hyperparameter of the loss function is lower than a preset value.

[0176] In an embodiment of the present application, a multimedia data recommendation device is provided. By using the above device, the temperature hyperparameter of the loss function is lowered, so that the model pays more attention to difficult negative samples, thereby improving the recommendation effect of the model.

[0177] Optionally, in the above Figure 8 On the basis of the corresponding embodiment, in another embodiment of the multimedia data recommendation device 80 provided in the embodiment of the present application, the extraction unit 802 is specifically used for:

[0178] Dividing the multimedia data into a plurality of data sets, the plurality of data sets including a training set;

[0179] Feature extraction is performed on the training set according to the second model to obtain a fifth embedding vector.

[0180] In an embodiment of the present application, a multimedia data recommendation device is provided. By using the above device, only part of the data is selected to participate in model training, which can save computing resources.

[0181] Optionally, in the above Figure 8 On the basis of the corresponding embodiment, in another embodiment of the multimedia data recommendation device 80 provided in the embodiment of the present application, the multiple data sets further include a test set, and the device 80 further includes a verification unit 807, and the verification unit 807 is specifically used to:

[0182] The first model is verified using the test set.

[0183] In an embodiment of the present application, a multimedia data recommendation device is provided. By using the above device, a test set is divided from sample data to verify the model, and since the test set and the training set are different data sets from the same source, the verification accuracy can be improved.

[0184] Optionally, in the above Figure 8 On the basis of the corresponding embodiment, in another embodiment of the multimedia data recommendation device 80 provided in the embodiment of the present application, the verification unit 807 is further used for:

[0185] One of the multiple data sets is determined as a test set in turn, and the data sets other than the test set are used as training sets.

[0186] In an embodiment of the present application, a multimedia data recommendation device is provided. By adopting the above device, the verification accuracy can be improved through a cross-validation algorithm.

[0187] Optionally, in the above Figure 8On the basis of the corresponding embodiment, in another embodiment of the multimedia data recommendation device 80 provided in the embodiment of the present application, the determination unit 803 is specifically used for:

[0188] Determine a characteristic distance between a first embedding vector corresponding to each target multimedia data and a third embedding vector of the target multimedia data;

[0189] The target multimedia data corresponding to the feature distance less than the preset threshold is used as the data to be recommended.

[0190] In an embodiment of the present application, a multimedia data recommendation device is provided. By using the above device, the feature distance between the multimedia side feature and the target object side feature is calculated, and the difference between the multimedia data and the user's selection habits is indicated by the feature distance, so that the target multimedia data that meets the user's selection habits can be intuitively and accurately screened, thereby accelerating the speed and accuracy of multimedia data recommendation.

[0191] Fig. 9 3 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device 300 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 322 (for example, one or more processors) and memory 332, and one or more storage media 330 (for example, one or more mass storage devices) storing application programs 342 or data 344. Among them, the memory 332 and the storage medium 330 can be short-term storage or permanent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the computer device. Furthermore, the central processing unit 322 can be configured to communicate with the storage medium 330 to execute a series of instruction operations in the storage medium 330 on the computer device 300.

[0192] The computer device 300 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input and output interfaces 358, and / or one or more operating systems 341, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0193] The steps performed by the server in the above embodiment can be based on the Fig. 9 The computer device structure shown.

[0194] A computer-readable storage medium is also provided in an embodiment of the present application, on which a computer program is stored. When the computer program is executed by a processor, the steps of the methods described in the above embodiments are implemented.

[0195] A computer program product is also provided in an embodiment of the present application, including a computer program, which, when executed by a processor, implements the steps of the methods described in the above embodiments.

[0196] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0197] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0198] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0199] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0200] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0201] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multimedia data recommendation method, characterized in that: include: Obtaining a recommendation request, the recommendation request including first object data of a first object; Performing feature extraction on the first object data according to the first model to obtain a first embedding vector of the first object, the first embedding vector is generated by connecting multiple second embedding vectors, the multiple second embedding vectors are generated by the first object data through multiple first virtual kernel variables in the first model, the multiple first virtual kernel variables are generated by training the multimedia side embedding vector and the target object side embedding vector, the target object side embedding vector is generated by weighted fusion based on the guidance of the multimedia side embedding vector, the first model is obtained by using a teacher network for knowledge distillation, the training data of the teacher network is the target object data, the multimedia data and the cross data, and the cross data is the data after the target object data and the multimedia data features are combined; Obtaining a third embedding vector corresponding to the target multimedia data, where the third embedding vector is generated by connecting an embedding vector obtained by performing feature extraction on the target multimedia data with the plurality of first virtual kernel variables; Determining data to be recommended from the plurality of target multimedia data according to the first embedding vector and the third embedding vector; The data to be recommended is outputted.

2. The method according to claim 1, characterized in that Before extracting features from the first object data according to the first model, the method further includes: Acquire the target object data and the multimedia data; Performing feature extraction on the target object data according to the second model to obtain a plurality of fourth embedding vectors; Perform feature extraction on the multimedia data according to the second model to obtain a fifth embedding vector; Performing weighted fusion on the plurality of the fourth embedding vectors according to a first weight to obtain a sixth embedding vector, wherein the first weight is indicated by the fifth embedding vector; Training the second model according to the sixth embedding vector and the fifth embedding vector to obtain the first model; Acquire the cross data; training the teacher network based on the target object data, the multimedia data and the cross data; Parameters of the teacher network and the first model are updated based on the loss of the teacher network, the loss of the first model, and the distillation loss.

3. The method according to claim 2, characterized in that The extracting features of the target object data according to the second model to obtain a plurality of fourth embedding vectors comprises: Performing feature extraction on the target object data through the first embedding layer of the second model to obtain a seventh embedding vector; The seventh embedding vector is input into a plurality of first expert networks of the second model to output a plurality of the fourth embedding vectors.

4. The method according to claim 3, characterized in that Before weighted fusion of the plurality of fourth embedding vectors according to the first weight to obtain a sixth embedding vector, the method further includes: Matrix multiplication and normalization are performed on the fifth embedding vector according to the second virtual kernel variable of each of the first expert networks in the plurality of first expert networks to obtain the first weight.

5. The method according to claim 2, characterized in that: The acquiring of target object data and multimedia data comprises: Based on the multimedia selection of the target object, obtaining the target object data of the target object; The multimedia selection data of the target object within a preset time range is acquired, where the multimedia data includes the multimedia selection data.

6. The method according to claim 5, characterized in that After acquiring the multimedia selection data of the target object within a preset time range, the method further includes: Taking the multimedia selection data as positive samples; A random multimedia target in a database is obtained as a negative sample, and the multimedia data includes the positive sample and the negative sample.

7. The method according to claim 6, characterized in that Before updating the parameters of the teacher network and the first model based on the loss of the teacher network, the loss of the first model and the distillation loss, the method further includes: The loss of the first model is determined according to a loss function, wherein a temperature hyperparameter of the loss function is lower than a preset value.

8. The method according to claim 2, characterized in that: The extracting features from the multimedia data according to the second model to obtain a fifth embedding vector comprises: Dividing the multimedia data into a plurality of data sets, the plurality of data sets including a training set; Perform feature extraction on the training set according to the second model to obtain the fifth embedding vector.

9. The method according to claim 8, characterized in that The plurality of data sets also include a test set, and the method further includes: The first model is verified using the test set.

10. The method according to claim 9, characterized in that The method further comprises: One of the multiple data sets is sequentially determined as the test set, and the data sets other than the test set in the multiple data sets are used as the training set.

11. The method according to claim 1, characterized in that: The determining the to-be-recommended data from the plurality of target multimedia data according to the first embedding vector and the third embedding vector comprises: Determine a characteristic distance between the first embedding vector corresponding to each target multimedia data and the third embedding vector of the target multimedia data; The target multimedia data corresponding to the characteristic distance less than a preset threshold is used as the data to be recommended.

12. A multimedia data recommendation device, characterized in that: include: an acquiring unit, configured to acquire a recommendation request, wherein the recommendation request includes first object data of a first object; an extraction unit, configured to perform feature extraction on the first object data according to a first model to obtain a first embedding vector of the first object, wherein the first embedding vector is generated by connecting a plurality of second embedding vectors, the plurality of second embedding vectors are generated by the first object data through a plurality of first virtual kernel variables in the first model, the plurality of first virtual kernel variables are generated by training of a multimedia side embedding vector and a target object side embedding vector, the target object side embedding vector is generated by weighted fusion based on the guidance of the multimedia side embedding vector, the first model is obtained by performing knowledge distillation using a teacher network, the training data of the teacher network is target object data, multimedia data and cross data, and the cross data is data after the target object data and the multimedia data features are combined; The acquisition unit is further used to acquire a third embedding vector corresponding to the target multimedia data, where the third embedding vector is generated by connecting the embedding vector obtained by performing feature extraction on the target multimedia data with the multiple first virtual kernel variables; a determining unit, configured to determine data to be recommended from the plurality of target multimedia data according to the first embedding vector and the third embedding vector; An output unit is used to output the data to be recommended.

13. A computer device, characterized in that: include: memories, transceivers, processors, and bus systems; Wherein, the memory is used to store programs; The processor is used to execute the program in the memory, including executing the method according to any one of claims 1 to 11; The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

14. A computer-readable storage medium comprising instructions, which, when executed on a computer, causes the computer to perform the method according to any one of claims 1 to 11.

15. A computer program product, characterized in that When the computer program product is executed on a computer, the computer performs the method according to any one of claims 1 to 11.

Images