A multi-model fusion knowledge distillation recommendation model

Abstract

The application discloses a multi-model fusion knowledge distillation recommendation model, comprising an ensemble learning module and a student module, the ensemble learning module uses a full connection layer to perform weighted voting on the prediction results of DeepFM models, DIN models and MMDIN models, and obtains a final prediction result, wherein the size of the weighted voting weight is adaptively adjusted by using a gradient descent method, the student module adopts a shallow DIN structure, and a soft label is used to guide the convergence of the student model. The application increases the full connection layer, integrates the advantages of three deep learning models DeepFM, DIN and MMDIN, and updates the parameters of the full connection layer by using the gradient descent method. The parameters of the model are initialized, and the parameter change range is limited, so that the integrated model converges faster and better, and the voting scene of each model is simulated.

Description

Technical Field

[0001] This invention belongs to the technical field of recommendation optimization models, and relates to a multi-model fusion knowledge distillation recommendation model. Background Technology

[0002] As humanity enters the information age, people produce vast amounts of information every moment and spend a significant amount of time browsing the internet. How to help users find information of interest from this massive amount of online data has become a crucial research topic. Similarly, the aerospace equipment manufacturing industry faces the challenge of identifying better upstream service providers from a vast array of services. Aerospace equipment manufacturing cloud service platforms have attracted numerous aerospace equipment manufacturing service providers. How to enable users to obtain information of interest from this vast array of cloud services has become a key research focus for platform developers, and recommendation algorithms are a vital approach to solving this problem.

[0003] In order to enable users to better obtain the information they are interested in, experts and scholars have proposed a variety of methods, from machine learning to deep learning.

[0004] However, there is a lack of research on integrating recommendation models with different advantages to enhance the recommendation effect. The integrated recommendation model has a large number of parameters; the prediction effect of a single recommendation model is poor; the integration of deep learning recommendation models is difficult; the integration method lacks parameter adaptation ability; the integrated model is too large, resulting in excessive model storage space and slow model inference speed; the activation function of the integrated model cannot simulate the model voting scenario, compressing both input and output to [0,1], requiring the design of activation functions for simulation. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a multi-model fusion knowledge distillation recommendation model to solve the technical problems existing in the prior art.

[0006] The technical solution adopted in this invention is as follows: a multi-model fusion knowledge distillation recommendation model, including an ensemble learning module and a student module. The ensemble learning module uses a fully connected layer to perform weighted voting on the prediction results of the DeepFM model, the DIN model, and the MMDIN model to obtain the final prediction result. The weighted voting weights are adaptively adjusted using the gradient descent method. The input and prediction values ​​of the ensemble learning module are transmitted to the student model for training. The student module adopts a shallow DIN structure and uses soft labels to guide the convergence of the student model.

[0007] The parameters of the fully connected layer are initialized and constrained, and the rate of change is set to specify the rate of parameter change. The parameter calculation method is shown in formula (1):

[0008]

[0009] Where V represents the current parameter value, P represents the previous parameter value, and R represents the rate of change. min V represents the minimum value specified by the parameter. max Indicates the maximum value limited by the parameter;

[0010] Meanwhile, a new excitation function is designed, and the calculation method of the excitation function is shown in formula (2):

[0011]

[0012] Where x is the input, y is the output, b is the initial factor, and k is the proportional adjustment coefficient;

[0013] The new activation function ensures that when the domain of the output of each sub-model is [0,1], the output range of the integrated model is also [0,1], which is the final output score range.

[0014] The loss function design for the student module knowledge distillation model is shown in formula (3):

[0015] L sum =L1α+L2(1-α) (3)

[0016] Among them, L sum L1 represents the total loss, L2 represents the loss between the soft value and the model prediction, and α is the distillation coefficient.

[0017] Gradient descent parameter update formula:

[0018] θ=θ-α(h(x)-y)x (4)

[0019] In the formula, θ represents the parameter that needs to be updated; α represents the learning rate, which is a hyperparameter (that needs to be set and adjusted manually); h(x) represents the output of the previous neuron (the expression is learned by the model through fitting data); y represents the true value; and x represents the input.

[0020] The beneficial effects of this invention are as follows: Compared with the prior art, this invention adds a fully connected layer, integrates the advantages of three deep learning models (DeepFM, DIN, and MMDIN), and uses gradient descent to update the parameters of the fully connected layer. The model parameters are initialized, and restrictions are imposed on the range of parameter changes, enabling the integrated model to converge faster and better and simulating a voting scenario among the models. Furthermore, to ensure that the output domain of the integrated model is also [0,1] when the domain of each sub-model's output is [0,1], a new activation function is designed.

[0021] Because the integrated model has a large number of parameters, it requires a large amount of storage space and a long training and prediction time. This invention performs knowledge distillation on the model, which greatly compresses the number of model parameters and reduces the storage space and training time required. Attached Figure Description

[0022] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0023] The present invention will be further described below with reference to specific embodiments.

[0024] Example 1: As Figure 1 As shown, a multi-model fusion knowledge distillation recommendation model mainly consists of a teacher model (i.e., an ensemble learning model, upper part of the figure) and a student model (lower part of the figure). The teacher model mainly integrates three mainstream recommendation algorithms: DIN (left), DeepFM (middle), and MMDIN (right). The student model is implemented using a simple shallow DIN model. The following sections provide a detailed description of each module.

[0025] (a) DIN module

[0026] The advantage of DIN is that it introduces an attention mechanism, avoiding treating all items requiring rating the same way. Its main idea is to use the user's five most recently rated movies or cloud services, respectively, to perform outer products with the current candidate movies, and then weighted pool the results to obtain the user's most recent points of interest. For example... Figure 1 As shown in the model in the top left corner, the sparse features of the original input, non-numerical data, are encoded to form a dense feature vector. Then, the current movie is multiplied by the five most recent movies, and these products are concatenated and activated with PReLU and sigmoid to obtain the similarity between the current movie and the most recently rated movies or cloud services. This similarity is then weighted and pooled across the most recently rated movies. Finally, it is connected to user features and scene features, and fitted using a multi-layer neural network to obtain the final score.

[0027] (b) DeepFM module

[0028] DeepFM improves upon the Wide&Deep model by replacing the wide part with the FM part, allowing features to be combined and cross-referenced before predicting item scores. The DeepFM model structure is as follows: Figure 1The upper middle section is shown. Numerical features can be directly input into the next layer for computation, while non-numerical features are densified to obtain corresponding embedding representations. Some features are directly combined through feature cross-linking and finally connected to a fully connected layer for activation and output; others undergo complex fitting and expression through multi-layer neural networks before finally being connected to the output layer. DeepFM not only ensures that the model has strong memory and generalization capabilities, but also enables the model to predict scores by combining features.

[0029] (c) MMDIN module

[0030] The MMDIN module, based on DIN, incorporates image features of items to predict user ratings and adds a multi-head mechanism, enabling the model to extract features from different dimensions.

[0031] The MMDIN model is mainly composed of a multimodal module, an attention module, and a multilayer neural network module. The multimodal module is primarily responsible for extracting color features from images; the attention module is responsible for extracting items that the user might be interested in from their most recently rated items. The multilayer neural network module mainly uses a Renset structure to alleviate the gradient vanishing problem as the model depth increases, making it less likely that rating predictions will worsen with deeper models. Simultaneously, the multilayer neural network employs a multi-head mechanism, enabling the model to better extract features from data across multiple dimensions.

[0032] (1) Integrated Learning Module

[0033] The ensemble learning module is primarily responsible for integrating the advantages of the DeepFM, DIN, and MMDIN models, resulting in better prediction performance and a more robust and stable algorithm. The ensemble learning module uses fully connected layers to perform weighted voting on the prediction results of the three models, obtaining the final prediction result. The weights are adaptively adjusted using gradient descent. To simulate the voting scenario among the models and thus enable faster and better convergence, this invention specifies the initialization and constraints on the parameters of the fully connected layers, and sets a rate of change to define the parameter change rate. The parameter calculation method is shown in Formula 1. Here, V represents the current parameter value, P represents the previous parameter value, and R represents the rate of change. min V represents the minimum value specified by the parameter. max This indicates the maximum value that the parameter is limited to.

[0034]

[0035] Meanwhile, in order to ensure that the output domain of the integrated model is also [0,1] when the domain of each sub-model output is [0,1], which is the final output score range and more in line with the actual changing scenarios, we studied and designed a new activation function. The activation function calculation method is shown in Formula 2.

[0036]

[0037] Where x is the input, y is the output, b is the initial factor, and k is the proportional adjustment coefficient. The closer the curve is to y = x, the better the activation effect of the activation function.

[0038] Gradient descent parameter update formula:

[0039] θ=θ-α(h(x)-y)x (4)

[0040] In the formula, θ represents the parameter that needs to be updated; α represents the learning rate, which is a hyperparameter (that needs to be set and adjusted manually); h(x) represents the output of the previous neuron (the expression is learned by the model through fitting data); y represents the true value; and x represents the input.

[0041] (2) Student Module:

[0042] The student module is primarily implemented using a simple shallow DIN structure. The advantage of knowledge distillation lies in using soft labels to guide the convergence of the student model, enabling students to accurately predict samples they have never seen before. The loss function of the distillation model is designed as shown in Equation 3. Where, L... sum L1 represents the total loss, L2 represents the loss between the soft value and the model prediction, and α is the distillation coefficient.

[0043] L sum =L1α+L2(1-α) (3)

[0044] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of protection of the claims.

Claims

1. A multi-model fusion knowledge distillation recommendation model, characterized in that: It includes an ensemble learning module and a student module. The ensemble learning module uses a fully connected layer to perform weighted voting on the prediction results of the DeepFM model, DIN model and MMDIN model to obtain the final prediction result. The weights of the weighted voting are adaptively adjusted using gradient descent. The input and prediction values ​​of the ensemble learning module are transmitted to the student model for training. The student module adopts a shallow DIN structure and uses soft labels to guide the convergence of the student model. The DIN model introduces an attention mechanism: it calculates the user's recent interest points by taking the outer product of the user's five most recently rated movies with the current candidate movie and then weighting and pooling the results. The sparse features of the original input, non-numerical data, are encoded into a dense feature vector. Then, the current movie is outer-producted with the five most recently rated movies, and the resulting vectors are concatenated and activated with PReLU and sigmoid to obtain the similarity between the current movie and the most recently rated movies. This similarity is then weighted and pooled across the most recently rated movies. Finally, it is connected to user features and scene features, and then fitted and represented through a multi-layer neural network to obtain the final score. The MMDIN model, based on the DIN model, incorporates image features of items to predict user ratings and adds a multi-head mechanism, enabling the model to extract features from different dimensions. It consists of a multimodal module, an attention mechanism module, and a multi-layer neural network module. The multimodal module is responsible for extracting color features from images; the attention mechanism module is responsible for extracting items that the user may be interested in from the items the user has recently rated; and the multi-layer neural network module adopts a Renset structure, while also employing a multi-head mechanism within the multi-layer neural network. The parameters of the fully connected layer are initialized and constrained, and the rate of change is set to specify the rate of change of the parameters. The parameter calculation method is shown in formula (1): （1）， Where V represents the current parameter value, P represents the previous parameter value, and R represents the rate of change. min V represents the minimum value specified by the parameter. max Indicates the maximum value limited by the parameter; At the same time, a new excitation function is designed, and the calculation method of the excitation function is shown in formula (2): （2）， Where x is the input, y is the output, b is the initial factor, and k is the proportional adjustment coefficient; The new activation function ensures that when the domain of the output of each sub-model is [0,1], the output domain of the integrated model is also [0,1], which is the final output score range.

2. The multi-model fusion knowledge distillation recommendation model according to claim 1, characterized in that: The loss function design for the student module knowledge distillation model is shown in formula (3): （3）， Among them, L sum L1 represents the total loss, L2 represents the loss between the soft value and the model prediction, and α is the distillation coefficient.

3. The multi-model fusion knowledge distillation recommendation model according to claim 1, characterized in that: Gradient descent parameter update formula: （4）， In the formula, θ represents the parameter that needs to be updated; α represents the learning rate, which is a hyperparameter; h(x) represents the output of the previous neuron; y represents the true value; and x represents the input.

Images