A multimedia generalization recommendation method based on invariant learning
By constructing a multimedia generalization recommendation method based on invariant learning, the problem of incomplete information in multimedia recommendation under the cold start problem is solved, and more accurate and efficient recommendation results are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HEFEI UNIV OF TECH
- Filing Date
- 2023-10-30
- Publication Date
- 2026-04-28
AI Technical Summary
Existing multimedia recommendation methods struggle to provide accurate recommendations in scenarios with incomplete information when facing the cold start problem, especially when new users or new products lack historical data.
We adopt a multimedia generalization recommendation method based on invariant learning. By constructing heterogeneous data, building a cold-start recommendation network, performing representation generation and modality fusion, and utilizing alignment loss and invariant learning, we optimize the generalization ability of the recommendation model.
It improves the accuracy and efficiency of recommendations in scenarios with incomplete information, maintains the stability of recommendation results when some modal information is missing, and enhances the generalization ability of the model.
Smart Images
Figure CN117349457B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of multimedia recommendation and cold start, specifically a multimedia generalization recommendation method based on invariant learning. Background Technology
[0002] Recommendation methods are an important information filtering technique designed to recommend products or content that users may be interested in. However, recommendation methods become particularly challenging when faced with the cold start problem. The cold start problem refers to the difficulty in providing accurate recommendations to new users or new products when there is a lack of sufficient user history data or product feature information.
[0003] Multimedia data, such as images, audio, video, and related content, is widely distributed across various information distribution channels, including the internet, social media platforms, digital media libraries, and online stores. Multimedia recommendation leverages rich multimedia features to model and represent new users and products, effectively mitigating the cold start problem in recommendation methods. However, multimedia information exhibits significant diversity across different distribution channels. This multimedia data can originate from various sources, formats, and quality standards. Furthermore, the distribution of multimedia information spans multiple domains, including social media, news, entertainment, and e-commerce, each with its specific content and product characteristics. In real-world scenarios, it is often impossible to collect all multimedia information for recommendation. Therefore, multimedia recommendation methods are frequently limited by the problem of incomplete information.
[0004] Despite efforts to address the cold start problem and the generalization of multimedia information, current methods struggle to generalize to scenarios with incomplete multimedia information. Therefore, a new approach is needed to overcome these challenges. Summary of the Invention
[0005] To address the shortcomings of existing technologies, this invention proposes a multimedia generalization recommendation method based on invariant learning, aiming to alleviate the performance issues of cold-start recommendation in scenarios with incomplete information. The method enhances its generalization ability based on alignment and invariant learning, thereby improving the accuracy and efficiency of cold-start recommendation.
[0006] The present invention adopts the following technical solution to solve the technical problem:
[0007] The multimedia generalization recommendation method based on invariant learning in this invention is characterized by the following steps:
[0008] Step 1: Construct heterogeneous data, including: constructing a user-product interaction matrix using user interaction records, and extracting multimedia features from the product's multimedia content.
[0009] Step 1.1: Let U represent the user set, N u Let N represent the total number of users; let V represent the product set, and N represent the total number of users. v Let V represent the total number of products; new N represents a new product set with no interaction history. new This indicates the total number of products in the new product cluster;
[0010] make Represents the user-product interaction matrix, where R uj Let R represent whether the u-th user has interacted with the j-th product. If the u-th user has interacted with the j-th product, then let R... uj =1, otherwise, let R = 1. uj =0;
[0011] Step 1.2, Let S = {S} 1 S m S M} represents the multimedia content of product set V, where S m Let M represent the set of multimedia content of the m-th modality in product set V, where M represents the total number of modalities, and 1≤m≤M;
[0012] Each modality's feature extraction model is used to extract features from the corresponding multimedia content, resulting in the original multimedia feature set X = {X...} 1 , ..., X m , ..., X M},in, Let represent the original feature matrix of the m-th modality of product set V, and in, d represents the original multimedia feature of the m-th modality of the j-th product. xm The dimension of the original features of multimedia;
[0013] Step 2: Construct a cold-start recommendation network, including a representation generator. Embedded layer User collaborative representation matrix P;
[0014] Step 3, the characterization generator The original features of the multimedia are processed to obtain the representation vectors of each modality in the product set V;
[0015] Step 3.1: Calculate the representation vector of the m-th mode of the j-th product according to equation (1). This yields the representation matrix of the m-th mode of product set V. This leads to the representation set C = {C} of each modality in the product set V. 1 C m C M}:
[0016]
[0017] In equation (1), Represents the representation generator for the m-th mode;
[0018] Step 3.2: Align the representation matrices of any two modes in the product set V according to equation (2) to obtain the alignment loss.
[0019]
[0020] In equation (2), C m′ The representation matrix of the m′-th mode of product set V;
[0021] Step 4, Embedding Layer Initialize user collaborative representation matrix Where, p u Let p represent the representation vector of the u-th user, and then according to equation (3) u After encoding, the final representation vector z of the u-th user is obtained. u :
[0022]
[0023] Step 5: Construct weight sets for modal fusion under M different environments using a cyclic mixing layer, which are used to generate representations of differentiated environments and optimize invariant representations;
[0024] Step 5.1: According to equation (4), sample a set of weights of dimension M from the Dirichlet distribution:
[0025]
[0026] In equation (5), Θ1 represents the set of weights for modal fusion of the first constructed environment. Let α represent the weight of the m-th mode in the first environment. m The m-th parameter used to adjust the Dirichlet distribution; ~ indicates that it follows;
[0027] Step 5.2, in Θ e-1 Based on this, the weight set Θ for modal fusion of the e-th environment is obtained according to equation (5). e :
[0028] Θ e =circle_shift(Θ e-1 (5)
[0029] In equation (6), circle_shift represents the cyclic shift operation; Θ e-1 This represents the result after the (e-2)th cycle displacement. When e = 2, let Θ e-1 =Θ1;
[0030] Step 5.3: Perform the process of step 5.2 M-1 times to obtain the weight set Θ = {Θ1, ..., Θ2} for modal fusion under M different environments. e ,...,Θ M};
[0031] Step 5.4: Process the representation vectors of the M modes of the j-th product according to equation (6) to generate the final representation vector z of the j-th product in the e-th environment. j，e :
[0032]
[0033] Step 6: Recommended loss based on different environments Calculate the invariant loss function
[0034] Step 6.1: Calculate the preference score of the u-th user for the j-th product in the e-th environment according to formula (7).
[0035]
[0036] In equation (7), A function representing a fractional tendency;
[0037] Step 6.2: Establish the recommended BPR loss for the e-th environment according to equation (8).
[0038]
[0039] In equation (8), R + Let j' represent the set of interactions with all products that all users have interacted with, and j' represent any product that the u-th user has not interacted with. Let represent the set of products interacted by the u-th user, σ(·) represent the activation function; (u, j) represent the interaction item of the j-th product that the u-th user has interacted with; This represents the preference score of the u-th user in the e-th environment for the j′-th product;
[0040] Step 6.3: Construct the cross-environment invariant loss function according to equation (9).
[0041]
[0042] In equation (9), Let Var represent the expected value, β represent the variance, and β represent the weights learned invariantly.
[0043] Step 7: Combine the various loss functions and perform multi-task learning on the recommendation product network to update the network parameters until convergence:
[0044] Step 7.1: Establish a multi-task optimization objective based on equation (10).
[0045]
[0046] In equation (10), These are all the parameters to be optimized, and λ represents the weight of the alignment loss.
[0047] Step 7.2: Train the recommendation product network using gradient descent and calculate the multi-task optimization objective L(ψ) to update the parameters ψ until... The optimal parameter ψ is obtained by continuing until the convergence condition is met. * The product recommendation model is used to implement the new product set V new Recommendation.
[0048] The present invention provides an electronic device, including a memory and a processor, wherein the memory is used to store a program that supports the processor in executing the multimedia generalization recommendation method, and the processor is configured to execute the program stored in the memory.
[0049] The present invention discloses a computer-readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, performs the steps of the multimedia generalization recommendation method.
[0050] Compared with existing technologies, the beneficial effects of this invention are reflected in:
[0051] 1. This invention uses independent generators to generate modal representations based on the original features of each modality. This independent design ensures that each generator is only associated with a specific modality, thereby mitigating the adverse effects on extracting information from the current modality when other modalities are unavailable.
[0052] 2. This invention achieves cross-modal alignment. By maximizing the mutual information between different modalities, preference information between modalities can be transferred to each other. When a certain modality is unavailable, other modalities can provide a certain degree of supplementation.
[0053] 3. By constructing a differential environment, this invention establishes a model that exhibits good performance when extracting multimedia features from any joint distribution, thereby improving the model's generalization ability and enabling the method to maintain the stability of recommendation results even when some modal information is missing.
[0054] 4. This invention assumes the existence of multiple environments, each with multimedia features from a different number of modalities. It reduces the loss of recommendation accuracy caused by inconsistent modal distributions by learning invariant representations across environments. Attached Figure Description
[0055] Figure 1 This is a flowchart of the cold start recommendation method based on environment construction and invariant characterization of the present invention. Detailed Implementation
[0056] In this embodiment, a multimedia generalization recommendation method based on invariant learning considers the information loss caused by modality unavailability and the diverse distribution of missing modalities. It captures information sharing between modalities and cross-environment invariant representations by introducing alignment loss, context construction, and invariant representation techniques to achieve a more accurate cold-start recommendation method. Specifically, as... Figure 1 As shown, the procedure is as follows:
[0057] Step 1: Construct heterogeneous data, including: constructing a user-product interaction matrix using user interaction records, and extracting multimedia features from the product's multimedia content.
[0058] Step 1.1: Let U denote the user set, and U = {1, ..., u, ..., N} u}, where u represents the u-th user, and N u Let V represent the total number of users; let V represent the product set, and V = {1, ..., j, ..., N}. v}, j represents the j-th product, N v V represents the total number of products; new N represents a new product set with no interaction history. new This indicates the total number of products in the new product cluster;
[0059] make Represents the user-product interaction matrix, where R uj Let R represent whether the u-th user has interacted with the j-th product. If the u-th user has interacted with the j-th product, then let R... uj =1, otherwise, let R = 1. uj =0;
[0060] Step 1.2, Let S = {S} 1 S m S M} represents the multimedia content of product set V, where S m Let M represent the set of multimedia content of the m-th modality in product set V, where M represents the total number of modalities, and 1≤m≤M;
[0061] Each modality's feature extraction model is used to extract features from the corresponding multimedia content, resulting in the original multimedia feature set X = {X...} 1 , ..., X m , ..., X M},in, Let represent the original feature matrix of the m-th modality of product set V, and in, d represents the original multimedia feature of the m-th modality of the j-th product. xm The dimension of the original features of multimedia;
[0062] Step 2: Construct a cold-start recommendation network, including a representation generator. Embedded layer User collaborative representation matrix P;
[0063] Step 3, Characterization Generator The original features of the multimedia are processed to obtain the representation vectors of each modality in the product set V:
[0064] Step 3.1: Calculate the representation vector of the m-th mode of the j-th product according to equation (1). This yields the representation matrix of the m-th mode of product set V. This leads to the representation set C = {C} of each modality in the product set V. 1 C m C M}:
[0065]
[0066] In equation (1), The m-th mode is represented by a characterization generator. This invention uses a linear layer to parameterize the process.
[0067] Step 3.2: Align the representation matrices of any two modes in the product set V according to equation (2) to obtain the alignment loss.
[0068]
[0069] In equation (2), C m′ Let m' be the representation matrix of the m-th mode of product set V.
[0070] Step 4, Embedding Layer Initialize user collaborative representation matrix Where, p u Let p represent the representation vector of the u-th user, and then according to equation (3) u After encoding, the final representation vector z of the u-th user is obtained.u :
[0071]
[0072] Step 5: Construct weight sets for modal fusion under M different environments using a cyclic mixing layer, which are used to generate representations of differentiated environments and optimize invariant representations;
[0073] Step 5.1: According to equation (4), sample a set of weights of dimension M from the Dirichlet distribution:
[0074]
[0075] In equation (5), Θ1 represents the set of weights for modal fusion of the first constructed environment. Let α represent the weight of the m-th mode in the first environment. m The m-th parameter used to adjust the Dirichlet distribution; ~ indicates that it follows;
[0076] Step 5.2, in Θ e-1 Based on this, the weight set Θ for modal fusion of the e-th environment is obtained according to equation (5). e :
[0077] Θ e =circle_shift(Θ e-1 In equation (5) and (6), circle_shift represents the cyclic shift operation, specifically... Θ e-1 Let Θ represent the result after the (e-2)th cycle displacement, where e ≥ 2. When e = 2, let Θ e-1 =Θ1.
[0078] Step 5.3: Perform the process of step 5.2 M-1 times to obtain the weight set Θ = {Θ1, ..., Θ2} for modal fusion under M different environments. e ,...,Θ M};
[0079] Step 5.4: Weight Θ of modal fusion based on the e-th environment of product set V e According to equation (7), the representation vectors of the M modes of the j-th product are processed to generate the final representation vector z of the j-th product in the e-th environment. j，e :
[0080]
[0081] In equation (7), Indicates according to Θ eCalculate the characterization of the j-th product in the e-th environment;
[0082] Step 6: Recommended loss based on different environments Calculate the invariant loss function
[0083] Step 6.1: Calculate the preference score of the u-th user for the j-th product in the e-th environment according to formula (7).
[0084]
[0085] In equation (7), The function representing the propensity score is used. In specific implementation, this invention uses the inner product to calculate the final representation vector of users and products instead of the propensity score function. This is the inner product operator.
[0086] Step 6.2: Establish the recommended BPR loss for the e-th environment according to equation (8).
[0087]
[0088] In equation (8), R + Let j' represent the set of interactions with all products that all users have interacted with, and j' represent any product that the u-th user has not interacted with. Let represent the set of products interacted by the u-th user, σ(·) represent the activation function; (u, j) represent the interaction item of the j-th product that the u-th user has interacted with; Let represent the preference score of the u-th user in the e-th environment for the j′-th product.
[0089] Step 6.3: Construct the cross-environment invariant loss function according to equation (9).
[0090]
[0091] In equation (9), β represents the expected value, Var represents the variance, and β represents the weights for invariant learning.
[0092] Step 7: Combine the various loss functions and perform multi-task learning on the recommendation product network to update the network parameters until convergence:
[0093] Step 7.1: Establish a multi-task optimization objective based on equation (10).
[0094]
[0095] In equation (10), These are all the parameters to be optimized, and λ represents the weight of the alignment loss.
[0096] Step 7.2: Train the recommendation product network using gradient descent and calculate the multi-task optimization objective L(ψ) to update the parameters ψ until... The optimal parameter ψ is obtained by continuing until the convergence condition is met. * The product recommendation model is used to implement the new product set V new Recommendation.
[0097] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described method, and the processor is configured to execute the program stored in the memory.
[0098] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.
[0099] Example:
[0100] To verify the effectiveness of this method, this invention uses Amazon-Baby, a commonly used publicly available dataset in the field of cold-start recommendation methods. The experiment was conducted with a partial product cold-start setting, meaning that products in the test set did not appear in the training set and were not involved in the model training process. This invention uses click-through rate (HR@K) and normalized discount gain (NDCG@K) as evaluation criteria for recommending the top K products. This invention selected eight methods for performance comparison, including (1) the content-based cold-start method DUIF; (2) the robustness-based cold-start methods DropoutNet and MTPR; (3) the alignment-based cold-start methods Heater, CLCRec, and CCFCRec; and (4) the generative cold-start methods GAR and GoRec.
[0101] Table 1. Cold start recommendation results of the method of the present invention and the comparative method on Amazon Baby.
[0102] Model Recall@10 Recall@20 NDCG@10 NDCG@20 DUIF 0.0217 0.0381 0.0117 0.0163 DropoutNet 0.0178 0.0208 0.0057 0.0071 MTPR 0.0261 0.0328 0.0156 0.0208 Heater 0.0303 0.0492 0.0178 0.0222 CLCRec 0.0247 0.0418 0.0133 0.0181 CCFCRec 0.0295 0.0464 0.0159 0.0209 GAR 0.0307 0.0485 0.0163 0.0210 GoRec 0.0335 0.0544 0.0179 0.0249 Ours 0.0381 0.0628 0.0215 0.0277
[0103] Specifically, Table 1 shows the experimental results of the Amazon-Baby dataset on all methods. It can be seen that the method proposed in this invention outperforms the eight methods in both HR@K and NDCG@K.
Claims
1. A multimedia generalization recommendation method based on invariant learning, characterized in that, The procedure is as follows: Step 1: Construct heterogeneous data, including: constructing a user-product interaction matrix using user interaction records. Extracting the original multimedia feature set of the product using its multimedia content. ; Step 2: Construct a cold-start recommendation network, including a representation generator. Embedded layer And the user collaborative representation matrix P; Step 3, the characterization generator The original features of the multimedia are processed to obtain the product set. Representation vectors of each mode Based on Construct the alignment loss from the representation matrices of any two modes. ; Step 4, Embedding Layer Initialize user collaborative representation matrix After encoding, the first... The final representation vector of each user ; Step 5: Construct from a cyclic blending layer A set of weights for modality fusion under different environments is used to generate representations of differentiated environments and optimize invariant representations; Step 5.1: Sample a set of dimensions from the Dirichlet distribution. The weight set for modal fusion in the first environment ; Step 5.2, in the... Weight set for modal fusion of an environment Based on this, Perform a cyclic displacement operation to obtain the first... Weight set for modal fusion of an environment ; thereby obtaining Weight sets for modality fusion under different environments ; Step 5.3, according to equation (6), the first... products The representation vector of the first mode is processed to generate the second mode. The product in the first The final representation vector under each environment ; (6) In equation (6), In the e-th different environment, the first... The weights of each modality; Indicates the first The first product The representation vector of a modality; Step 6, according to and The inner product is used to calculate the final representation vectors of users and products to obtain the first... The first environment The user on the first Product preference score , used to establish the first Recommended BPR loss under various conditions Thus, an invariant loss function is constructed. ,in, Expressing expectations, Indicates variance; Represents the weights for invariant learning; Step 7, Combine and Multi-task learning is performed on the product recommendation network to update the network parameters until convergence, thereby obtaining the optimal parameters. Product recommendation models are used to implement new product sets Recommendation.
2. The multimedia generalization recommendation method based on invariant learning according to claim 1, characterized in that, Step 1 is performed as follows: Step 1.1, let Represents a user set, Represents the total number of users; let Indicates a product set, Indicates the total number of products; let This indicates a new product set with no interaction history. This indicates the total number of products in the new product cluster; make This represents the user-product interaction matrix, where, Indicates the first Did the user interact? The product, if the first The first user interaction Each product then makes Otherwise, let ; Step 1.2, let Indicates product set Multimedia content, including Indicates product set The Middle A collection of multimedia content in various modalities. Indicates the total number of modes. ; Each modality's feature extraction model is used to extract features from the corresponding multimedia content, resulting in the original multimedia feature set. ,in, Indicates product set The The original feature matrix of multimedia in each modality, and ,in, Indicates the first The first product Multimedia primitive features of various modalities This refers to the dimension of the original features of multimedia.
3. The multimedia generalization recommendation method based on invariant learning according to claim 2, characterized in that, Step 3 is performed as follows: Step 3.1: Calculate the first step according to formula (1). The first product Representation vector of a mode Thus, a product set is obtained. The Characterization matrix of each mode Thus, a product set is obtained. The representation set of each mode : (1) In equation (1), Indicates the first A representation generator for various modes; Step 3.2: According to formula (2), analyze the product set. Align the representation matrices of any two modes to obtain the alignment loss. : (2) In equation (2), Indicates product set The The representation matrix of each mode.
4. The multimedia generalization recommendation method based on invariant learning according to claim 3, characterized in that, Step 5 is performed as follows: Step 5.1, according to equation (4), we obtain ; (4) In equation (5), This represents the set of weights for modal fusion in the first constructed environment. Indicates the first environment The weights of each modality The first one used to adjust the Dirichlet distribution One parameter; To show obedience; Step 5.2: According to equation (5), we obtain the first... Weight set for modal fusion of an environment : (5) In equation (6), Indicates a cyclic displacement operation; Indicates the first The result after the second cycle displacement, when e=2, let = .
5. The multimedia generalization recommendation method based on invariant learning according to claim 4, characterized in that, Step 6 is performed as follows: Step 6.1: Calculate the first step according to equation (7). The first environment The user on the first Product preference score : (7) In equation (7), A function representing a fractional tendency; Step 6.2: Establish the first equation according to formula (8). Recommended BPR loss under various conditions : (8) In equation (8), This represents the set of interactions with all products that all users have ever interacted with. Indicates the first Any product that a user has not interacted with. Indicates the first A collection of products that facilitate user interaction. Indicates the activation function; Indicates the first The first user interaction Interactive items for each product; Indicates the first The first environment The user on the first The preference score for each product; Step 6.3: Construct the cross-environment invariant loss function according to equation (9). : (9)。 6. The multimedia generalization recommendation method based on invariant learning according to claim 5, characterized in that, Step 7 is performed as follows: Step 7.1: Establish a multi-task optimization objective based on equation (10). : (10) In equation (10), These are all the parameters to be optimized. The weights representing the alignment loss; Step 7.2: Train the product recommendation network using gradient descent and calculate the multi-task optimization objective. To update parameters until The optimal parameters are obtained by continuing until the convergence condition is met. Product recommendation models are used to implement new product sets Recommendation.
7. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing any of the multimedia generalization recommendation methods of claims 1-6, the processor being configured to execute the program stored in the memory.
8. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program, when executed by a processor, performs the steps of any of the multimedia generalization recommendation methods described in claims 1-6.