Recommendation system modal recovery method and system based on gated convolution generation stream

Abstract

The invention discloses a recommendation system modal recovery method and system based on a gated convolution generation stream. The method comprises the following steps: projecting modal features of a non-missing modal through convolution; performing affine transformation on the projected modal features by applying a multi-level stacked affine coupling layer based on a gated convolutional neural network to obtain flow features without missing modals; mapping to an embedding space with Gaussian distribution so as to enable the distribution to be consistent with the feature distribution of the mode features of the missing mode, and executing distribution transfer between modes so as to recover the initial mode features of the missing mode; and refining the recovered preliminary modal features by using a pre-trained decoder R to obtain the finally recovered modal features of the missing modal. The method aims at solving the problem of item mode missing in a multi-mode recommendation system, the missing mode is recovered through the existing mode, description of item features is enhanced, and more accurate user personalized recommendation is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data recommendation processing of multi-modal recommendation systems (MRSs), and particularly relates to a recommendation system modality recovery method and system based on a gated convolutional generative flow. BACKGROUND

[0002] With the rapid development of information technology, recommendation systems play a crucial role in e-commerce, social media, content distribution, and other fields. Traditional recommendation systems (RSs) mainly rely on a single data modality, such as user behavior data or item attribute data, to provide personalized recommendation services for users. However, this single modality approach has obvious limitations in dealing with complex user needs and item characteristics, making it difficult to fully capture the multi-dimensional association between users and items, thus limiting the accuracy and personalization of the recommendation results. To overcome this limitation, multi-modal recommendation systems (MRSs) have emerged. MRSs integrate multiple heterogeneous modalities of data, such as text, images, audio, video, etc., to more comprehensively understand user needs and item characteristics. Compared with traditional RSs, the complementary information within the multi-modal framework can significantly improve the accuracy and personalization level of the recommendation results. For example, the text modality can provide detailed descriptions of items and user evaluations of items, while the image modality can show the appearance characteristics of items. These different modalities of information complement each other and help to more accurately predict user interests and preferences. However, in practical applications, MRSs face a serious challenge, namely the problem of missing multi-modal data. The absence of text and image modalities can severely damage the performance of MRSs in real-world scenarios. In real-world scenarios, due to incomplete data collection, data storage loss, or data transmission errors, etc., the data of text, image, and other modalities may be partially or completely missing. This data missing phenomenon can severely damage the performance of MRSs, leading to a significant decline in the accuracy and personalization level of the recommendation results. For example, when an image modality of data is missing in a recommendation system, it may not be able to accurately recommend items with specific appearance characteristics to users; when the text modality of data is missing, the system may not be able to fully understand user evaluations and expectations of items, thus affecting the accuracy of recommendations.

[0003] To address the problem of missing multi-modal data and improve the robustness and performance of MRSs, researchers have begun to explore various advanced techniques. Among them, the gating convolution technique and the generative flow technique have shown great potential in this field. The gating convolution technique is a powerful feature extraction and modeling tool. In the design of flow models, constructing flow operations is crucial to enhance model expressiveness and maintain the reversibility of the computational process. A typical example is the affine coupling layer, which is a parameterized bijection function that updates a portion of the input vector through a function that is easy to invert, and this update is in a complex way dependent on the remaining part of the input vector. However, this traditional flow operation has certain limitations when dealing with multi-modal data. To better enhance the density modeling of modal distribution, the gating convolution technique is introduced into the recommendation system. Gating convolution can more flexibly control the weights of the convolution kernel, thereby more effectively extracting and modeling the features of multi-modal data. In the recommendation system, the gating convolution technique can significantly enhance the learning ability of modal features, thereby improving the recovery quality of missing modalities and improving the accuracy and personalization level of multi-modal recommendation systems. The generative flow technique is another important technique in the recovery and modeling of multi-modal data. Normalizing flow is a generative probability model used for accurate and efficient density estimation. It converts a complex, unknown distribution into a known, manageable distribution through a series of reversible mappings. In the recommendation system, the generative flow technique can map the features of different modalities into a unified Gaussian embedding space. In this unified embedding space, the model can more effectively capture the data distribution characteristics of existing modal features, thereby learning a unified representation. This unified representation not only better reflects the internal relationship between different modalities, but also provides more accurate guidance for the recovery of missing modalities. Through the generative flow technique, the recommendation system can more accurately recover missing modal data, thereby improving user experience and satisfaction, and further improving the accuracy and personalization level of the recommendation system. In summary, multi-modal recommendation systems have significant advantages in dealing with complex user needs and item characteristics, but the problem of missing multi-modal data seriously hinders their performance. The introduction of gating convolution and generative flow techniques provides new ideas and methods to solve this problem. SUMMARY

[0004] The technical problem solved by the present application: In view of the above problems of the prior art, a recommendation system modal recovery method and system based on gating convolution generative flow are provided. The present application aims to recover missing modalities using existing modalities, thereby enhancing the description of item features and achieving more accurate user personalized recommendations in multi-modal recommendation systems.

[0005] To solve the above technical problems, the technical scheme adopted by the present application is: A modality recovery method for a recommendation system based on gated convolutional generation flow includes the following steps: S101, for the image and text modalities of the items to be recommended in the original dataset of the recommender system, determine the missing modalities and missing modalities, and extract the modal features of the items' missing modalities from the original dataset; S102, the modal features of the non-missing modalities are projected into the same dimensional space of both image and text modalities through convolution; S103, apply a multi-level stacked affine coupling layer based on gated convolutional neural network to the projected modal features to perform affine transformation to obtain the flow features of the non-missing modalities; S104, Map the flow features of the non-missing modes to an embedding space with a Gaussian distribution to make them consistent with the feature distribution of the modal features of the missing modes, and perform intermodal distribution transfer on the flow features of the non-missing modes mapped to the embedding space with a Gaussian distribution to recover the initial modal features of the missing modes. S105, the recovered preliminary modal features are refined using the pre-trained decoder R to obtain the modal features of the finally recovered missing modal features.

[0006] Optionally, in step S102, the function expression for projecting the modal features of the non-missing modalities into the same dimensional space of both image and text modalities through convolution is: ; in, The non-missing modes after projection modal characteristics, For non-missing modes modal characteristics, For non-missing modes The transformation matrix.

[0007] Optionally, in step S103, when applying an affine transformation to the projected modal features using a multi-level stacked affine coupling layer based on a gated convolutional neural network to obtain the flow features without missing modalities, the affine coupling layer based on the gated convolutional neural network includes a gated convolutional neural network, a scaling factor network, and a translation factor network. The scaling factor network and translation factor network employ a multilayer perceptron (MLP) or convolutional layers. Divided into and The function expression for the two-part feature, and the affine transformation performed by the affine coupling layer based on the gated convolutional neural network, is as follows: ; ; ; wherein, is an output feature of an affine coupled layer based on the gated convolutional neural network, and are two components of the output feature, is the gated convolutional neural network, is a Hadamard product, is a scaling result of the feature by a scaling factor network, is a translation result of the feature by a translation factor network.

[0008] Optionally, the function expression for mapping the flow feature of the non-missing modality into the embedding space with Gaussian distribution to make it consistent with the feature distribution of the modality feature of the missing modality in step S104 is: ; wherein, denotes the mapping of the flow feature of , is the modality feature of the non-missing modality after projection, ” is the output, is the modality feature of the non-missing modality mapped into the embedding space with Gaussian distribution, is the Gaussian distribution of the non-missing modality , is the mean of the modality feature of the non-missing modality , is the variance of the modality feature of the non-missing modality ; and the function expression for performing the inter-modal distribution transfer on the flow feature of the non-missing modality mapped into the embedding space with Gaussian distribution to restore the preliminary modality feature of the missing modality is: ; wherein, denotes the inter-modal distribution transfer on , is the preliminary modality feature of the missing modality .

[0009] Optionally, the function expression for refining the restored preliminary modality feature by using the pre-trained decoder R to obtain the final recovered modality feature of the missing modality in step S105 is: ; wherein, is the final recovered modality feature of the missing modality , is the decoder, is the preliminary modal feature of the missing modal The decoder R is composed of a plurality of residual channel attention blocks RCAB and a 1x1 convolution in series, and the processing of each residual channel attention block RCAB on the input feature includes: first extracting local features from the input feature through a 3x3 convolution; then a channel attention module generates channel weights, the channel attention module is composed of a global average pooling and two layers of multilayer perceptron MLP, and the channel weights are multiplied with the input feature to obtain weighted features; finally, the original input feature and the weighted feature are added through a residual connection, and the output of the plurality of residual channel attention blocks RCAB is finally obtained after the output dimension is adjusted by the 1x1 convolution to obtain the decoding output result.

[0010] Optionally, the loss function adopted by the decoder R during training is: ; ; ; wherein, is the loss function adopted by the decoder R during training, is the feature distribution loss of the recovered modal, is the mean square error loss between the feature of the recovered modal and the feature of the real modal, and are the standard deviation and variance of the modal feature of the item without missing modal , respectively, is the modal feature of the item without missing modal , is the mean of the modal feature of the item without missing modal , is the number of stacked affine coupling layers, denotes the log determinant of the Jacobian matrix of , is the Jacobian matrix of with respect to , and are the results of the first affine transformation and the second affine transformation, respectively, , is the modal feature of the recovered missing modal , is the real modal feature of the missing modal .

[0011] Optionally, step S105 further includes: ​S201, extracting a user-item collaborative view from the original dataset of the recommendation system, the user-item collaborative view refers to a normalized interaction matrix , and the normalized interaction matrix is obtained by extracting a user-item interaction matrix of the user-item from the input original dataset , the user-item interaction matrix The element value in is 1 or 0 to indicate whether the user interacts with the corresponding item, and the user-item interaction matrix The normalized interaction matrix is calculated according to the following formula: ; Wherein, is a normalized degree value matrix; S202, splice the modal feature of the non-missing modal and the modal feature of the missing modal which is finally recovered into a joint view embedding : ; Wherein, is a connection operation, is the splicing result of the modal feature of the non-missing modal , the modal feature of the missing modal which is finally recovered, is a normalized interaction matrix; S203, split the joint view embedding According to and into user representation and item representation , wherein is the dimension of user or item representation; S204, calculate the inner product of the user representation and the item representation According to to obtain the final prediction score of each user to each item , wherein in the superscript is a transpose operation; select the specified number K of items with the highest final prediction score for each user as the recommendation result output of the user .

[0012] In addition, the present application also provides a recommendation system modal recovery system based on a gated convolution generative flow, comprising a microprocessor and a memory connected to each other, the microprocessor being programmed or configured to execute the recommendation system modal recovery method based on the gated convolution generative flow.

[0013] In addition, the present application also provides a computer readable storage medium, wherein a computer program or instructions are stored, the computer program or instructions being programmed or configured to execute the recommendation system modal recovery method based on the gated convolution generative flow by a processor.

[0014] In addition, the present application also provides a computer program product, comprising a computer program or instructions, the computer program or instructions being programmed or configured to execute the recommendation system modal recovery method based on the gated convolution generative flow by a processor.

[0015] Compared with the prior art, the present application mainly has the following beneficial effects: Multi-modal recommendation systems (MRSs) aim to understand user needs through heterogeneous modalities, and compared with traditional recommendation systems (RSs), the complementary information within the multi-modal framework helps to achieve more accurate recommendation results. The present application uses specific flows of gated convolution generative models to analyze the distribution of available modalities and effectively recover missing modalities, while introducing specific flows for different modalities to map their features into a Gaussian distribution embedding space, thereby reducing the distribution difference between different modal features. By utilizing both gated convolution technology and generative flow technology, the present application has unique advantages in enhancing modal feature learning and improving modal recovery quality, effectively solving the problem of missing multi-modal data, significantly improving the accuracy and personalization level of multi-modal recommendation systems, effectively solving the problem of missing multi-modal data, and providing more accurate and personalized recommendation services for users. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 The basic flowchart of the method of the embodiment of the present application is shown.

[0017] Figure 2 The basic principle and application diagram of the method of the embodiment of the present application are shown. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to better understand the technical solutions of the present application, the technical solutions of the present application will be further described in detail below with reference to the drawings in the embodiments of the present application.

[0019] As shown in Figure 1 The recommendation system modal recovery method based on gated convolution generative flow of the present embodiment comprises the following steps: S101, determining the non-missing modality and the missing modality of the image and text modalities of the to-be-recommended item in the original data set of the recommendation system, and extracting the modal feature of the non-missing modality of the item from the original data set; S102, projecting the modal feature of the non-missing modality into the same dimensional space of the image and text modalities by convolution; S103, performing affine transformation on the projected modal feature by using a multi-level stacked affine coupling layer based on a gated convolutional neural network to obtain the flow feature of the non-missing modality; S104, mapping the flow feature of the non-missing modality into an embedding space with a Gaussian distribution to make it consistent with the feature distribution of the modal feature of the missing modality, and performing inter-modal distribution transfer on the flow feature of the non-missing modality mapped into the embedding space with the Gaussian distribution to restore the preliminary modal feature of the missing modality; S105, refining the restored preliminary modal feature by using a pre-trained decoder R to obtain the final restored modal feature of the missing modality.

[0020] In step S101 of the embodiment, the image feature and the text feature of the item can use 4096-dimensional visual data and 384-dimensional text data, and the title, description, category and brand of each item are combined by using a Bert and Resnet pre-training model to obtain the original image feature and text feature of the item. In step S102 of the embodiment, the function expression for projecting the modal feature of the non-missing modality into the same dimensional space of the image and text modalities is: ; wherein, is the modal feature of the projected non-missing modality , is the modal feature of the non-missing modality , is the transformation matrix of the non-missing modality . Considering the difference in semantic space between the original image feature and the text feature of the item, two independent convolutional layers are designed in the embodiment to project the image feature and the text feature of the item into the same dimensional space, and the function expressions of the two independent convolutional layers are: ; ; wherein, is the final obtained text feature of the item, is the original text feature of the item, is the transformation matrix corresponding to the convolutional layer of the text modality, is the final obtained image feature of the item, for the original image feature of the item, for the transformation matrix corresponding to the convolutional layer of the image modality, representing the text modality, representing the image modality.

[0021] In order to enhance the density modeling of the flow, in the embodiment, the gated convolutional neural network is applied to the affine coupling layer, and in step S103, when the projected modality feature is applied to the multi-level stacked affine coupling layer based on the gated convolutional neural network to perform affine transformation to obtain the flow feature of the non-missing modality, the affine coupling layer based on the gated convolutional neural network includes a gated convolutional neural network, a scaling factor network and a translation factor network, the scaling factor network and the translation factor network adopt a multi-layer perception (MLP) or a convolutional layer, the projected modality feature is divided into and two parts of features, and the function expression of the affine transformation performed by the affine coupling layer based on the gated convolutional neural network is: wherein, is the output feature of the affine coupling layer based on the gated convolutional neural network, and are two components of the output feature, is the gated convolutional neural network, is the Hadamard product, is the scaling result of the scaling factor network on the feature , is the translation result of the translation factor network on the feature . In order to enhance the density modeling of the flow, in the embodiment, the value of the component y2 is calculated by applying the gated convolutional neural network to the affine coupling layer, and the function expression of the gated convolutional neural network is: wherein, represents a ReLU activation function, represents a function, is a weight matrix for the gating mechanism, is a weight matrix for feature transformation, is the Hadamard product, and represent the feature mapping of the vector . In the embodiment, and respectively represent​​​​ 0th and 1st power, where the first term is used to generate a gate for modulating features. The second term is used to generate a vector of features, both of which achieve dynamic feature selection and augmentation through Hadamard product. Thus, an arbitrary input feature is transformed after passing through an affine coupling layer as follows: ; where is the input feature, “ ” is the output, is the gating CNN, is the weight matrix for the gating mechanism, is the weight matrix for the feature transformation, is the Hadamard product, is the scaling result of the feature by the scaling factor network, is the translation result of the feature by the translation factor network.

[0022] The features of the specific stream for different modalities are mapped into an embedding space with Gaussian distribution, obtaining a highly consistent feature distribution. The function expression for mapping the stream features of the non-missing modality into an embedding space with Gaussian distribution to make them consistent with the feature distribution of the modal features of the missing modality in step S104 of this embodiment is: ; wherein denotes mapping of the stream features of , is the modal feature of the non-missing modality after projection, “ ” is the output, is the modal feature of the non-missing modality mapped into an embedding space with Gaussian distribution, is the Gaussian distribution of the non-missing modality , is the mean of the modal feature of the non-missing modality , is the variance of the modal feature of the non-missing modality ; the inter-modal distribution transfer is performed on the stream features of the non-missing modality mapped into an embedding space with Gaussian distribution to recover the preliminary modal feature of the missing modality from the modal feature of the non-missing modality of the item, and the function expression for recovering the preliminary modal feature of the missing modality from the modal feature of the non-missing modality of the item is: ; wherein, represents an inter-modal distribution transfer to , is a preliminary modal feature of the missing modal , wherein one of the two, the non-missing modal and the missing modal , is an image modal and the other is a text modal.

[0023] The decoder R of the embodiment refines the recovered modal through multiple residual channel attention blocks RCAB to minimize the gap between the real data and the reconstructed data. In step S105 of the embodiment, the recovered preliminary modal feature is refined using the pre-trained decoder R to obtain the function expression of the final recovered modal feature of the missing modal: ; wherein, is the final recovered modal feature of the missing modal , is the decoder, is the preliminary modal feature of the missing modal . As an optional implementation, the decoder R in the embodiment is a lightweight decoder composed of multiple residual channel attention blocks RCAB and 1x1 convolution in series, and the processing of each residual channel attention block RCAB on the input feature includes: first extracting local features from the input feature through 3x3 convolution; then a channel attention module generates channel weights, the channel attention module is composed of global average pooling and two layers of multi-layer perceptron MLP, and the channel weights are multiplied with the input feature to obtain weighted features; finally, the original input feature and the weighted feature are added through residual connection, and the output of the multiple residual channel attention blocks RCAB is finally adjusted in output dimension through 1x1 convolution to obtain the decoding output result. As shown in Figure 2 , the loss function used by the decoder R in the training of the embodiment is: ; ; ; wherein, is the loss function used by the decoder R in the training, is the feature distribution loss of the recovered modal, is the mean square error loss between the feature of the recovered modal and the real modal feature, and are the standard deviation and variance of the modal feature of the non-missing modal , respectively, The project has no missing modalities modal characteristics, The project has no missing modalities The mean of the modal features, The number of stacking levels of affine coupling layers. express The logarithmic determinant of the Jacobian matrix, for about Jacobian matrix, and The first The affine transformation and the th The result of the affine transformation For the finally recovered missing modes modal characteristics, For missing modes The true modal features. Assuming the image modality is missing, but the text modality exists. Modal features of the text modality. The maximum likelihood estimate can be expressed in the following form: ; in, For samples with a prior distribution, The number of stacking levels of affine coupling layers. express The logarithmic determinant of the Jacobian matrix, for about Jacobian matrix, and The first The affine transformation and the th The result of this affine transformation allows us to obtain the feature distribution loss for recovering the mode under image mode loss as follows: ; in, and These are the text modalities for which the project is not missing. The standard deviation and variance of the modal characteristics For text modalities where the project is not missing modal characteristics, For text modalities where the project is not missing The mean of the modal features. Since in this embodiment, the decoder R... Decode to obtain the missing image modalities Refined modal features , the decoder R is used to refine the distribution gap between the recovered missing modality and the real modality. It optimizes the feature distribution by calculating the mean square error loss between the features of the recovered modality and the real modality, which is expressed as follows: ; wherein, is the refined modality feature of the missing image modality , and is the modality feature of the text modality of the item without missing.

[0024] As shown in Figure 2 , the embodiment step S105 further includes the step of performing recommendation: S201, extracting a user-item collaborative view from an original data set of a recommendation system, wherein the user-item collaborative view refers to a normalized interaction matrix , and the normalized interaction matrix is obtained by extracting a user-item interaction matrix of the user-item from the input original data set, wherein the user-item interaction matrix has an element value of 1 or 0 to represent whether the user interacts with the corresponding item, and the user-item interaction matrix is calculated according to the following formula to obtain the normalized interaction matrix : ; wherein, is a normalized degree value matrix; S202, splicing the modality feature of the non-missing modality and the modality feature of the finally recovered missing modality to form a joint view embedding : ; wherein, is a connection operation, is a splicing result of the modality feature of the non-missing modality and the modality feature of the finally recovered missing modality , and is the normalized interaction matrix; S203, splitting the joint view embedding into a user representation and an item representation according to and , wherein​ a dimension for user or item representation; S204, for the user representation with the item representation According to Inner product calculation is performed to obtain each user The final prediction score of each item wherein the superscript is a transpose operation; for each user The final prediction score The highest specified number K of items are selected as the recommended results output for the user .

[0025] Participation Figure 2 It can be seen that when all modalities are complete, the steps S201-S204 (the preliminary modal feature recovered is directly used as the modal feature extracted) can be directly used for recommendation; if a certain modality (such as an image) is missing, the preliminary modal feature recovered is refined using the pre-trained decoder R to obtain the final recovered modal feature of the missing modality according to the steps S101-S105 of the embodiment method, and then the steps S201-S204 are used to complete the recommendation. The core is to realize the information migration between modalities through distribution alignment and reversible transformation, and to ensure the robustness of the recommendation system when data is missing.

[0026] ​To verify the performance of the recommendation system modal recovery method based on the gating convolution generated flow in the embodiment, the embodiment assumes that the image modal is missing and the text modal is available. In the feature extraction part, the embodiment uses 4096-dimensional visual feature extraction and text features, obtains 384-dimensional text embedding by combining the title, description, category and brand of each item using Bert and Resnet. Considering the difference between the text modal features and the image modal features in the semantic space, the embodiment designs two independent convolutional layers to project them into the same dimensional space. In the available modal distribution transmission part, the embodiment adopts a specific flow based on gating convolution to recover the missing image modal. In addition, in order to make the recovered image modal distribution closer to the real modal distribution, the embodiment utilizes a decoder R composed of residual attention channels. In the joint optimization part, in order to control the influence of the modal distribution loss and effectively construct the user preference representation, the embodiment adopts the feature distribution loss of the recovered modal and the mean square error loss between the feature of the recovered modal and the feature of the real modal, in the prediction part, the recovered image modal is used together with the text modal and the ID embedding for the downstream recommendation task. Conversely, when the text modal is missing and the image modal is available, the recovery process is consistent with the above recovery process. In the embodiment, experiments are carried out on the Baby dataset and the Sports dataset in the Amazon_baby commodity review dataset, and two indexes of Recall (recall rate) and NDCG (normalized discounted cumulative gain) are adopted, and the existing recommendation methods VBPR, MMGCN, LATTICE and BM3 are combined for comparison. The existing recommendation methods VBPR, MMGCN, LATTICE and BM3 all include two varieties, the first variety with the suffix “w / o image” represents the modal feature of the missing image modal, and the second variety with the suffix “SFMRec” represents the modal feature of the image modal recovered by the method in the embodiment. Finally, the performance comparison experimental results of the two varieties of the above existing recommendation methods are shown in Table 1 and Table 2.

[0027] Table 1: Performance comparison experimental results of the method in the embodiment on the baby dataset

[0028] Table 2: Performance comparison experimental results of the method in the embodiment on the sports dataset

[0029] In Table 1 and Table 2, R@K and N@K respectively represent the index values of the two indicators of Recall and NDCG under different K, K is the number of items recommended to the user. As can be seen from Table 1 and Table 2, on the Baby dataset and the Sports dataset, after the existing recommendation methods VBPR, MMGCN, LATTICE and BM3 recover the modal features of the image modal based on the modal recovery method of the recommendation system modal recovery method based on the generation of flow of the gating convolution in the embodiment, compared with the missing image modal, more excellent performance is achieved on the two indicators of Recall and NDCG.

[0030] In addition, the embodiment also provides a recommendation system modal recovery system based on the generation of flow of the gating convolution, which comprises a microprocessor and a memory connected with each other, and the microprocessor is programmed or configured to execute the recommendation system modal recovery method based on the generation of flow of the gating convolution.

[0031] In addition, the embodiment also provides a computer readable storage medium, which stores a computer program or instructions, and the computer program or instructions are programmed or configured to execute the recommendation system modal recovery method based on the generation of flow of the gating convolution by a processor.

[0032] In addition, the embodiment also provides a computer program product, which comprises a computer program or instructions, and the computer program or instructions are programmed or configured to execute the recommendation system modal recovery method based on the generation of flow of the gating convolution by a processor.

[0033] Those skilled in the art should understand that the technical solutions provided by the present application can be in the form of a method, a system or a computer program product. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer readable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer usable program codes. The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general purpose computer, a special purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one flow or multiple flows and / or blocks Figure 1apparatus for performing each function specified in a flow or flows and / or blocks. These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flow Figure 1 a flow or flows and / or blocks Figure 1 a flow or flows and / or blocks Figure 1 a flow or flows and / or blocks Figure 1 a flow or flows and / or blocks

[0034] The above description is only preferred embodiments of the present application, the protection scope of the present application is not limited to the above-mentioned embodiments, any technical solutions falling within the concept of the present application shall be considered as falling within the protection scope of the present application. It should be noted that, for ordinary skilled in the art, some improvements and refinements without departing from the principles of the present application, these improvements and refinements should also be considered as falling within the protection scope of the present application.

Claims

1. A modality recovery method for a recommendation system based on gated convolutional generative flow, characterized in that, Includes the following steps: S101, for the image and text modalities of the items to be recommended in the original dataset of the recommender system, determine the missing modalities and missing modalities, and extract the modal features of the items' missing modalities from the original dataset; S102, the modal features of the non-missing modalities are projected into the same dimensional space of both image and text modalities through convolution; S103, apply a multi-level stacked affine coupling layer based on gated convolutional neural network to the projected modal features to perform affine transformation to obtain the flow features of the non-missing modalities; S104, Map the flow features of the non-missing modes to an embedding space with a Gaussian distribution to make them consistent with the feature distribution of the modal features of the missing modes, and perform intermodal distribution transfer on the flow features of the non-missing modes mapped to the embedding space with a Gaussian distribution to recover the initial modal features of the missing modes. S105, the recovered preliminary modal features are refined using the pre-trained decoder R to obtain the modal features of the finally recovered missing modal features.

2. The modality recovery method for recommendation systems based on gated convolutional generation flow according to claim 1, characterized in that, In step S102, the modal features of the non-missing modalities are projected onto the same dimensional space of both image and text modalities via convolution. The function expression is as follows: ； in, The non-missing modes after projection modal features, For non-missing modes modal characteristics, For non-missing modes The transformation matrix.

3. The modality recovery method for recommendation systems based on gated convolutional generation flow according to claim 1, characterized in that, In step S103, when applying an affine transformation to the projected modal features using a multi-level stacked affine coupling layer based on a gated convolutional neural network to obtain the flow features without missing modalities, the affine coupling layer based on the gated convolutional neural network includes a gated convolutional neural network, a scaling factor network, and a translation factor network. The scaling factor network and translation factor network employ a multilayer perceptron (MLP) or convolutional layers. Divided into and The function expression for the two-part feature, and the affine transformation performed by the affine coupling layer based on the gated convolutional neural network, is as follows: ； ； ； in, The output features of the affine coupling layer based on a gated convolutional neural network, and The two components are the output features. For gated convolutional neural networks, For Hadamard product, For scaling factor networks to feature The scaling result, For the translation factor network to the features The translation result.

4. The modality recovery method for recommendation systems based on gated convolutional generation flow according to claim 1, characterized in that, In step S104, the function expression for mapping the flow features of the non-missing modes to an embedding space with a Gaussian distribution to ensure consistency with the feature distribution of the modal features of the missing modes is as follows: ； in, Indicates to Flow feature mapping, The non-missing modes after projection modal features, " is the output, To map to the non-missing modes in an embedding space with a Gaussian distribution modal characteristics, For non-missing modes Gaussian distribution, For non-missing modes The mean of the modal features, For non-missing modes The variance of the modal features; the intermodal distribution transfer is performed on the flow features mapped to the non-missing modes in the embedding space with a Gaussian distribution to recover the missing modes. The functional expression for the preliminary modal characteristics is: ； in, Indicates to Intermodal distribution transfer, For missing modes Preliminary modal characteristics.

5. The modality recovery method for recommendation systems based on gated convolutional generation flow according to claim 1, characterized in that, In step S105, the recovered preliminary modal features are refined using the pre-trained decoder R to obtain the functional expression of the finally recovered modal features of the missing modalities. ； in, For the finally recovered missing modes modal characteristics, For decoder, For missing modes The decoder R consists of multiple residual channel attention blocks (RCABs) and 1×1 convolutions. Each residual channel attention block (RCAB) processes the input features as follows: first, it extracts local features from the input features through a 3×3 convolution; then, the channel attention module generates channel weights, which consists of global average pooling and two layers of multilayer perceptron (MLP), and multiplies the channel weights with the input features to obtain weighted features; finally, it adds the original input features with the weighted features through residual connections, and the outputs of multiple residual channel attention blocks (RCABs) are finally adjusted by a 1×1 convolution to obtain the decoding output.

6. The modality recovery method for recommendation systems based on gated convolutional generation flow according to claim 1, characterized in that, The loss function used by the decoder R during training is: ； ； ； in, Let R be the loss function used by the decoder during training. To recover the feature distribution loss of the mode, To recover the mean squared error loss between the modal features and the true modal features, and The project has no missing modalities. The standard deviation and variance of the modal characteristics The project has no missing modalities modal characteristics, The project has no missing modalities The mean of the modal features, The number of stacking levels of affine coupling layers, express The logarithmic determinant of the Jacobian matrix, for about Jacobian matrix, and The first The affine transformation and the ... The result of the affine transformation For the finally recovered missing modes modal characteristics, For missing modes The true modal characteristics.

7. The modality recovery method for recommendation systems based on gated convolutional generation flow according to claim 1, characterized in that, Step S105 is followed by: S201, Extract the user-item collaboration view from the original dataset of the recommender system. The user-item collaboration view refers to the normalized interaction matrix. And the normalized interaction matrix The acquisition includes extracting the user-item user-item interaction matrix from the input raw dataset. The user-project interaction matrix The elements in the matrix take values ​​of 1 or 0 to indicate whether the user has interacted with the corresponding item, and the user-item interaction matrix is ​​then used to... The normalized interaction matrix is ​​calculated according to the following formula. : ； in, This is the normalization degree matrix; S202, will include the non-missing modes modal features and the finally recovered missing modes modal features The two are combined into a joint view embedding : ； in, For connection operations, For non-missing modes modal features The missing modes that are finally recovered modal features The splicing result, This is the normalized interaction matrix; S203, embedding the union view according to and Split into user representation Representation of items ,in Dimensions representing users or projects; S204, indicating to the user Representation of items according to Perform inner product calculation to obtain the result for each user. For each project Final predicted score Among them, the superscript For transpose operation; for each user Select final prediction score The highest specified number of K items as the user The recommended results are output.

8. A modal recovery system for a recommendation system based on gated convolutional generation flow, comprising a microprocessor and a memory interconnected, characterized in that, The microprocessor is programmed or configured to execute the mode recovery method for recommendation systems based on gated convolutional generation streams as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute, via a processor, the modality recovery method for recommendation systems based on gated convolutional generation streams as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute, via a processor, the modality recovery method for recommendation systems based on gated convolutional generation streams as described in any one of claims 1 to 7.

Images