A time interval-aware multimodal sequence recommendation method

Abstract

The present invention discloses a time interval-aware multimodal sequence recommendation method in the technical field of recommendation systems, and the steps are as follows: extracting text features, image features and embeddings of modalities in a project, and extracting time interval representations of user interaction sequences on the modalities, adding the time interval representations to the modal representations to obtain time interval-aware modal representations, establishing user representations based on modal learning, and aggregating the user's preference scores on each modality to obtain an overall preference score, and making recommendations based on the overall preference scores. By introducing a time interval encoding method and a flexible two-stage fusion strategy, the present invention not only provides a richer user preference representation for a multimodal recommendation system, but also greatly improves the recommendation performance.

Description

Technical Field

[0001] The present invention relates to the technical field of recommendation systems, and in particular to a time interval-aware multimodal sequence recommendation method. Background Art

[0002] The technology and product applications of recommendation systems have penetrated into multiple fields and are constantly expanding their scope of application. Known fields such as e-commerce, media, and social networks have widely used recommendation technology to improve user experience and business value, while potential fields such as healthcare, smart homes, and smart transportation are also exploring how to optimize services and resource allocation through recommendation systems. With the continuous advancement of technology, the application prospects of recommendation systems will be broader and deeper.

[0003] Recommendation systems play an important role in solving the data explosion problem of e-commerce platforms. In order to improve the performance of recommendation systems, researchers have integrated project attributes, project images, interaction sequences, etc. into the recommendation systems. This type of recommendation system is called a multimodal sequence recommendation system.

[0004] The existing methods for learning interaction order are to arrange the items that the user has interacted with in order, learn a representation for each position, and use this position representation for subsequent user modeling. The existing methods for multimodal fusion strategies can be divided into two types: one is early fusion, which is to merge into a unified representation before user modeling, and the other is late fusion, which is to fuse the independent processing results of different modalities. The above two fusion strategies have their own advantages and disadvantages, and usually need to be analyzed in combination with specific problems.

[0005] Existing methods only use the interaction order, without considering the impact of time intervals on user preferences. Users are likely to change their preferences after a long period of inactivity. Ignoring the impact of time intervals on user preferences may result in the inability to capture user preferences that change over time, leading to suboptimal recommendation performance. We investigated the time intervals in the dataset and found that: (1) 24.25% of the interaction time intervals exceed 100 days; (2) The interaction intervals between the same pair of items range from 1 day to 210 days. The above situation confirms that it is necessary to consider the time interval. In addition, the rough modal fusion methods of existing methods may fail to explore the complementary characteristics of multimodal data, resulting in the loss of valuable information. An effective multimodal fusion method needs to consider the commonalities between different modalities, as well as the potential differences between different modalities. Summary of the invention

[0006] The purpose of this section is to summarize some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0007] Therefore, the purpose of the present invention is to provide a time interval-aware multimodal sequence recommendation method, which not only provides a richer user preference representation for the multimodal recommendation system but also greatly improves the recommendation performance by introducing a time interval encoding method and a flexible two-stage fusion strategy.

[0008] To solve the above technical problems, according to one aspect of the present invention, the present invention provides the following technical solutions:

[0009] A time interval-aware multimodal sequence recommendation method, comprising:

[0010] S1. Extraction Project Medium Mode Text features , picture features And ID embedding , and extract the user In modal Interval characterization of interaction sequences on , Indicates ID, Represents text, Indicates a picture;

[0011] S2. Characterize the time interval Add to modal characterization The modal representation of time interval perception can be obtained by ,Right now ;

[0012] S3. Create a user Modality-based Learned user representation ;

[0013] in, Indicates that in modal The user encoder used on represents the learnable parameters in the user encoder;

[0014] S4. User The preference scores on each modality are aggregated to get an overall preference score , based on the overall preference score Make recommendations;

[0015] Among them, users For Projects The preference score on each modality m is calculated as follows:

[0016] , It is a project In modal Project representation on .

[0017] As a preferred solution of the time interval-aware multimodal sequence recommendation method of the present invention, in step S1, ResNet50 is used to extract image features. , using Recformer to extract text features .

[0018] As a preferred solution of the time interval-aware multimodal sequence recommendation method of the present invention, in step S1, the text feature , picture features 、ID embedding and time interval characterization The specific formula is as follows:

[0019] ;

[0020] ;

[0021] ;

[0022] ;

[0023] Among them, TextExtractor() is a text encoder used to extract text features from text information, ImageExtractor() is an image encoder used to extract image features from image information, Embedding() is an embedding matrix used to assign ID embedding to each item, and TimeIntervalEncoder() is a time encoder used to extract time codes from interaction sequences with given timestamps. , and Respectively indicate the project text information, picture information and ID information.

[0024] As a preferred solution of the time interval-aware multimodal sequence recommendation method of the present invention, in step S1, ;

[0025] , is the time-aware position encoding matrix.

[0026] As a preferred solution of the time interval-aware multimodal sequence recommendation method of the present invention, ;

[0027] In the formula, represents the dimension of representation and requires is an integer divisible by 6, is a zero matrix of size 6*6.

[0028] As a preferred solution of the time interval-aware multimodal sequence recommendation method of the present invention, in step S4, the overall preference score .

[0029] Compared with the prior art, the present invention has the following beneficial effects: the present invention introduces a new time interval encoding method in the field of multimodal recommendation, which is significantly different from the traditional location information encoding method. Traditional location information encoding usually only depends on the order of interaction between users and items, and lacks an in-depth understanding of time factors. The time interval encoding of the present invention starts from three time dimensions to comprehensively capture user preferences that change over time. These three dimensions cover time information at the daily, monthly and annual levels, respectively, so that the model can carefully analyze the changing laws of user behavior, thereby more accurately reflecting the evolution of user preferences in different time periods. And the present invention uses a two-stage fusion method for the fusion of four modal information, which combines the advantages of early fusion and late fusion, can effectively avoid conflicts between different modalities and effectively utilize information within the modalities. In short, by introducing a time interval encoding method and a flexible two-stage fusion strategy, the present invention not only provides a richer user preference representation for the multimodal recommendation system, but also greatly improves the recommendation performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below in combination with the accompanying drawings and detailed embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0031] Figure 1 A schematic flow chart of a time interval-aware multimodal sequence recommendation method according to the present invention. DETAILED DESCRIPTION

[0032] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0033] The present invention provides a time interval-aware multimodal sequence recommendation method, which introduces the time interval as a mode together with the other three modes (ID, picture, and text) into the recommendation system, thereby enriching the recommendation system's understanding of user behavior. Specifically, a time interval encoder TIE is designed to process time interval information at three levels (daily level, monthly level, and annual level), and the corresponding independent periodic patterns are obtained by obtaining the time differences of the three levels to more accurately capture the time characteristics of user behavior, and the time information is aggregated to obtain user preferences that change over time. In order to fully explore the correlation and mutual influence between the four modes, a two-stage fusion strategy is designed. In the first stage, the time interval information is injected into each mode to obtain the user's dynamic preference on a specific mode. This means that the time interval not only affects the user's choice, but also reflects the changes in the user's preferences in different time periods. Through this process, the model can capture more detailed user behavior patterns. In the second stage, the user preferences from different modes are aggregated to obtain the user's overall preference. The design of this stage aims to maximize the use of the information complementarity between the modes, and avoid possible conflicts between different modes through a reasonable fusion mechanism, thereby improving the overall recommendation performance. The two-stage fusion strategy can better mine multimodal information and avoid conflicts between different modalities, thereby improving recommendation performance. The design idea of ​​the present invention is to significantly improve recommendation accuracy and user experience by deeply mining the important factor of time interval and combining multimodal information.

[0034] For more details, see Figure 1 In this embodiment, a time interval-aware multimodal sequence recommendation method is provided, and the specific steps are as follows:

[0035] S1. Extraction Project Medium Mode Text features , picture features And ID embedding , and extract the user In modal Interval characterization of interaction sequences on , Indicates ID, Represents text, Represents a picture. Specifically, ResNet50 is used to extract picture features , using Recformer to extract text features , the corresponding ID features are extracted from the learnable embedding table, which is randomly initialized. In addition, the time interval representation The feature extraction process can be summarized as follows:

[0036] ;

[0037] ;

[0038] ;

[0039] ;

[0040] Among them, TextExtractor() is a text encoder used to extract text features from text information, ImageExtractor() is an image encoder used to extract image features from image information, Embedding() is an embedding matrix used to assign ID embedding to each item, and TimeIntervalEncoder() is a time encoder used to extract time codes from interaction sequences with given timestamps. , and Respectively indicate the project text information, picture information and ID information.

[0041] Compared with the traditional one-dimensional time information encoding, the time interval encoder TIE can encode time information in three dimensions, processing time information at the day level, month level, and year level. For each item in the user interaction sequence, the time information corresponding to the , first decouple it according to day, month, and year to get , and , and then according to the modal representation The time interval encoding on mode m can be calculated as follows:

[0042] ;

[0043] In the formula, Display time information The encoding on mode m, , is the time-aware position encoding matrix. The form is as follows:

[0044]

[0045] In the formula, represents the dimension of representation and requires is an integer divisible by 6, It is a zero matrix of size 6*6, each submatrix Different levels of temporal information are injected into different subspaces of dimension d, as follows:

[0046]

[0047] in 1,2,...,d / 6], in short, the time interval encoder TIE uses sine transform to encode the three-dimensional time information into dimensional vector, this time interval encoding can help us capture users’ preferences that change over time from different levels.

[0048] S2. Characterize the time interval Add to modal characterization The modal representation of time interval perception can be obtained by , forming the first stage of fusion, that is ;

[0049] S3. Create a user Modality-based Learned user representation ;

[0050] in, Indicates that in modal The user encoder used on represents the learnable parameters in the user encoder;

[0051] S4. User The preference scores on each modality are aggregated to get an overall preference score , to achieve the second stage of fusion, and then according to the overall preference score Make recommendations;

[0052] Among them, users For Projects The preference score on each modality m is calculated as follows:

[0053] , It is a project In modal Project representation on .

[0054] Although the present invention has been described above with reference to the embodiments, various modifications may be made thereto and parts thereof may be replaced by equivalents without departing from the scope of the present invention. In particular, as long as there is no structural conflict, the various features in the embodiments disclosed in the present invention may be used in combination with each other in any manner, and the fact that these combinations are not exhaustively described in this specification is only for the sake of omitting space and saving resources. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A time interval-aware multimodal sequence recommendation method, characterized in that: include: S1. For mode m∈{id,te,im}, take the features of item i on mode m respectively; When m=te, extract text feature e te , when m = im, extract the image feature e im , when m = id, extract the ID and embed it into e id ; Among them, id represents ID, te represents text, and im represents image; And extract the time interval representation of the interaction sequence of user u on modality m S2. Characterize the time interval Add to modal characterization The modal representation of time interval perception can be obtained by Right now S3. Establish the user representation of user u based on modality m learning Among them, SASRec m represents the user encoder used on modality m, represents the learnable parameters in the user encoder; S4. Calculate the preference score of user u for item i in modality m is the item representation of item i in modality m∈{id,te,im}, Aggregate the preference scores of all modes to get the overall preference score According to the overall preference score r u,i Make a recommendation.

2. The time interval-aware multimodal sequence recommendation method according to claim 1, characterized in that: In step S1, ResNet50 is used to extract image features e im , using Recformer to extract text features te .

3. The time interval-aware multimodal sequence recommendation method according to claim 1, characterized in that: In step S1, the text feature e te 、Image features im 、ID embedded id and time interval characterization The specific formula is as follows: e te =TextExtractor(i te ) e im =ImageExtractor(i im ) e id =Embedding(i id ) Among them, TextExtractor() is a text encoder used to extract text features from text information, ImageExtractor() is an image encoder used to extract image features from image information, Embedding() is an embedding matrix used to assign ID embedding to each item, TimeIntervalEncoder() is a time encoder used to extract time encoding from an interaction sequence with a given timestamp, i te 、i im and i id Respectively represent the text information, image information and ID information of item i.

4. The time interval-aware multimodal sequence recommendation method according to claim 1, characterized in that: In step S1, Where θ={θ j , j = [1, 2, ..., d / 6]}, is the time-aware position encoding matrix, d represents the dimension of representation and is required to be an integer divisible by 6, O is a zero matrix of size 6*6, Used to inject time information of different dimensions into the j-th subspace of dimension d.

Images