Article recommendation method and device based on multi-modal representation, equipment and medium

Abstract

The invention relates to the technical field of article recommendation, in particular to an article recommendation method and device based on multi-modal representation, equipment and a medium. The method comprises the following steps: firstly, carrying out alignment coding processing on a text and an image to obtain a text semantic coding matrix and an image semantic coding matrix, then predicting the preference degree of a user for each article according to the text semantic coding matrix, the image semantic coding matrix and an article representation matrix, and finally, determining the preference degree of the user for each article according to the preference degree of the user. And obtaining the target recommended article. According to the method, alignment is carried out to enable the same row corresponding to the text semantic coding matrix and the image semantic coding matrix to point to the same article so as to effectively represent the same article. Due to the fact that the text semantics and the image semantics corresponding to the same article can be accurately matched through alignment, the preference degree of the user for each article can be calculated subsequently according to the text semantics and the image semantics of the articles, and then recommendation accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of item recommendation, and in particular to an item recommendation method, device, equipment and medium based on multimodal representation. Background Art

[0002] Based on the user's historical interaction information, the user's preference for each item is predicted, and then the corresponding items are recommended to the user based on the preference. Historical interaction information refers to the user's behavior in obtaining information related to an item. For example, a user watching a teaching video on how to use a certain item is the user's historical interaction information. The historical interaction information includes multiple modal information such as item identification information, item text information, and item image information. The item identification information is the item number, the item text information is the description of the item in the form of text, and the item image information is the display of the item in the form of an image. Although the prior art also recommends items to users based on the user's multimodal historical interaction information, it does not align the multimodal historical interaction information, which makes it impossible to distinguish which modal information belongs to the same item, thereby reducing the accuracy of item recommendations.

[0003] Therefore, the prior art still needs to be improved and enhanced. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a method, device, equipment and medium for item recommendation based on multimodal representation, which solves the problem that the prior art reduces the accuracy of item recommendation.

[0005] To achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides an item recommendation method based on multimodal representation, comprising: Acquire the user's historical interactive item text and historical interactive item image, and perform alignment coding processing on the historical interactive item text and historical interactive item image to obtain a text semantic coding matrix and an image semantic coding matrix, wherein the same row and column elements of the text semantic coding matrix and the image semantic coding matrix correspond to the same item; Generate an item representation matrix for representing the interaction time and identification information of each item, and predict the user's preference for each item based on the text semantic coding matrix, the image semantic coding matrix and the item representation matrix; According to the user's preference for each of the items, recommended items are selected from the items.

[0006] In one implementation, alignment coding is performed on the historical interactive item text and the historical interactive item image to obtain a text semantic coding matrix and an image semantic coding matrix, including: Applying a semantic discriminator to the historical interactive item text and the historical interactive item image to obtain the item semantics of the historical interactive item text and the item semantics of the historical interactive item image; Applying a text encoder to the historical interaction item texts marked with the item semantics to obtain a text semantic encoding matrix; An image encoder is applied to the historical interaction item images marked with the item semantics to obtain an image semantic encoding matrix.

[0007] In one implementation, the text encoder and the image encoder are trained text encoders and trained image encoders, and the training method of the text encoder and the image encoder is joint training, and constructing a total loss function of the joint training includes: During the training process, a text semantic coding sample matrix output by the text encoder is obtained, and an image semantic coding sample matrix output by the image encoder is obtained; Dividing the sample text semantics contained in the text semantic encoding sample matrix into text sample pair groups, each of the text sample pair groups includes a first positive sample pair and a first negative sample pair, and the items corresponding to the first positive sample pair and the first negative sample pair include only one identical item; Obtaining a first loss function according to the first positive sample pair and the first negative sample pair; Semantically dividing each image sample contained in the image semantic coding sample matrix into image sample pair groups, each of the image sample pair groups includes a second positive sample pair and a second negative sample pair, and the objects corresponding to the second positive sample pair and the second negative sample pair include only one identical object; Obtaining a second loss function according to the second positive sample pair and the second negative sample pair; According to the text semantic coding sample matrix and the image semantic coding sample matrix, a text image sample group is obtained, each of the text image sample groups includes a text image positive sample pair and a text image negative sample pair, the sample text semantics and image sample semantics included in the text image positive sample pair of each group correspond to the same object, and the sample text semantics and image sample semantics included in the text image negative sample pair correspond to two different objects; Obtaining a third loss function according to the text image positive sample pair and the text image negative sample pair; A total loss function is constructed based on the first loss function, the second loss function, and the third loss function.

[0008] In one implementation, obtaining a first loss function according to the first positive sample pair and the first negative sample pair includes: Determining the semantic similarity of two sample texts included in the first positive sample pair; Determining the semantic similarity between two sample texts included in the first negative sample pair; A first loss function is obtained according to the similarity corresponding to the first positive sample pair and the similarity corresponding to the first negative sample pair.

[0009] In one implementation, predicting the user's preference for each of the items based on the text semantic encoding matrix, the image semantic encoding matrix, and the item representation matrix includes: According to the item representation matrix, a behavior preference sequence is obtained, where the behavior preference sequence is used to represent the user's preference order for each item; Aggregating the text semantic coding matrix and the image semantic coding matrix to obtain a content preference sequence; Aggregating the behavior preference sequence and the content preference sequence to obtain a mixed preference sequence; Obtaining a user final preference representation vector according to the behavior preference sequence, the content preference sequence, and the mixed preference sequence; The user's preference for each of the items is predicted based on the user's final preference representation vector.

[0010] In one implementation, the item representation matrix is ​​subjected to an item number modal sequence encoder to obtain a behavior preference sequence; the text semantic encoding matrix and the image semantic encoding matrix are subjected to a content modal sequence encoder for aggregation processing to obtain a content preference sequence; the behavior preference sequence and the content preference sequence are subjected to a mixed modal sequence decoder for aggregation processing to obtain a mixed preference sequence; After completing the training of the article number modal sequence encoder, the content modal sequence encoder and the mixed modal sequence decoder are jointly trained based on the trained article number modal sequence encoder.

[0011] In one implementation, the final loss function used for the joint training of the content modality sequence encoder and the mixed modality sequence decoder is composed of a contrast loss function, a cross entropy loss function and sample information, and the sample information is generated by a sample identifier, a sample text and a sample image.

[0012] In a second aspect, an embodiment of the present invention further provides an item recommendation device based on multimodal representation, wherein the device includes the following components: An alignment module is used to obtain the user's historical interactive item text and historical interactive item image, and perform alignment coding processing on the historical interactive item text and historical interactive item image to obtain a text semantic coding matrix and an image semantic coding matrix, wherein the same row and column elements of the text semantic coding matrix and the image semantic coding matrix correspond to the same item; A prediction module, used to generate an item representation matrix for representing the interaction time and identification information of each item, and predict the user's preference for each item based on the text semantic encoding matrix, the image semantic encoding matrix and the item representation matrix; The recommendation module is used to obtain target recommended items based on the user's preference for each of the items.

[0013] In a third aspect, an embodiment of the present invention further provides a terminal device, wherein the terminal device comprises a memory, a processor, and an item recommendation program based on multimodal representation stored in the memory and executable on the processor, and when the processor executes the item recommendation program based on multimodal representation, the steps of the above-mentioned item recommendation method based on multimodal representation are implemented.

[0014] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which is stored an item recommendation program based on multimodal representation, and when the item recommendation program based on multimodal representation is executed by a processor, the steps of the above-mentioned item recommendation method based on multimodal representation are implemented.

[0015] Beneficial effects: The present invention first performs alignment coding processing on the user's historical interactive item texts and historical interactive item images to obtain a text semantic coding matrix and an image semantic coding matrix, and then predicts the user's preference for each item based on the text semantic coding matrix, the image semantic coding matrix and the item representation matrix, and finally selects recommended items from each item to the user based on the user's preference. The alignment of the present invention is to make the same row corresponding to the text semantic coding matrix and the image semantic coding matrix point to the same item, that is, the purpose of alignment is to make the text semantic vector and the image semantic vector of the same item as similar as possible, so as to shorten the spatial distance represented by the two vectors, so as to effectively represent the same item. Since alignment can accurately match the text semantics and image semantics corresponding to the same item, it makes it possible to subsequently calculate the user's preference for each item based on the text semantics and image semantics of the item, thereby improving the accuracy of recommendation. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is the overall flow chart of the present invention; Figure 2A flow chart for constructing content modality sample pairs in an embodiment of the present invention; Figure 3 is a schematic diagram of alignment in an embodiment of the present invention; Figure 4 is a schematic diagram of preference prediction in an embodiment of the present invention; Figure 5 A structural diagram of an item recommendation device based on multimodal representation provided by the present invention; Figure 6 This is a block diagram of the internal structure principle of the terminal device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following is a clear and complete description of the technical solution of the present invention in combination with the embodiments and the accompanying drawings. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] Research has found that based on the user's historical interaction information, the user's preference for each item can be predicted, and then the corresponding items can be recommended to the user based on the preference. Historical interaction information refers to the user's behavior in obtaining information related to an item. For example, a user watching a teaching video on how to use a certain item is the user's historical interaction information. The historical interaction information includes multiple modal information such as item identification information, item text information, and item image information. The item identification information is the item number, the item text information is the description of the item in the form of text, and the item image information is the display of the item in the form of an image. Although the existing technology also recommends items to users based on the user's multimodal historical interaction information, it does not align the multimodal historical interaction information, which makes it impossible to distinguish which modal information belongs to the same item, thereby reducing the accuracy of item recommendations.

[0019] In order to solve the above technical problems, the present invention provides a method, device, equipment and medium for item recommendation based on multimodal representation, which solves the problem that the prior art reduces the accuracy of item recommendation.

[0020] The item recommendation method based on multimodal representation of this embodiment can be applied to a terminal device, which can be a terminal product with image processing and text processing functions, such as a computer. Figure 1 As shown in , the item recommendation method based on multimodal representation specifically includes the following steps: S100, obtaining the user's historical interactive item text and historical interactive item image, and performing alignment coding processing on the historical interactive item text and the historical interactive item image to obtain a text semantic coding matrix and an image semantic coding matrix, wherein the same row and column elements of the text semantic coding matrix and the image semantic coding matrix correspond to the same item; S200, generating an item representation matrix for representing the interaction time and identification information of each item, and predicting the user's preference for each item based on the text semantic encoding matrix, the image semantic encoding matrix and the item representation matrix; S300, obtaining target recommended items according to the user's preference for each of the items.

[0021] The application scenarios of the item recommendation method based on the multimodal representation of steps S100, S200, and S300 are as follows: The user views text and images on the mobile phone. These images and texts describe the functions and appearance of the items. These texts and images are the historical interactive item texts and historical interactive item images generated when the user interacts with the mobile phone. For example, the user browses three sets of historical interactive item texts and three sets of historical interactive item images in the process of interacting with the mobile phone, which respectively involve three items A, B, and C. The three sets of historical interactive item texts and three sets of historical interactive item images are aligned and encoded to obtain a text semantic encoding matrix (a matrix with one row and three columns) and an image semantic encoding matrix (a matrix with one row and three columns). Alignment means that if the text semantics of the first row and first column of the text semantic encoding matrix corresponds to item A, then the image semantics of the first row and first column of the image semantic encoding matrix also corresponds to item A; similarly, the first row and second column of the text semantic encoding matrix and the first row and second column of the image semantic encoding matrix also correspond to the same item C, and similarly, the first row and third column of the text semantic encoding matrix and the first row and third column of the image semantic encoding matrix also correspond to the same item B.

[0022] The first row and the first column of the item representation matrix are used to represent the number ID of item A (the number ID is the identification information) and the interaction time of item A (that is, the time the user views the text and images related to item A on the mobile phone); the first row and the second column of the item representation matrix are used to represent the number ID of item C and the interaction time of item C; the first row and the third column of the item representation matrix are used to represent the number ID of item B and the interaction time of item B.

[0023] During the process of user interaction with the mobile phone, the text semantic coding matrix, image semantic coding matrix and item representation matrix are counted, and the user's preference for the above three items is calculated based on these three matrices. Finally, the shopping software on the mobile phone recommends corresponding items to the user based on the user's preference.

[0024] In the first embodiment, the historical interactive item text and the historical interactive item image are passed through a semantic discriminator to mark the item semantics on the text and the item semantics on the image, and then the historical interactive item text marked with the item semantics is input into a text encoder to obtain a text semantics encoding matrix, and the historical interactive item image marked with the item semantics is input into an image encoder to obtain an image semantics encoding matrix. The text encoder and the image encoder are both trained encoders. In this embodiment, the text encoder and the image encoder are trained in a joint training manner, and the training process is as follows: The user-item interaction sequence is used as a training sample to input into the semantic discriminator. The item information in the sequence includes the item ID and text information, and the most similar item pairs are selected based on the text similarity (similar item pairs are Figure 3 ), the similar item pair includes two item images or two item texts.

[0025] The semantic discriminator marks the item semantics of the user-item interaction sequence, and then inputs the text pairs containing similar item semantics into the text encoder, which outputs the text semantic encoding sample matrix. Similarly, the image pairs containing similar item semantics are input into the image encoder, which outputs the image semantic encoding sample matrix. Then, the total loss function for joint training is constructed based on these two matrices. (The total loss function The supervised fine-tuning loss function SFT is used, and finally the total loss function is used Fine-tune the image encoder and text encoder to complete the training of the image encoder and text encoder.

[0026] The above similar items are item pairs. The construction process of item pairs is as follows: Figure 2 As shown, the big model is called as a semantic discriminator. A prompt word is constructed based on the user-item interaction behavior sequence and input into the big model. In the prompt word, the ID and text information of the last item interacted by the user are used as the target, and the ID and text information of each item interacted before are used as candidates. The big model selects the candidate item with the most similar text semantics to the target item from the candidate items and outputs the candidate item ID. According to the candidate item ID selected by the big model, the target item ID-candidate item ID pair can be obtained, and the database is retrieved to obtain (target item text-candidate item text) and (target item image-candidate item image).

[0027] Among them, construct the total loss function , including the following specific steps S011 to S0110: S011, obtaining a text semantic coding sample matrix output by the text encoder, and obtaining an image semantic coding sample matrix output by the image encoder.

[0028] S012, dividing the sample text semantics contained in the text semantic encoding sample matrix into text sample pair groups, each group of the text sample pair groups includes a first positive sample pair and a first negative sample pair, and the items corresponding to the first positive sample pair and the first negative sample pair include only one identical item.

[0029] For example, the text semantic encoding sample matrix contains five groups of sample text semantics, namely, sample text semantics A, sample text semantics B, sample text semantics C, sample text semantics D, and sample text semantics E. If A describes item A, B describes item B, C describes item C, D describes item D, and E also describes item A. If A describing item A and B describing item B are taken as the first positive sample pair AB, and C describing item C and D describing item D are taken as the first positive sample pair CD, then E describing item A and C describing item C are the first negative sample pair CE, E describing item A and D describing item D are the first negative sample pair ED, and A describing item A and D describing item D are the first negative sample pair AD, wherein the first positive sample pair AB and the first negative sample pair CE are one group of text sample pairs (the first positive sample pair AB and the first negative sample pair CE only include one identical item A), and the first positive sample pair AB and the first negative sample pair ED are another group of text sample pairs.

[0030] The purpose of training a text encoder is to make the text encoder use the same text semantics to describe the same item as much as possible.

[0031] S013: Determine the semantic similarity of the two sample texts contained in the first positive sample pair : ; In the formula, t2t Represents two texts, For one of the sample text semantics (using express One of the first positive sample pairs corresponding to Represents the text, For the The semantics of the other sample text in the first positive sample pair (using represents the other of the first positive sample pair), for and The vector inner product of represents the vector inner product operation, is the parameter used to control the similarity score, .

[0032] S014, determining the semantic similarity of the two sample texts contained in the first negative sample pair : ; For one of the sample text semantics, In Corresponding to the first negative sample pair, For the The semantics of another sample in the first negative sample pair ( Indicates the other one of the first negative sample pairs), for and The vector inner product of .

[0033] S015: based on the similarity corresponding to the first positive sample pair The similarity with the first negative sample pair , and get the first loss function : ; is the total number of first positive sample pairs. In this embodiment, given a batch of item-text pairs, each text pair is considered as a positive sample pair, and the item texts from other text pairs in the same batch are considered as negative samples. This allows the encoder to learn the representation of positive sample pairs in the semantic space, shorten the distance between positive sample vectors to make them more similar, and push the distance between negative sample vectors to make them dissimilar. The first loss function is used Jointly training the text encoder and image encoder allows the encoder to learn the alignment task between text and image.

[0034] S016, semantically dividing each image sample contained in the image semantic coding sample matrix into image sample pair groups, each group of the image sample pair groups includes a second positive sample pair and a second negative sample pair, and the objects corresponding to the second positive sample pair and the second negative sample pair include only one identical object.

[0035] The second positive sample pair and the second negative sample pair are obtained in the same manner as step S012, that is, if an image in the second positive sample pair describes an object, then other images describing the object are in the second negative sample pair.

[0036] S017, obtaining a second loss function according to the second positive sample pair and the second negative sample pair ,in Represents an image.

[0037] The first loss function will be calculated The similarity in the formula can be calculated using the semantics of the image samples to obtain the second loss function .

[0038] S018. According to the text semantic coding sample matrix and the image semantic coding sample matrix, a text image sample group is obtained, each group of the text image sample group includes a text image positive sample pair and a text image negative sample pair, the sample text semantics and image sample semantics included in the text image positive sample pair of each group correspond to the same object, and the sample text semantics and image sample semantics included in the text image negative sample pair correspond to two different objects.

[0039] S019, obtaining a third loss function based on the text image positive sample pair and the text image negative sample pair : ; ; ; For the The sample text in the text image positive sample pair, For the Sample images in text-image positive pairs, Represents an image, for and The vector inner product of For the The similarity of the text image positive sample pairs, For the A sample image in a text image negative sample pair. for and The similarity of, N is the total number of positive sample pairs of text images, is the base of natural logarithms.

[0040] S0110, according to the first loss function The second loss function , the third loss function , construct the total loss function : ; This embodiment adopts The text encoder and image encoder are trained for each downstream recommendation domain, and the fine-tuned encoder is used as a feature encoder to encode the text semantic vector and image semantic vector of each item. These semantic vectors are integrated into the semantic encoding matrix and participate in the modeling of the sequence preference learning module.

[0041] Embodiment 2, based on embodiment 1, step S200 adopts the following method: Figure 4The item number modal sequence encoder shown (the encoder is a Transformer model, which is a neural network model used to process language data), the content modal sequence encoder (the encoder is used to mine user content preferences from text sequences and picture sequences of user interactions), and the mixed modal sequence decoder predict the user's preference for each item. The item number modal sequence encoder, content modal sequence encoder, and mixed modal sequence decoder are all existing technologies.

[0042] A two-step training strategy with low-rank representation optimization is adopted to train the item number modal sequence encoder, content modal sequence encoder, and mixed modal sequence decoder. That is, the item number modal sequence encoder is trained first, and the weight of the encoder is fixed. Then, based on the encoder output of the user's behavior preference sequence (the behavior preference sequence is the user's preference order for items), the content modal sequence encoder and mixed modal sequence decoder are trained.

[0043] Among them, fixed weights are given to the pre-trained item number modal sequence encoder , the update process satisfies the following relationship: ; In the formula, is the updated weight, and are all learnable lightweight weights, d, r, k are vector dimensions, A can be a normal distribution matrix with a mean of 0, and C is an all-zero matrix.

[0044] The content modality sequence encoder and mixed modality sequence decoder are trained by joint training, and the final loss function used in the training is : ; In the formula, To control the hyperparameters of the contrastive learning task, is the contrast loss function, for The regularization parameter of .

[0045] ; In the formula, is the parameter used to control the similarity score, , M refers to the batch size of input data at the same time, and m refers to the interaction sequence of the mth user. a is the first letter of the English word aggregation, and aggregation represents the final user preference representation vector , which is aggregated by the output of the item number modality sequence encoder, content modality sequence encoder, and mixed modality sequence decoder. B refers to the number of negative sample pairs in the contrast loss. To mark the Negative samples. Refers to the content modality representation vector of the next item as a label in the interaction sequence of the mth user. Refers to the The content modality representation vector of the item corresponding to the negative samples.

[0046] is the cross entropy loss function, , where for Figure 4 The preference aggregation layer targets The training prediction value of items, is the true label, representing the user’s preference for the item, and I is the total number of predicted items.

[0047] In the formula, is a set of sample information of three modes, , is an information set consisting of sample identifiers, To number this item, is an information set consisting of sample texts, It is a collection of information consisting of images.

[0048] Embodiment 3: This embodiment is based on the codec trained by Embodiment 1 and Embodiment 2, and predicts the predicted score of each item according to the user's historical interaction item text and historical interaction item image. The score is used to represent the user's preference for the item.

[0049] In this embodiment, the alignment coding process of the historical interactive item text and the historical interactive item image in step S100 to obtain the text semantic coding matrix and the image semantic coding matrix includes the following specific steps S101, S102, and S103: S101, applying a semantic discriminator to a historical interactive item text and a historical interactive item image to obtain the item semantics of the historical interactive item text and the item semantics of the historical interactive item image.

[0050] That is, the text of the item that the user has viewed on the mobile phone (the text is the historical interactive item text) is input into the semantic discriminator, and the semantic discriminator analyzes the item semantics on the item text, and the item semantics is the name of the item. Similarly, the image that the user has viewed on the mobile phone (the image is the historical interactive item image) is input into the semantic discriminator, and the semantic discriminator recognizes the item in the image and analyzes the attribute information of the item.

[0051] S102, applying a text encoder to the historical interactive item texts marked with the item semantics to obtain a text semantic encoding matrix.

[0052] The text semantic encoding matrix is ​​composed of several vectors, each of which includes the encoding information of multiple historical interactive item texts corresponding to the same item semantics. That is, the text encoder divides the historical interactive item texts according to the item semantics of the historical interactive item texts, so that the information of multiple historical interactive item texts belonging to the same item semantics is encoded in the same vector.

[0053] S103, applying an image encoder to the historical interaction item images marked with the item semantics to obtain an image semantic encoding matrix.

[0054] Similarly, the image semantic encoding matrix is ​​composed of several vectors, each of which includes the encoding information of multiple historical interactive item images corresponding to the same item semantics. That is, the image encoder divides the historical interactive item images according to the item semantics of the historical interactive item images, so that the information of multiple historical interactive item images belonging to the same item semantics is encoded in the same vector.

[0055] In this embodiment, the interaction time and identification information of each item are input into the item number modal sequence encoder, and the item number modal sequence encoder outputs the item representation matrix , that is, the elements in the item representation matrix are used to represent the interaction time and identification information of the items.

[0056] ; In the formula, is a vector consisting of the ID of the first item (ID is the item number), is a vector consisting of the position of the first item. The position encoding is used to represent each item in The position in the matrix (the position is related to the interaction time of the item, for example, the interactive item farther away from the current moment is the closer the position in the is a vector consisting of the ID of the first item, is the vector formed by the position of the second object; is a vector consisting of the ID of the nth item, is the vector formed by the position of the nth object. The potential dimensions of all the above vectors (the potential dimension is the preset vector dimension) are , .

[0057] In this embodiment, step S200 includes the following specific steps S201 to S205: S201, based on the item representation matrix ( ), and obtain a behavior preference sequence, where the behavior preference sequence is used to characterize the user's preference order for each item.

[0058] That is, the item is represented by a matrix Input into the item number modal sequence encoder, the item number modal sequence encoder outputs the behavior preference sequence .

[0059] The item number modal sequence encoder uses the following steps to output : The item number modal sequence encoder uses its internal self-attention mechanism to transform the item representation matrix Mapped into query vector Q, key vector K and value vector V: , , , where , , is the linear mapping matrix in the self-attention mechanism.

[0060] The self-attention mechanism outputs a matrix based on Q, K, and V : ; In the formula, represents the self-attention mechanism, stands for normalization and T stands for the transpose of the matrix.

[0061] Then Input into the point-by-point feedforward neural network FNN to output the behavior preference sequence : ; In the formula, ReLU represents the activation function, is a learnable weight matrix.

[0062] is a learnable bias vector, , is the first behavioral preference, For the second behavioral preference, is the nth behavior preference.

[0063] S202, performing aggregation processing on the text semantic coding matrix and the image semantic coding matrix to obtain a content preference sequence .

[0064] That is, firstly transform the text semantic encoding matrix The dimension of the text vector corresponding to each item in is converted to the same dimension as the item number to obtain the item text input matrix ; The image semantic encoding matrix The dimension of the image vector corresponding to each item is converted to the same dimension as the item number to obtain the item image input matrix .

[0065] ; ; Represents the adaptation layer, which is used for dimension size conversion. Represents the first item text vector, Represents the second item text vector, represents the nth item text vector, represents the first item image vector, represents the second item image vector, Represents the nth item image vector, formula and The sizes of the vectors involved are , contains n vectors, each of which is 1xd in size. Embedding refers to Embedding technology, which is a common encoding technology in deep learning. It maps high-dimensional data (such as text and pictures) to low-dimensional continuous vector space, mainly used to capture semantic or structural information in data.

[0066] like Figure 4 As shown, the matrix of the object image input and the item text input matrix Apply the content modality sequence encoder to obtain the content preference sequence .

[0067] That is to say Each item text vector in and Each item image vector in Perform aggregation to obtain the aggregate vector ,in Represents the serial number.

[0068] ; ; In the formula, Representative content, represents the two-norm. represents the aggregation matrix. This embodiment uses It helps to unify the semantic representations among various modalities and reduce the computational complexity.

[0069] Use the causal learning model Transformer to learn user content preferences and obtain content preference sequences : ; Attention refers to the attention mechanism, and FNN refers to the feedforward neural network. Transformer is composed of the attention mechanism and the feedforward neural network stacked together. , is the first content preference, For the second content preference, is the nth content preference.

[0070] S203, the behavior preference sequence and the content preference sequence Perform aggregation processing to obtain a mixed preference sequence : ; in, , For the first mixed preference, For the second mixed preference, For the mixed preferences. This aggregation mechanism can capture the behavior preference sequence and content preference sequence relationship.

[0071] S204, according to the behavior preference sequence , the content preference sequence , the mixed preference sequence , get the user's final preference .

[0072] That is, the behavior preference sequence No. Behavioral preferences , content preference sequence The Content Preferences , mixed preference sequence The Mixed Preferences After polymerization, we get : ; In the formula, Projection represents the linear mapping layer, Represents a fully connected layer.

[0073] S205, representing the final preference vector of the user , predict the user's preference for each of the items : ; In the formula, use Representative items, For items The ID encoding representation vector.

[0074] In this embodiment, step S300 determines the user's preference for each item. Filter the target recommended items from each item, or you can The text and image of the corresponding item are analyzed to find out the features of the item, and then other items are recommended to the user based on the features.

[0075] The following comparative experiments illustrate that the recommended method of the present invention is superior to the existing recommended method: Four real videos collected from a website are used as a data set. An item in the data set contains an item number, a picture of the item, and a text title describing the item. The leave-one-out method is used to divide the data set, that is, the last item in the user interaction sequence is used for testing, the second-to-last item is used as a validation set, and the previous items are used as a training set. As shown in Table 1, indicators 1, 2, 3, and 4 are used to evaluate the recommendation method of the present invention and the recommendation method in the prior art, wherein indicator 1 is a hit rate indicator with a parameter of 5, indicator 2 is a hit rate indicator with a parameter of 10, indicator 3 is a normalized cumulative loss gain with a parameter of 5, and indicator 4 is a normalized cumulative loss gain with a parameter of 10.

[0076] In Table 1, GRU4Rec, SASRec, and BERT4Rec that use only IDs, UniSRec, VQ-rec, and MISSRec that use only texts, and UniSRec and MISSRec that use text plus IDs are all prior art. MISSRec that uses images plus IDs plus texts is also prior art. Among them, GRU4Rec is a sequence recommendation method based on recurrent neural networks, which uses gated neural networks to model user preferences. SASRec is a sequence recommendation method based on causal Transformer. BERT4Rec is a sequence recommendation method based on bidirectional attention. UniSRec that uses text is a sequence recommendation method based on text modality, which learns general semantic representations from text information as item representations. VQ-Rec that uses text is a semantic representation transfer recommendation method based on vector quantization, which maps text representations into multiple learnable codebook representations. MISSRec that uses text is a hybrid enhanced sequence recommendation method based on text modality, which mines user preferences from text sequences. UniSRec with text plus ID is an extended version of UniSRec. The extended version is fine-tuned in the downstream field by combining the item ID representation matrix to improve the recommendation effect. MISSRec with text plus ID is an extended version of MISSRec. The extended version is fine-tuned in the downstream field by combining the item ID representation matrix and the text representation matrix. MISSRec with image plus ID plus text is a method of jointly modeling user preferences by combining the item ID representation matrix, text representation matrix and image representation matrix.

[0077] Table 1

[0078] The optimal index value of the recommended method in the prior art is the underlined index value. It can be seen from Table 1 that, compared with the optimal index value corresponding to the prior art, the index value of index three of the recommended method of the present invention is increased by 6.87%, and the index value of index four is increased by 5.87%.

[0079] Ablation experiments were conducted on four datasets to verify the effectiveness of the modal semantic alignment module of the present invention. As shown in Table 2, the four datasets are comic dataset, dance dataset, food dataset, and movie dataset. Six ablation experiments were set up, namely w / o (remove the content modality-aware contrastive learning of the present invention), w / o LoRAlayer (remove the low-rank representation layer of the present invention), w / o Emb L2 norm (remove the L2 regularization of the present invention, L2 is the two norm), w / o MixDec (remove the mixed modality decoder of the present invention), w / o IDEnc (remove the ID modality sequence encoder of the present invention), w / o ContentEnc (remove the content modality sequence encoder of the present invention).

[0080] Table 2

[0081] Indicator 4 is used as the evaluation indicator to evaluate the above six ablation experiments and the recommended method of the present invention. It can be seen from Table 2 that the supervised fine-tuning of the combined text encoder and image encoder of the present invention effectively improves the effectiveness of content modality representation learning; removing any alignment task of the present invention (alignment tasks include text-to-text alignment tasks, text-to-image alignment tasks, and image-to-image alignment tasks) will lead to a decrease in model performance.

[0082] By comparing different training strategies, the effectiveness of the two-step training strategy based on low-rank representation optimization proposed in the present invention is verified. The comparison results are shown in Table 3: Table 3

[0083] The #epo in Table 3 represents the number of iterations. Fixed IDEmb refers to fixing the encoding matrix weights of the item ID and updating the other parts of the model; Fixed IDEnc refers to fixing the weights of the ID modal sequence encoder and updating the other parts of the model; Fixed IDEmb&Enc refers to fixing the encoding matrix of the item ID and the weights of the ID modal sequence encoder, and updating the other parts of the model; Fixed ConEmb refers to fixing the weights of the item content modal matrix (including the image semantic encoding matrix and the text semantic encoding matrix), and updating the other parts of the model; Fixed ConEnc refers to fixing the weights of the item content modal encoder and updating the other parts of the model; Fixed ConEmb&ConEnc refers to fixing the weights of the item content modal matrix and the item content modal encoder, and updating the other parts of the model; End2end (not fixed) refers to using end-to-end training without fixing the weights of any part of the model.

[0084] As can be seen from Table 3, end-to-end training is very inefficient and can lead to unstable performance. This is because the joint training of content modality and ID modality makes it difficult for the model to converge; first training the sequence encoder of pure content modality usually requires more training times and leads to suboptimal recommendation performance; pre-training the ID modality encoder and then training the content modality encoder and other components (i.e., the two-step training strategy proposed in this invention) brings stable performance improvement and requires fewer iterations to achieve model convergence.

[0085] This embodiment also provides an item recommendation device based on multimodal representation, such as Figure 5 As shown, the device comprises the following components: Alignment module 01, used to obtain the user's historical interactive item text and historical interactive item image, and perform alignment coding processing on the historical interactive item text and historical interactive item image to obtain a text semantic coding matrix and an image semantic coding matrix, wherein the same row and column elements of the text semantic coding matrix and the image semantic coding matrix correspond to the same item; Prediction module 02, used to generate an item coding matrix for representing the interaction time and identification information of each item, and predict the user's preference for each item based on the text semantic coding matrix, the image semantic coding matrix and the item coding matrix; The recommendation module 03 is used to obtain target recommended items according to the user's preference for each of the items.

[0086] Based on the above embodiments, the present invention further provides a terminal device, whose principle block diagram can be shown as follows: Figure 6 As shown. The terminal device includes a processor, a memory, a network interface, and a display screen connected via a system bus. Among them, the processor of the terminal device is used to provide computing and control capabilities. The memory of the terminal device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for recommending items based on multimodal representation is implemented. The display screen of the terminal device can be a liquid crystal display screen or an electronic ink display screen.

[0087] Those skilled in the art will understand that Figure 6 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal device to which the scheme of the present invention is applied. The specific terminal device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0088] In one embodiment, a terminal device is provided. The terminal device includes a memory, a processor, and an item recommendation program based on multimodal representation stored in the memory and executable on the processor. When the processor executes the item recommendation program based on multimodal representation, the following operation instructions are implemented: Acquire the user's historical interactive item text and historical interactive item image, and perform alignment coding processing on the historical interactive item text and historical interactive item image to obtain a text semantic coding matrix and an image semantic coding matrix, wherein the same row and column elements of the text semantic coding matrix and the image semantic coding matrix correspond to the same item; Generate an item coding matrix for representing the interaction time and identification information of each item, and predict the user's preference for each item based on the text semantic coding matrix, the image semantic coding matrix and the item coding matrix; According to the user's preference for each of the items, the target recommended items are obtained.

[0089] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An item recommendation method based on multimodal representation, characterized in that: include: Acquire the user's historical interactive item text and historical interactive item image, and perform alignment coding processing on the historical interactive item text and the historical interactive item image to obtain a text semantic coding matrix and an image semantic coding matrix, wherein the same row and column elements of the text semantic coding matrix and the image semantic coding matrix correspond to the same item; Generate an item representation matrix for representing the interaction time and identification information of each item, and predict the user's preference for each item based on the text semantic encoding matrix, the image semantic encoding matrix and the item representation matrix; According to the user's preference for each of the items, the target recommended items are obtained.

2. The method for recommending items based on multimodal representation according to claim 1, wherein: Perform alignment coding processing on the historical interactive item text and the historical interactive item image to obtain the text semantic coding matrix and the image semantic coding matrix, including: Applying a semantic discriminator to the historical interactive item text and the historical interactive item image to obtain the item semantics of the historical interactive item text and the item semantics of the historical interactive item image; Applying a text encoder to the historical interaction item texts marked with the item semantics to obtain a text semantic encoding matrix; An image encoder is applied to the historical interaction item images marked with the item semantics to obtain an image semantic encoding matrix.

3. The method for recommending items based on multimodal representation according to claim 2, wherein: The text encoder and the image encoder are trained text encoders and trained image encoders, and the training method of the text encoder and the image encoder is joint training, wherein constructing the total loss function of the joint training includes: During the training process, a text semantic coding sample matrix output by the text encoder is obtained, and an image semantic coding sample matrix output by the image encoder is obtained; Dividing the sample text semantics contained in the text semantic encoding sample matrix into text sample pair groups, each of the text sample pair groups includes a first positive sample pair and a first negative sample pair, and the items corresponding to the first positive sample pair and the first negative sample pair include only one identical item; Obtaining a first loss function according to the first positive sample pair and the first negative sample pair; Semantically dividing each image sample contained in the image semantic coding sample matrix into image sample pair groups, each of the image sample pair groups includes a second positive sample pair and a second negative sample pair, and the objects corresponding to the second positive sample pair and the second negative sample pair include only one identical object; Obtaining a second loss function according to the second positive sample pair and the second negative sample pair; According to the text semantic coding sample matrix and the image semantic coding sample matrix, a text image sample group is obtained, each of the text image sample groups includes a text image positive sample pair and a text image negative sample pair, the sample text semantics and image sample semantics included in the text image positive sample pair of each group correspond to the same object, and the sample text semantics and image sample semantics included in the text image negative sample pair correspond to two different objects; Obtaining a third loss function according to the text image positive sample pair and the text image negative sample pair; A total loss function is constructed based on the first loss function, the second loss function, and the third loss function.

4. The method for recommending items based on multimodal representation according to claim 3, characterized in that: According to the first positive sample pair and the first negative sample pair, a first loss function is obtained, including: Determining the semantic similarity of two sample texts included in the first positive sample pair; Determining the semantic similarity between two sample texts included in the first negative sample pair; A first loss function is obtained according to the similarity corresponding to the first positive sample pair and the similarity corresponding to the first negative sample pair.

5. The method for recommending items based on multimodal representation according to claim 1, wherein: Predicting the user's preference for each of the items based on the text semantic coding matrix, the image semantic coding matrix, and the item representation matrix includes: According to the item representation matrix, a behavior preference sequence is obtained, where the behavior preference sequence is used to represent the user's preference order for each item; Aggregating the text semantic coding matrix and the image semantic coding matrix to obtain a content preference sequence; Aggregating the behavior preference sequence and the content preference sequence to obtain a mixed preference sequence; Obtaining a user final preference representation vector according to the behavior preference sequence, the content preference sequence, and the mixed preference sequence; The user's preference for each of the items is predicted based on the user's final preference representation vector.

6. The method for recommending items based on multimodal representation according to claim 5, characterized in that: Applying an item number modal sequence encoder to the item representation matrix to obtain a behavior preference sequence; applying a content modal sequence encoder to the text semantic encoding matrix and the image semantic encoding matrix to perform aggregation processing to obtain a content preference sequence; applying a mixed modal sequence decoder to the behavior preference sequence and the content preference sequence to perform aggregation processing to obtain a mixed preference sequence; After completing the training of the article number modal sequence encoder, the content modal sequence encoder and the mixed modal sequence decoder are jointly trained based on the trained article number modal sequence encoder.

7. The method for recommending items based on multimodal representation according to claim 6, wherein: The final loss function used for the joint training of the content modality sequence encoder and the mixed modality sequence decoder is composed of a contrast loss function, a cross entropy loss function and sample information, and the sample information is generated by a sample identifier, a sample text and a sample image.

8. An item recommendation device based on multimodal representation, characterized in that: The device comprises the following components: An alignment module is used to obtain the user's historical interactive item text and historical interactive item image, and perform alignment coding processing on the historical interactive item text and historical interactive item image to obtain a text semantic coding matrix and an image semantic coding matrix, wherein the same row and column elements of the text semantic coding matrix and the image semantic coding matrix correspond to the same item; A prediction module, used to generate an item representation matrix for representing the interaction time and identification information of each item, and predict the user's preference for each item based on the text semantic encoding matrix, the image semantic encoding matrix and the item representation matrix; The recommendation module is used to obtain target recommended items based on the user's preference for each of the items.

9. A terminal device, characterized in that: The terminal device includes a memory, a processor, and an item recommendation program based on multimodal representation stored in the memory and executable on the processor. When the processor executes the item recommendation program based on multimodal representation, the steps of the item recommendation method based on multimodal representation as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an item recommendation program based on multimodal representation. When the item recommendation program based on multimodal representation is executed by a processor, the steps of the item recommendation method based on multimodal representation as described in any one of claims 1 to 7 are implemented.

Images