Task-oriented multi-modal article recommendation method and device, electronic equipment and medium

Abstract

The embodiment of the invention discloses a task-oriented multi-mode article recommendation method and device, electronic equipment and a medium. One specific embodiment of the method comprises the following steps: acquiring an article modal data set and a user article interaction data set; generating an initial article modal relation graph matrix set, a user article interaction matrix and an inter-article second-order interaction relation matrix; generating a training sample set, and executing the following training steps: inputting the initial interaction embedding representation set, each article modal embedding representation set, the initial article modal relation graph matrix set and the user article interaction matrix into an initial article recommendation model; determining an interaction prediction value corresponding to each training sample; generating an article recommendation loss value; in response to the fact that the article recommendation loss value is smaller than a preset loss threshold value, determining the initial article recommendation model as an article recommendation model; and generating article recommendation information by using the article recommendation model. According to the embodiment, the performance of the article recommendation model can be improved, and the articles interested by the user are recommended to the user.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a task-oriented multimodal item recommendation method, device, electronic device, and medium. Background Art

[0002] With the rapid development of multimodal learning technology, how to recommend multimodal items has become an important research direction of recommendation systems. At present, when recommending multimodal items, the usual approach is: first, in the modal representation extraction stage, pre-trained models (such as text pre-trained models and visual pre-trained models) are used to extract item modal representations. Next, in the user interest modeling stage, the modal representation generated by the pre-trained model is integrated into the user and item interaction representation learning process of the downstream recommendation task to generate the final user and item representations. Finally, the item recommendation results are generated based on the final user and item representations.

[0003] However, when using the above method to recommend multimodal items, the following technical problems often occur:

[0004] Since the above-mentioned item recommendation method is divided into two stages: modal representation extraction and user interest modeling, the item modal representation output by the pre-trained model is usually not well adapted to the final interactive representation learning between users and items, which can easily cause the model to fall into local optimality and reduce the accuracy of item representation. Furthermore, it is difficult to accurately determine the candidate items that users are interested in, thereby making it difficult to recommend the candidate items that users are interested in to users.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure concept and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0006] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0007] Some embodiments of the present disclosure propose task-oriented multimodal item recommendation methods, devices, electronic devices, and media to solve one or more of the technical problems mentioned in the above background technology section.

[0008] In a first aspect, some embodiments of the present disclosure provide a task-oriented multimodal item recommendation method, the method comprising: obtaining an item modal dataset and a user-item interaction dataset, wherein the item modal data in the above-mentioned item modal dataset corresponds to the items in the item set; generating an initial item modal relationship graph matrix set, a user-item interaction matrix and an item second-order interaction relationship matrix based on the above-mentioned item modal dataset and the above-mentioned user-item interaction dataset; generating a training sample set based on the above-mentioned user-item interaction dataset, and performing the following training steps based on the training sample set: inputting a pre-generated initial interaction embedding representation set, each item modal embedding representation set, an initial item modal relationship graph matrix set and a user-item interaction matrix into an initial item recommendation model to obtain a high-order user embedding vector set and a high-order item embedding vector set, wherein the initial interaction embedding representation set includes an initial user embedding vector set and an initial item embedding vector set, and the initial item recommendation model includes a first graph neural network submodule, a second graph neural network submodule, and a A graph neural network submodule, a multi-head attention submodule and an item embedding representation fusion layer; based on the above-mentioned high-order user embedding vector set and the above-mentioned high-order item embedding vector set, determining the interaction prediction value corresponding to each training sample in the at least one training sample included in the training sample set; through a pre-designed information constraint loss function and a recommendation loss function, based on the initial item modal relationship graph matrix set, the above-mentioned second-order interaction relationship matrix between items, the interaction prediction value corresponding to each training sample in the above-mentioned at least one training sample and the corresponding sample label, generating an item recommendation loss value, wherein the above-mentioned information constraint loss function is used to minimize the information difference between each initial item modal relationship graph matrix and the above-mentioned second-order interaction relationship matrix between items; in response to determining that the above-mentioned item recommendation loss value is less than a preset loss threshold, determining the trained initial item recommendation model as the item recommendation model; using the above-mentioned item recommendation model, generating item recommendation information corresponding to the target user, and sending the above-mentioned item recommendation information to the browsing terminal corresponding to the above-mentioned target user.

[0009] In a second aspect, some embodiments of the present disclosure provide a task-oriented multimodal item recommendation device, the device comprising: an acquisition unit, configured to acquire an item modal data set and a user-item interaction data set, wherein the item modal data in the above-mentioned item modal data set corresponds to the items in the item set; a construction unit, configured to generate an initial item modal relationship graph matrix set, a user-item interaction matrix and an item second-order interaction relationship matrix based on the above-mentioned item modal data set and the above-mentioned user-item interaction data set; a first generation and execution unit, configured to generate a training sample set based on the above-mentioned user-item interaction data set, and based on the training sample set, perform the following training steps: input a pre-generated initial interaction embedding representation set, each item modal embedding representation set, an initial item modal relationship graph matrix set and a user-item interaction matrix into an initial item recommendation model to obtain a high-order user embedding vector set and a high-order item embedding vector set, wherein the initial interaction embedding representation set includes an initial user embedding vector set and an initial item embedding vector set, and the initial item recommendation model includes a first graph neural network. network submodule, a second graph neural network submodule, a multi-head attention submodule and an item embedding representation fusion layer; based on the above-mentioned high-order user embedding vector set and the above-mentioned high-order item embedding vector set, determining the interaction prediction value corresponding to each training sample in the at least one training sample included in the training sample set; through a pre-designed information constraint loss function and a recommendation loss function, based on the initial item modal relationship graph matrix set, the above-mentioned second-order interaction relationship matrix between items, the interaction prediction value corresponding to each training sample in the above-mentioned at least one training sample and the corresponding sample label, generating an item recommendation loss value, wherein the above-mentioned information constraint loss function is used to minimize the information difference between each initial item modal relationship graph matrix and the above-mentioned second-order interaction relationship matrix between items; in response to determining that the above-mentioned item recommendation loss value is less than a preset loss threshold, determining the trained initial item recommendation model as the item recommendation model; a second generating and sending unit, configured to use the above-mentioned item recommendation model to generate item recommendation information corresponding to the target user, and send the above-mentioned item recommendation information to the browsing terminal corresponding to the above-mentioned target user.

[0010] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0011] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0012] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the task-oriented multimodal item recommendation method of some embodiments of the present disclosure, the accuracy of item recommendation is improved, and it is convenient to recommend items of interest to users. Specifically, the reason why it is difficult to recommend candidate items of interest to users is that: since the above-mentioned item recommendation method is divided into two stages, namely, modal representation extraction and user interest modeling, the item modal representation output by the pre-trained model is usually not well adapted to the final interactive representation learning of users and items, which easily causes the model to fall into a local optimum, and the accuracy of item representation is also reduced. Further, it is difficult to accurately determine the candidate items of interest to users, thereby making it difficult to recommend candidate items of interest to users. Based on this, the task-oriented multimodal item recommendation method of some embodiments of the present disclosure first obtains an item modal data set and a user-item interaction data set. Among them, the item modal data in the above-mentioned item modal data set corresponds to the items in the item set. Thus, item data of different modal types representing items and user-item interaction records can be obtained, which is convenient for the subsequent learning of item modal representation constraints based on the user-item interaction records, and the generation of task-oriented item modal representation. Then, based on the above-mentioned item modality dataset and the above-mentioned user-item interaction dataset, an initial item modality relationship graph matrix set, a user-item interaction matrix, and an item second-order interaction relationship matrix are generated. Thus, the relationship between items in different item modalities and the relationship under user-item interaction, as well as the interaction relationship between users and items can be determined.Afterwards, based on the above-mentioned user-item interaction data set, a training sample set is generated, and based on the training sample set, the following training steps are performed: the pre-generated initial interaction embedding representation set, each item modal embedding representation set, the initial item modal relationship graph matrix set and the user-item interaction matrix are input into the initial item recommendation model to obtain a high-order user embedding vector set and a high-order item embedding vector set, wherein the initial interaction embedding representation set includes an initial user embedding vector set and an initial item embedding vector set, and the initial item recommendation model includes a first graph neural network submodule, a second graph neural network submodule, a multi-head attention submodule and an item embedding representation fusion layer; based on the above-mentioned high-order user embedding vector set and the above-mentioned high-order item The method comprises the following steps: a set of item embedding vectors, determining the interaction prediction value corresponding to each training sample in at least one training sample included in the training sample set; generating an item recommendation loss value based on the initial item modal relationship graph matrix set, the second-order interaction relationship matrix between the items, the interaction prediction value corresponding to each training sample in the at least one training sample and the corresponding sample label through a pre-designed information constraint loss function and a recommendation loss function, wherein the information constraint loss function is used to minimize the information difference between each initial item modal relationship graph matrix and the second-order interaction relationship matrix between the items; in response to determining that the item recommendation loss value is less than a preset loss threshold, determining the trained initial item recommendation model as the item recommendation model. Thus, an item recommendation model including high-order embedding representations of each user and high-order embedding representations of each item can be obtained, so as to facilitate the subsequent determination of each item of interest to the user based on the high-order embedding representations of the user and the item. Finally, using the item recommendation model, the item recommendation information corresponding to the target user is generated, and the item recommendation information is sent to the browsing terminal corresponding to the target user. Thus, each item of interest to the user can be recommended to the user. Therefore, the task-oriented multimodal item recommendation method of some embodiments of the present disclosure can improve the accuracy and generalization ability of the item recommendation model by effectively matching the two processes of modal representation learning and user-item interaction representation learning, uniformly learning modal representation and user-item interaction representation from the perspective of graph, and using the second-order interaction relationship between items extracted from the user-item interaction relationship to guide the learning of modal representation through a constraint mechanism, so that the accuracy of item representation is higher. Therefore, the candidate items of interest to the user can be determined more accurately, and the candidate items of interest to the user can be recommended to the user. In turn, the performance of the recommendation system can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0014] Figure 1 is a flowchart of some embodiments of the task-oriented multimodal item recommendation method according to the present disclosure;

[0015] Figure 2 is a schematic diagram of the structure of some embodiments of the task-oriented multimodal item recommendation device according to the present disclosure;

[0016] Figure 3 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0018] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0019] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0020] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0021] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0022] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0023] Figure 1 The process 100 of some embodiments of the task-oriented multimodal item recommendation method according to the present disclosure is shown. The task-oriented multimodal item recommendation method comprises the following steps:

[0024] Step 101: Obtain an item modality dataset and a user-item interaction dataset.

[0025] In some embodiments, the execution subject (e.g., computing device) of the task-oriented multimodal item recommendation method may obtain an item modal data set and a user-item interaction data set from a database through a wired connection or a wireless connection. Among them, the item modal data in the above-mentioned item modal data set may correspond one-to-one to the items in the item set. The item modal data in the above-mentioned item modal data set may include an item identifier and a modal data group. The above-mentioned item identifier may be a unique identifier of the item. The modal data in the above-mentioned modal data group may include a modal identifier and item detail data. The above-mentioned modal identifier may be a unique identifier of the item modality. The above-mentioned item modality may be the type of data describing the item. For example, the above-mentioned item modality may be, but is not limited to, one of the following: image type, text type, short video type. The above-mentioned item detail data may be the description data of the corresponding item in the corresponding modality. For example, the description data may be an image of the item, or may be text introducing the item. The user-item interaction data in the above-mentioned user-item interaction data set may represent the interaction record between the user and the item. The user-item interaction data in the above-mentioned user-item interaction data set may include, but is not limited to, a user identifier, an item identifier, and an interaction time. The above-mentioned user identifier may be a unique identifier of the user. The above interaction time may be the start time of an interaction between the user and the object. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultrawideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0026] In practice, each item usually has descriptive data in multiple different modalities.

[0027] Step 102: Based on the item modal data set and the user-item interaction data set, generate an initial item modal relationship graph matrix set, a user-item interaction matrix, and an item second-order interaction relationship matrix.

[0028] In some embodiments, the execution subject may generate an initial item modal relationship graph matrix set, a user-item interaction matrix, and an inter-item second-order interaction relationship matrix based on the item modal data set and the user-item interaction data set in various ways. The initial item modal relationship graph matrix in the initial item modal relationship graph matrix set may correspond to the item modality one-to-one. The initial item modal relationship graph matrix in the initial item modal relationship graph matrix set may be a matrix to be updated, whose element values ​​represent the degree of similarity of modal data between items. The user-item interaction matrix may be a matrix whose element values ​​represent whether there is an interaction relationship between the user and the item. The inter-item second-order interaction relationship matrix may be a matrix whose element values ​​represent the similarity relationship between items.

[0029] In some optional implementations of some embodiments, the execution subject may generate an initial item modal relationship graph matrix set, a user-item interaction matrix, and an inter-item second-order interaction relationship matrix based on the item modal data set and the user-item interaction data set through the following steps:

[0030] In the first step, a set of item modal data embedding information is generated based on the above-mentioned item modal data set through a preset data embedding pre-training model set. Among them, each item modal data embedding information may include a modal data embedding information group. Each modal data embedding information may include a modality identifier and an item data embedding vector. The above-mentioned item data embedding vector may be an embedded representation of the item detail data of the corresponding item under the corresponding item modality. The data embedding pre-training model in the above-mentioned data embedding pre-training model set may correspond one-to-one to the item modality. The data embedding pre-training model in the above-mentioned data embedding pre-training model set may be a pre-training model for embedding the item detail data.

[0031] As an example, when the item modality is of text type, the corresponding data embedding pre-training model may be a language pre-training model. The language pre-training model may be, but is not limited to, one of the following: Sentence-BERT sentence embedding model, GPT model. When the item modality is of image type or short video type, the corresponding data embedding pre-training model may be a visual pre-training model. The visual pre-training model may be, but is not limited to, one of the following: ResNet residual network model, VIT (Vision Transformer) model. The above-mentioned execution subject may perform the following steps for each item modality data in the above-mentioned item modality data set to generate the item modality data embedding information in the item modality data embedding information set:

[0032] Step 1: for each modal data included in the above-mentioned modal data of the item, the item details data included in the above-mentioned modal data is input into the target data embedding pre-training model to obtain the item data embedding vector, and the modal identifier corresponding to the above-mentioned modal data and the above-mentioned item data embedding vector are determined as modal data embedding information. The above-mentioned target data embedding pre-training model can be a data embedding pre-training model in the above-mentioned data embedding pre-training model set, whose corresponding modal identifier is the same as the modal identifier included in the above-mentioned modal data.

[0033] Step 2: The obtained modal data embedding information group and the item identifier included in the above-mentioned item modal data are determined as item modal data embedding information.

[0034] The second step is to classify and process each item data embedding vector included in the above-mentioned item modal data embedding information set to obtain a set of homomodal item data embedding vector groups. The homomodal item data embedding vector groups in the above-mentioned homomodal item data embedding vector group set may be each item data embedding vector corresponding to the same item modality. The individual item data embedding vectors included in the above-mentioned item modal data embedding information set may be classified and processed according to the modality identifier corresponding to each item data embedding vector to obtain a set of homomodal item data embedding vector groups.

[0035] The third step is to construct an initial item modal relationship graph matrix set based on the above same-modal item data embedding vector set. The initial item modal relationship graph matrix set can be constructed based on the above same-modal item data embedding vector set in various ways.

[0036] In some optional implementations of some embodiments, the execution subject may perform the following steps for each homomodal item data embedding vector group in the homomodal item data embedding vector group set to construct an initial item modal relationship graph matrix in the initial item modal relationship graph matrix set:

[0037] Step 1: determine the item modal data similarity set corresponding to the same modal item data embedding vector group. Each item modal data similarity can represent the similarity between two different items under the corresponding item modality. The item modal data similarity between each two same modal item data embedding vectors in the same modal item data embedding vector group can be determined by cosine similarity to obtain the item modal data similarity set.

[0038] Step 2: Using the items in the above item set as nodes and the item modal data similarity in the above item modal data similarity set as the weight of the edge between nodes, construct an item modal relationship graph. The above item modal relationship graph may be a fully connected graph that characterizes the modal similarity between items. The above item modal relationship graph contains a large number of unimportant connection edges.

[0039] Step three, perform graph sparsification processing on the above-mentioned item modal relationship graph to obtain a sparse item modal relationship graph. Among them, the above-mentioned sparse item modal relationship graph can be a sparse graph with fewer edges than the item modal relationship graph. First, sort the edge weights of the above-mentioned item modal relationship graph in descending order to obtain a sorted edge weight sequence. Then, the sorted edge weights that meet the preset sequence number conditions in the above-mentioned sorted edge weight sequence are determined as reference weights. Finally, according to the graph sparsification technology, the edges whose corresponding weights in the edge set corresponding to the above-mentioned item modal relationship graph are deleted, and the deleted item modal relationship graph is obtained as a sparse item modal relationship graph.

[0040] Step 4: Based on the above sparse item modal relationship graph, construct the initial item modal relationship graph matrix. First, construct the initial adjacency matrix with the items in the item set as rows and columns and 0 as the element. Then, according to the weights of the edges corresponding to the row items and column items in the above sparse item modal relationship graph, update the elements at the corresponding positions in the above initial adjacency matrix to the weight values ​​to obtain the initial item modal relationship graph matrix.

[0041] The fourth step is to determine the user set, interactive item set and user-item pair set corresponding to the above-mentioned user-item interaction data set. First, each non-repeated user in the above-mentioned user-item interaction data set is determined as a user set. Then, each non-repeated item in the above-mentioned user-item interaction data set is determined as an interactive item set. Finally, for each user-item interaction data in the above-mentioned user-item interaction data set, the user and interactive item included in the above-mentioned user-item interaction data set are determined as a user-item pair. It should be noted that the above-mentioned item set can be larger than the above-mentioned interactive item set.

[0042] The fifth step is to construct a user-item interaction matrix based on the user set, the interactive items in the interactive item set, and the user-item pair set. The elements in the user-item interaction matrix can represent whether there is an interaction between the corresponding user and the interactive item. For example, when the element in the user-item interaction matrix is ​​1, it means that there is an interaction between the corresponding user and the interactive item, and when the element is 0, it means that there is no interaction between the corresponding user and the interactive item.

[0043] Step 6: Perform a second-order relationship extraction process on the user-item interaction matrix to obtain a second-order interaction relationship matrix between items. The second-order relationship extraction process can be performed using the following formula:

[0044] S=R T ×R.

[0045] Where R represents the user-item interaction matrix. T represents the transposed matrix of the user-item interaction matrix. S represents the second-order interaction relationship matrix between items.

[0046] In practice, the above-mentioned execution entity extracts the potential interaction relationship between items from the interaction behavior pattern of users and items, which can be used to subsequently guide the update of the item modal relationship graph matrix, thereby facilitating the full combination of the two processes of user interest modeling and item modal relationship extraction.

[0047] Step 103: Generate a training sample set based on the user-item interaction dataset, and perform the following training steps based on the training sample set:

[0048] Step 1031, input the pre-generated initial interaction embedding representation set, each item modal embedding representation set, initial item modal relationship graph matrix set and user-item interaction matrix into the initial item recommendation model to obtain a high-order user embedding vector set and a high-order item embedding vector set.

[0049] In some embodiments, the execution subject may input the pre-generated initial interaction embedding representation set, each item modal embedding representation set, the initial item modal relationship graph matrix set and the user-item interaction matrix into the initial item recommendation model to obtain a high-order user embedding vector set and a high-order item embedding vector set. Among them, the initial interaction embedding representation set may include an initial user embedding vector set and an initial item embedding vector set. The initial user embedding vector in the initial user embedding vector set may correspond one-to-one to the users in the user set. The initial user embedding vector in the initial user embedding vector set may be an initialized embedding representation of the corresponding user. The initial item embedding vector in the initial item embedding vector set may be an initialized embedding representation of the corresponding item. For example, the initial user embedding vector and the initial item embedding vector may both be 32-dimensional vectors, and each dimension may be initially 0. The item modal embedding representation set in each item modal embedding representation set may correspond one-to-one to the items in the item set. The item modal embedding representation in the item modal embedding representation set corresponds one-to-one to the item modality. The item modal embedding representation set in each item modal embedding representation set may be an initialized set of embedding representations of different modal data corresponding to the item. For example, the modal embedding representation of an item can be a 32-dimensional vector, and each dimension can be initially 0. The initial item recommendation model can include a first graph neural network submodule, a second graph neural network submodule, a multi-head attention submodule, and an item embedding representation fusion layer. The above-mentioned first graph neural network submodule can be used to output an initial high-order interaction representation of a user and an item. The above-mentioned second graph neural network submodule can be used to output a high-order modal representation of an item. The graph neural network used in the above-mentioned first graph neural network submodule has a similar structure to the graph neural network used in the above-mentioned second graph neural network submodule. The above-mentioned graph neural network can be a graph convolutional network. The above-mentioned multi-head attention submodule can be used to fuse item representations under different modalities. The above-mentioned item embedding representation fusion layer can be used to fuse the interaction representation and modal representation of an item.

[0050] In some optional implementations of some embodiments, the execution subject may input the pre-generated initial interaction embedding representation set, each item modality embedding representation set, the initial item modality relationship graph matrix set, and the user-item interaction matrix into the initial item recommendation model through the following steps to obtain a high-order user embedding vector set and a high-order item embedding vector set:

[0051] In the first step, the pre-generated initial interaction embedding representation set and the user-item interaction matrix are input into the first graph neural network submodule included in the initial item recommendation model to obtain a high-order interaction embedding representation set. Among them, the above-mentioned high-order interaction embedding representation set may include a high-order user embedding vector set and an initial high-order item embedding vector set. The high-order user embedding vectors in the above-mentioned high-order user embedding vector set may correspond one-to-one to the users in the user set. The initial high-order item embedding vectors in the above-mentioned initial high-order item embedding vector set may correspond one-to-one to the items in the above-mentioned item set. Each high-order user embedding vector may be a high-order interaction representation of the corresponding user. Each initial high-order item embedding vector may be a high-order interaction representation to be updated for the corresponding item. It should be noted that the high-order interaction representation may be a high-order embedding vector generated by the target node continuously aggregating the embedding vectors from the neighboring nodes and obtaining the information of multi-hop (multi-order) neighboring nodes in the process of using the graph neural network technology to learn the embedding vectors of the nodes (users or items) in the graph.

[0052] As an example, the processing process of the first graph neural network submodule above can be expressed by the following formula:

[0053]

[0054] Where r represents the interaction between users and items. l represents the number of layers in the graph neural network. l+1 represents the l+1th layer of the graph neural network. The value range of l can be 0 to the upper limit L. L represents the total number of layers in the graph neural network. GNN(·) represents the graph neural network. and They represent the sets of interactive embedding representations of the l+1th layer and the lth layer respectively. The initial input of the graph neural network is the initial interaction embedding representation set E r .L r Represents the normalized adjacency matrix R. And D represents the degree matrix. Finally, the set of high-order interaction embedding representations of users and items can be determined by the following formula:

[0055]

[0056] Among them, the high-order interaction embedding represents the set H r Includes a collection of high-level user embedding vectors and the initial high-order item embedding vector set express u represents a user. |U| represents the number of users. represents the high-order user embedding vector corresponding to the |U|th user. i represents the item. |I| represents the number of items. express Represents the initial high-order item embedding vector corresponding to the |I|th item.

[0057] In practice, this solution can not only fully capture the potential relationship between users and items by using the information propagation mechanism of graph neural networks, but also further extract the second-order item-item interaction relationship from the user-item interaction matrix R to guide the learning of item modal representation in the modal relationship graph. Therefore, in order to learn item modal representation from the modal relationship graph, this disclosure decides to adopt the following steps:

[0058] In the second step, the pre-generated set of modal embedding representations of each item and the initial item modal relationship graph matrix set are input into the second graph neural network submodule included in the initial item recommendation model to obtain a set of high-order item modal embedding vector groups. Among them, each high-order item modal embedding vector group in the above-mentioned high-order item modal embedding vector group set corresponds one-to-one to an item in the above-mentioned item set. Each high-order item modal embedding vector in the high-order item modal embedding vector group corresponds one-to-one to an item modality. Each high-order item modal embedding vector group in the above-mentioned high-order item modal embedding vector group set can be each high-order modal representation under different item modalities of the corresponding item.

[0059] As an example, for any item modality, the processing process of the second graph neural network submodule can be expressed by the following formula:

[0060]

[0061] Among them, m represents the serial number of the item mode. and They represent the item modal representation sets of the l+1th layer and the lth layer respectively. The initial input of the graph neural network is the item modality embedding representation set E m . A m Represents the initial item modality relationship graph matrix corresponding to item modality m. Indicates A m Normalized adjacency matrix. And B represents the degree matrix. Finally, the high-order modal representation of each item can be determined by the following formula:

[0062]

[0063] Among them, H m When the item modality corresponding to each item is m item modality, the high-order modality representation of each item (high-order item modality embedding vector) should be noted that A m is a dynamic learnable matrix parameter. In each iteration of the model, the adjacency matrix A mThe updating process is similar to the construction process of the item modal relationship graph. The difference is that the present disclosure uses the high-order item modal representation H updated in each iteration process. m To construct the adjacency matrix A of the item modal relationship graph m The construction process is the same as the construction process of the initial item modal relationship graph matrix. In addition, the second graph neural network submodule can also use the high-order item modal embedding vectors under different item modalities corresponding to each item as a high-order item modal embedding vector group to obtain a high-order item modal embedding vector group set corresponding to each item.

[0064] The third step is to input the above-mentioned high-order item modal embedding vector group set into the multi-head attention submodule included in the initial item recommendation model to obtain the item multimodal embedding vector group set. Among them, each item multimodal embedding vector group in the above-mentioned item multimodal embedding vector group set can correspond one-to-one to the high-order item modal embedding vector group in the above-mentioned high-order item modal embedding vector group set. Each item multimodal embedding vector in the above-mentioned item multimodal embedding vector group set can be a vector after fusing different modal representations of the corresponding item. The item multimodal embedding vector in the above-mentioned item multimodal embedding vector group set can be generated according to the multi-head attention mechanism through the following formula:

[0065]

[0066] Where M represents the set of item modalities. |M| represents the number of item modalities. m′ represents an item modality in the M set that is different from the mth item modality, that is, m≠m′. P represents the number of attention heads. P represents the sequence number of the attention head. d represents the dimension of the item modality representation. Indicates the multimodal embedding vector of the item when the item detail data of item i is of the mth item modality. Indicates the multimodal embedding vector of the item when the item detail data of item i is of the m′th item modality. It represents the multimodal embedding vector of the item after the multi-head attention mechanism. ρ(·) represents the softmax activation function. and They represent the multi-head query and key conversion matrices in the multi-head attention mechanism, and

[0067] The fourth step is to input the above-mentioned multimodal embedding vector group of each item in the above-mentioned multimodal embedding vector group set of items and the corresponding initial high-order item embedding vector into the item embedding representation fusion layer included in the initial item recommendation model to obtain a high-order item embedding vector. Among them, the high-order item embedding vector is the embedding representation of the item after the high-order interaction representation and modal representation of the corresponding item are fused. The high-order item embedding vector corresponding to each item can be generated through the item embedding representation fusion layer according to the following formula:

[0068]

[0069] Among them, h i Represents the high-order item embedding vector corresponding to item i. Represents the initial high-order item embedding vector corresponding to item i.

[0070] It should be noted that the training samples in the above training sample set are composed of data of one user and one item. The above training sample set includes a positive training sample set and a negative training sample set. The positive training samples in the positive training sample set can represent the real interaction records between the user and the item. The negative training samples in the negative training sample set can represent that there is no real interaction record between the user and the item. The training sample set can be generated based on the user-item interaction data set by performing the following steps:

[0071] In the first step, for each user-item interaction data in the user-item interaction data set, the user identifier and the item identifier included in the user-item interaction data are determined as positive training samples.

[0072] The second step is to perform negative sampling on each of the obtained positive training samples to obtain a negative training sample set. First, for each positive training sample, each item that has not been interacted with by the user included in the positive training sample can be sampled, and each sampled item and the user included in the positive training sample form a negative training sample to obtain a negative training sample group. Then, each negative training sample in each of the obtained negative training sample groups is determined as a negative training sample set.

[0073] Step 1032: Determine an interaction prediction value corresponding to each training sample in at least one training sample included in the training sample set based on the high-order user embedding vector set and the high-order item embedding vector set.

[0074] In some embodiments, the execution subject may determine the interaction prediction value corresponding to each training sample in at least one training sample included in the training sample set based on the high-order user embedding vector set and the high-order item embedding vector set. The interaction prediction value may be a probability that indicates the possible interaction between the user and the item. The interaction prediction value corresponding to each training sample in at least one training sample included in the training sample set may be determined according to the following interaction prediction formula:

[0075]

[0076] in, represents the interaction prediction value between user u and item i. δ(·) represents the sigmoid activation function. Represents the high-order user embedding vector corresponding to user u.

[0077] Continuing, considering that in a recommendation system that integrates multimodal item information, users usually comprehensively evaluate whether to interact with the item based on the item's modal information, further, the modal connection relationship between items will also be reflected in the second-order item-item interaction relationship. Therefore, in order to obtain the potential connection between the modal connection relationship between items and the second-order item-item interaction relationship, and to inject the second-order item-item interaction information into the modal relationship graph to constrain the learning of the item modal representation by narrowing the information difference between the two matrices, the present disclosure decides to introduce the pre-designed information constraint loss function into the following model loss calculation step 1033.

[0078] Step 1033, through the pre-designed information constraint loss function and recommendation loss function, based on the initial item modal relationship graph matrix set, the second-order interaction relationship matrix between items, the interaction prediction value corresponding to each training sample in at least one training sample and the corresponding sample label, an item recommendation loss value is generated.

[0079] In some embodiments, the execution subject may generate an item recommendation loss value through a pre-designed information constraint loss function and a recommendation loss function, based on the initial item modal relationship graph matrix set, the second-order interaction relationship matrix between the items, the interaction prediction value corresponding to each training sample in the at least one training sample, and the corresponding sample label. Among them, the information constraint loss function may be used to minimize the information difference between each initial item modal relationship graph matrix and the second-order interaction relationship matrix between the items. The recommendation loss function may be used to minimize the difference between the probability distribution of the interaction prediction value and the probability distribution of the true label. The sample label may be 1 or 0. When the training sample is a positive training sample, the corresponding sample label may be 1. When the training sample is a negative training sample, the corresponding sample label may be 0. The item recommendation loss value may characterize the degree of difference between the predicted user-item interaction result and the actual user-item interaction result.

[0080] In some optional implementations of some embodiments, the execution subject may generate an item recommendation loss value based on the initial item modal relationship graph matrix set, the second-order interaction relationship matrix between items, the interaction prediction value corresponding to each training sample in the at least one training sample, and the corresponding sample label through the following steps, using a pre-designed information constraint loss function and a recommendation loss function:

[0081] The first step is to determine the first loss value between the initial item modal relationship graph matrix set and the above-mentioned second-order interaction relationship matrix between items through the information constraint loss function. The above-mentioned first loss value represents the information difference between the initial item modal relationship graph matrix and the second-order interaction relationship matrix between items under each item modality. The above-mentioned information constraint loss function can be:

[0082]

[0083] in, Represents the information constraint loss value. represents the normalized adjacency matrix of S. ||·|| F represents the Frobenius norm.

[0084] As an example, first, the execution subject may determine, for each initial item modal relationship graph matrix in the initial item modal relationship graph matrix set, an information constraint loss value between the initial item modal relationship graph matrix and the second-order interaction relationship matrix between items according to the information constraint loss function. Then, the sum of the obtained information constraint loss values ​​may be determined as the first loss value.

[0085] In addition, the present disclosure considers that KL divergence can also be used to measure the degree of difference between two probability distributions, and therefore designs another constraint mechanism, namely, by minimizing and The KL divergence between them is used to constrain the adjacency matrix of the modal relationship graph To guide the learning of item modal representation. Among them, the constraint mechanism based on KL divergence can be represented by the following information constraint loss function.

[0086] Optionally, the above information constraint loss function can also be:

[0087]

[0088] Where N is the number of items. n is the serial number of the item. KL(·||·) represents the KL divergence of two discrete distributions. Represents the adjacency matrix The row vector corresponding to the nth item in . Represents the adjacency matrix The row vector corresponding to the nth item in .

[0089] The information constraint loss functions of the above two methods and their related contents, as the inventive points of the present invention, can evaluate the modal relationship between items and the second-order interaction relationship between items in the modal relationship graph, and by minimizing the information constraint loss value, utilize the interaction representation learning of users and items in the recommendation task to constrain the modal representation learning process in the modal relationship graph, so that the generated item modal representation contains the interaction information between users and items, and finally generates a task-oriented item modal representation.

[0090] In the second step, a second loss value between the interaction prediction value corresponding to each training sample in the at least one training sample and the corresponding sample label is determined by recommending a loss function.

[0091] Among them, the above recommendation loss function can be:

[0092]

[0093] in, represents the second loss value. X represents the training sample set. u，i Represents the sample label.

[0094] The third step is to generate a recommended loss value for the item based on the first loss value and the second loss value. The recommended loss value for the item can be generated based on the first loss value and the second loss value using the following formula:

[0095]

[0096] in, Represents the item recommendation loss value. represents the regularization term. α represents the weight coefficient of the information constraint loss. β represents the weight coefficient of the regularization term.

[0097] Step 1034, in response to determining that the item recommendation loss value is less than a preset loss threshold, determining the trained initial item recommendation model as the item recommendation model.

[0098] In some embodiments, the execution subject may determine the trained initial item recommendation model as the item recommendation model in response to determining that the item recommendation loss value is less than a preset loss threshold, wherein the preset loss threshold may be a preset upper limit of the item recommendation loss value.

[0099] Optionally, the execution subject may also adjust the trainable network parameters in the initial item recommendation model in response to determining that the item recommendation loss value is greater than or equal to the preset loss threshold, and use unused training samples to form a training sample set, and use the adjusted initial item recommendation model to perform the training step again. The trainable network parameters may include an initial interaction embedding representation set and an initial item modal relationship graph matrix set.

[0100] As an example, the above-mentioned execution entity can use the Tensorflow framework to build an item recommendation model. In the item recommendation model training stage, the above-mentioned execution entity optimizes the trainable parameters of the model through the Adam optimizer, and sets the initial learning rate to 0.001. The amount of training data for each batch is set to 128, the interaction representation dimension of users and items is set to 32, and the modal representation dimension of items is also set to 32. To prevent overfitting, the present disclosure also uses dropout technology to randomly remove nodes in the graph structure with a probability of 0.3. The number of layers of the graph neural network used in the item recommendation model is set to 3. When calculating the item recommendation loss value, the weight coefficient α is set to 0.02 and β is set to 0.0001.

[0101] Step 104: Generate item recommendation information corresponding to the target user using the item recommendation model, and send the item recommendation information to the browsing terminal corresponding to the target user.

[0102] In some embodiments, the execution subject may generate item recommendation information corresponding to the target user by using the item recommendation model in various ways, and send the item recommendation information to the browsing terminal corresponding to the target user. The target user may be a user who has logged in to the item display platform on which the item recommendation model is deployed. The item display platform may be a platform for displaying various items. For example, the item display platform may be a shopping website or an e-book online reading platform. The item recommendation information may be information about various items to be recommended to the user for interactive display. The browsing terminal may be a terminal with a display screen used by the target user when logging in to the item display platform.

[0103] In some optional implementations of some embodiments, the execution subject may generate item recommendation information corresponding to the target user by using the item recommendation model through the following steps:

[0104] The first step is to determine a non-interacted item identification set corresponding to the target user based on the user-item interaction dataset and the item set, wherein the non-interacted item identification in the non-interacted item identification set may represent items that have not been interacted with by the target user.

[0105] The second step is to determine the target high-order user embedding vector corresponding to the target user based on the high-order user embedding vector set, wherein the high-order user embedding vector corresponding to the target user can be selected from the high-order user embedding vector set as the target high-order user embedding vector.

[0106] Step 3: For each non-interacted item identifier in the non-interacted item identifier set, perform the following steps:

[0107] The first sub-step is to determine the target high-order item embedding vector corresponding to the non-interacted item identifier based on the high-order item embedding vector set. The high-order item embedding vector corresponding to the non-interacted item identifier can be selected from the high-order item embedding vector set as the target high-order item embedding vector.

[0108] In the second sub-step, the interaction prediction rate is determined based on the target high-order user embedding vector corresponding to the target user and the target high-order item embedding vector corresponding to the non-interacted item identifier through the item recommendation model. The interaction prediction rate may be the probability that an interaction may occur between the user and the item. The interaction prediction rate may be determined based on the target high-order user embedding vector corresponding to the target user and the target high-order item embedding vector corresponding to the non-interacted item identifier through the interaction prediction formula in the item recommendation model.

[0109] In the third sub-step, the non-interacted item identifier and the interaction prediction rate are determined as candidate item information.

[0110] In the fourth step, the determined candidate item information is sorted in descending order to obtain a candidate item information sequence. The determined candidate item information can be sorted in descending order by a preset sorting algorithm to obtain a candidate item information sequence. For example, the sorting algorithm can be, but is not limited to, one of: bubble sort and insertion sort.

[0111] The fifth step is to generate the item recommendation information corresponding to the target user based on the above-mentioned candidate item information sequence. First, the candidate item information whose corresponding sequence number is less than or equal to the preset sequence number threshold can be selected from the above-mentioned candidate item information sequence as the target candidate item information to obtain the target candidate item information group. Among them, the above-mentioned preset sequence number threshold can be the upper limit value of the preset sequence number. Then, for each target candidate item information in the above-mentioned target candidate item information group, the target item information corresponding to the above-mentioned target candidate item information is determined as the item information to be recommended. Among them, the above-mentioned target item information can be the item details information of the item corresponding to the uninteracted item identifier in the target candidate item information. Finally, each item information to be recommended is determined as the item recommendation information.

[0112] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the task-oriented multimodal item recommendation method of some embodiments of the present disclosure, the accuracy of item recommendation is improved, and it is convenient to recommend items of interest to users. Specifically, the reason why it is difficult to recommend candidate items of interest to users is that: since the above-mentioned item recommendation method is divided into two stages, namely, modal representation extraction and user interest modeling, the item modal representation output by the pre-trained model is usually not well adapted to the final interactive representation learning of users and items, which easily causes the model to fall into a local optimum, and the accuracy of item representation is also reduced. Further, it is difficult to accurately determine the candidate items of interest to users, thereby making it difficult to recommend candidate items of interest to users. Based on this, the task-oriented multimodal item recommendation method of some embodiments of the present disclosure first obtains an item modal data set and a user-item interaction data set. Among them, the item modal data in the above-mentioned item modal data set corresponds to the items in the item set. Thus, item data of different modal types representing items and user-item interaction records can be obtained, which is convenient for the subsequent learning of item modal representation constraints based on the user-item interaction records, and the generation of task-oriented item modal representation. Then, based on the above-mentioned item modality dataset and the above-mentioned user-item interaction dataset, an initial item modality relationship graph matrix set, a user-item interaction matrix, and an item second-order interaction relationship matrix are generated. Thus, the relationship between items in different item modalities and the relationship under user-item interaction, as well as the interaction relationship between users and items can be determined.Afterwards, based on the above-mentioned user-item interaction data set, a training sample set is generated, and based on the training sample set, the following training steps are performed: the pre-generated initial interaction embedding representation set, each item modal embedding representation set, the initial item modal relationship graph matrix set and the user-item interaction matrix are input into the initial item recommendation model to obtain a high-order user embedding vector set and a high-order item embedding vector set, wherein the initial interaction embedding representation set includes an initial user embedding vector set and an initial item embedding vector set, and the initial item recommendation model includes a first graph neural network submodule, a second graph neural network submodule, a multi-head attention submodule and an item embedding representation fusion layer; based on the above-mentioned high-order user embedding vector set and the above-mentioned high-order item The method comprises the following steps: a set of item embedding vectors, determining the interaction prediction value corresponding to each training sample in at least one training sample included in the training sample set; generating an item recommendation loss value based on the initial item modal relationship graph matrix set, the second-order interaction relationship matrix between the items, the interaction prediction value corresponding to each training sample in the at least one training sample and the corresponding sample label through a pre-designed information constraint loss function and a recommendation loss function, wherein the information constraint loss function is used to minimize the information difference between each initial item modal relationship graph matrix and the second-order interaction relationship matrix between the items; in response to determining that the item recommendation loss value is less than a preset loss threshold, determining the trained initial item recommendation model as the item recommendation model. Thus, an item recommendation model including high-order embedding representations of each user and high-order embedding representations of each item can be obtained, so as to facilitate the subsequent determination of each item of interest to the user based on the high-order embedding representations of the user and the item. Finally, using the item recommendation model, the item recommendation information corresponding to the target user is generated, and the item recommendation information is sent to the browsing terminal corresponding to the target user. Thus, each item of interest to the user can be recommended to the user. Therefore, the task-oriented multimodal item recommendation method of some embodiments of the present disclosure can improve the accuracy and generalization ability of the item recommendation model by effectively matching the two processes of modal representation learning and user-item interaction representation learning, uniformly learning modal representation and user-item interaction representation from the perspective of graph, and using the second-order interaction relationship between items extracted from the user-item interaction relationship to guide the learning of modal representation through a constraint mechanism, so that the accuracy of item representation is higher. Therefore, the candidate items of interest to the user can be determined more accurately, and the candidate items of interest to the user can be recommended to the user. In turn, the performance of the recommendation system can be improved.

[0113] Further references Figure 2 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a task-oriented multimodal item recommendation device. These device embodiments are similar to Figure 1Corresponding to the method embodiments shown, the task-oriented multimodal item recommendation device 200 can be specifically applied to various electronic devices.

[0114] like Figure 2 As shown, the task-oriented multimodal item recommendation device 200 of some embodiments includes: an acquisition unit 201, a construction unit 202, a first generation and execution unit 203 and a second generation and sending unit 204. Among them, the acquisition unit 201 is configured to acquire an item modal data set and a user-item interaction data set, wherein the item modal data in the above-mentioned item modal data set corresponds to the items in the item set; the construction unit 202 is configured to generate an initial item modal relationship graph matrix set, a user-item interaction matrix and an inter-item second-order interaction relationship matrix based on the above-mentioned item modal data set and the above-mentioned user-item interaction data set; the first generation and execution unit 203 is configured to generate a training sample set based on the above-mentioned user-item interaction data set, and based on the training sample set, perform the following training steps: input the pre-generated initial interaction embedding representation set, each item modal embedding representation set, the initial item modal relationship graph matrix set and the user-item interaction matrix into the initial item recommendation model to obtain a high-order user embedding vector set and a high-order item embedding vector set, wherein the initial interaction embedding representation set includes an initial user embedding vector set and an initial item embedding vector set, and the initial item recommendation model includes a first graph neural network submodule, a second graph neural network submodule block, a multi-head attention submodule and an item embedding representation fusion layer; based on the above-mentioned high-order user embedding vector set and the above-mentioned high-order item embedding vector set, determining the interaction prediction value corresponding to each training sample in the at least one training sample included in the training sample set; through a pre-designed information constraint loss function and a recommendation loss function, based on the initial item modal relationship graph matrix set, the above-mentioned second-order interaction relationship matrix between items, the interaction prediction value corresponding to each training sample in the above-mentioned at least one training sample and the corresponding sample label, generating an item recommendation loss value, wherein the above-mentioned information constraint loss function is used to minimize the information difference between each initial item modal relationship graph matrix and the above-mentioned second-order interaction relationship matrix between items; in response to determining that the above-mentioned item recommendation loss value is less than a preset loss threshold, determining the trained initial item recommendation model as the item recommendation model; the second generating and sending unit 204 is configured to use the above-mentioned item recommendation model to generate item recommendation information corresponding to the target user, and send the above-mentioned item recommendation information to the browsing terminal corresponding to the above-mentioned target user.

[0115] It can be understood that the various units recorded in the task-oriented multimodal item recommendation device 200 are similar to the reference Figure 1Therefore, the operations, features and beneficial effects described above for the method are also applicable to the task-oriented multimodal item recommendation device 200 and the units included therein, and will not be described in detail here.

[0116] Further references Figure 3 , which shows a structural schematic diagram of an electronic device 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0117] like Figure 3 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0118] Typically, the following devices may be connected to the I / O interface 305: input devices 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 309. The communication devices 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0119] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0120] It should be noted that the computer-readable medium in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0121] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0122] The above-mentioned computer-readable medium may be included in the above-mentioned device; or it may exist independently and not be installed in the electronic device. The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: obtains an item modal data set and a user-item interaction data set, wherein the item modal data in the above-mentioned item modal data set corresponds to the items in the item set; based on the above-mentioned item modal data set and the above-mentioned user-item interaction data set, generates an initial item modal relationship graph matrix set, a user-item interaction matrix and an item second-order interaction relationship matrix; based on the above-mentioned user-item interaction data set, generates a training sample set, and based on the training sample set, performs the following training steps: input the pre-generated initial interaction embedding representation set, each item modal embedding representation set, the initial item modal relationship graph matrix set and the user-item interaction matrix into the initial item recommendation model to obtain a high-order user embedding vector set and a high-order item embedding vector set, wherein the initial interaction embedding representation set includes an initial user embedding vector set and an initial item embedding vector set, and the initial item recommendation model includes a first graph neural network sub-model. module, a second graph neural network submodule, a multi-head attention submodule and an item embedding representation fusion layer; based on the above-mentioned high-order user embedding vector set and the above-mentioned high-order item embedding vector set, determine the interaction prediction value corresponding to each training sample in the at least one training sample included in the training sample set; through a pre-designed information constraint loss function and a recommendation loss function, based on the initial item modal relationship graph matrix set, the above-mentioned second-order interaction relationship matrix between items, the interaction prediction value corresponding to each training sample in the above-mentioned at least one training sample and the corresponding sample label, generate an item recommendation loss value, wherein the above-mentioned information constraint loss function is used to minimize the information difference between each initial item modal relationship graph matrix and the above-mentioned second-order interaction relationship matrix between items; in response to determining that the above-mentioned item recommendation loss value is less than a preset loss threshold, determine the trained initial item recommendation model as the item recommendation model; using the above-mentioned item recommendation model, generate item recommendation information corresponding to the target user, and send the above-mentioned item recommendation information to the browsing terminal corresponding to the above-mentioned target user.

[0123] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0124] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0125] The units described in some embodiments of the present disclosure may be implemented by software or hardware. The described units may also be provided in a processor, for example, may be described as: a processor comprising: an acquisition unit, a construction unit, a first generation and execution unit, and a second generation and sending unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, the acquisition unit may also be described as a "unit for acquiring item modal data sets and user-item interaction data sets".

[0126] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0127] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A task-oriented multimodal item recommendation method, comprising: Acquire an item modality dataset and a user-item interaction dataset, wherein the item modality data in the item modality dataset corresponds to items in the item set; Based on the item modality dataset and the user-item interaction dataset, generate an initial item modality relationship graph matrix set, a user-item interaction matrix, and an item second-order interaction relationship matrix; Based on the user-item interaction dataset, a training sample set is generated, and based on the training sample set, the following training steps are performed: Input the pre-generated initial interaction embedding representation set, each item modal embedding representation set, the initial item modal relationship graph matrix set and the user-item interaction matrix into the initial item recommendation model to obtain a high-order user embedding vector set and a high-order item embedding vector set, wherein the initial interaction embedding representation set includes an initial user embedding vector set and an initial item embedding vector set, and the initial item recommendation model includes a first graph neural network submodule, a second graph neural network submodule, a multi-head attention submodule and an item embedding representation fusion layer; Determine, based on the high-order user embedding vector set and the high-order item embedding vector set, an interaction prediction value corresponding to each training sample in at least one training sample included in the training sample set; Generate an item recommendation loss value based on an initial item modal relationship graph matrix set, the second-order interaction relationship matrix between items, an interaction prediction value corresponding to each training sample in the at least one training sample, and a corresponding sample label through a pre-designed information constraint loss function and a recommendation loss function, wherein the information constraint loss function is used to minimize the information difference between each initial item modal relationship graph matrix and the second-order interaction relationship matrix between items; In response to determining that the item recommendation loss value is less than a preset loss threshold, determining the trained initial item recommendation model as the item recommendation model; The item recommendation model is used to generate item recommendation information corresponding to the target user, and the item recommendation information is sent to the browsing terminal corresponding to the target user.

2. The method according to claim 1, wherein: The method further comprises: In response to determining that the item recommendation loss value is greater than or equal to the preset loss threshold, adjusting the trainable network parameters in the initial item recommendation model, and using unused training samples to form a training sample set, and using the adjusted initial item recommendation model to perform the training step again, wherein the trainable network parameters include an initial interaction embedding representation set and an initial item modal relationship graph matrix set.

3. The method according to claim 1, wherein: The generating, based on the item modality dataset and the user-item interaction dataset, an initial item modality relationship graph matrix set, a user-item interaction matrix and an item second-order interaction relationship matrix comprises: Generate an item modal data embedding information set based on the item modal data set by using a preset data embedding pre-training model set, wherein each item modal data embedding information includes a modal data embedding information group, and each modal data embedding information includes a modality identifier and an item data embedding vector; Classifying and processing each item data embedding vector included in the item modal data embedding information set to obtain a same-modal item data embedding vector group set; Based on the same-modality item data embedding vector group set, construct an initial item modality relationship graph matrix set; Determine a user set, an interaction item set, and a user-item pair set corresponding to the user-item interaction dataset; Constructing a user-item interaction matrix based on the user set and the user-item pair set, taking the users in the user set as rows, taking the interactive items in the interactive item set as columns, and constructing a user-item interaction matrix based on the user-item pair set; The user-item interaction matrix is ​​subjected to an item second-order relationship extraction process to obtain an inter-item second-order interaction relationship matrix.

4. The method according to claim 3, wherein: The constructing an initial item modal relationship graph matrix set based on the same-modal item data embedding vector set includes: For each homomodal item data embedding vector group in the homomodal item data embedding vector group set, the following steps are performed: Determine a set of item modal data similarities corresponding to the same-modal item data embedding vector group, wherein each item modal data similarity represents the degree of similarity of the corresponding modalities between two different items; Taking items in the item set as nodes and item modal data similarities in the item modal data similarity set as weights of edges between nodes, constructing an item modal relationship graph; Performing graph sparse processing on the item modal relationship graph to obtain a sparse item modal relationship graph; Based on the sparse item modal relationship graph, an initial item modal relationship graph matrix is ​​constructed.

5. The method according to any one of claims 1 to 4, wherein: The pre-generated initial interaction embedding representation set, each item modality embedding representation set, initial item modality relationship graph matrix set and user-item interaction matrix are input into the initial item recommendation model to obtain a high-order user embedding vector set and a high-order item embedding vector set, including: Inputting the pre-generated initial interaction embedding representation set and the user-item interaction matrix into the first graph neural network submodule included in the initial item recommendation model to obtain a high-order interaction embedding representation set, wherein the high-order interaction embedding representation set includes a high-order user embedding vector set and an initial high-order item embedding vector set, and the initial high-order item embedding vector in the initial high-order item embedding vector set corresponds to the items in the item set; Inputting the pre-generated set of modal embedding representations of each item and the initial item modal relationship graph matrix set into the second graph neural network submodule included in the initial item recommendation model to obtain a set of high-order item modal embedding vector groups, wherein each high-order item modal embedding vector group in the set of high-order item modal embedding vector groups corresponds to an item in the item set; Inputting the high-order item modal embedding vector set into the multi-head attention submodule included in the initial item recommendation model to obtain an item multimodal embedding vector set, wherein each item multimodal embedding vector set in the item multimodal embedding vector set corresponds to a high-order item modal embedding vector set in the high-order item modal embedding vector set; For each item multimodal embedding vector group in the item multimodal embedding vector group set, the item multimodal embedding vector group and the corresponding initial high-order item embedding vector are input into the item embedding representation fusion layer included in the initial item recommendation model to obtain a high-order item embedding vector.

6. The method according to claim 1, wherein: The step of using the item recommendation model to generate item recommendation information corresponding to the target user includes: Determine a non-interacted item identification set corresponding to the target user based on the user-item interaction dataset and the item set; Determining a target high-order user embedding vector corresponding to the target user based on the high-order user embedding vector set; For each non-interacted item identifier in the non-interacted item identifier set, perform the following steps: Determining a target high-order item embedding vector corresponding to the non-interacted item identifier based on the high-order item embedding vector set; Determining an interaction prediction rate based on a target high-order user embedding vector corresponding to the target user and a target high-order item embedding vector corresponding to the non-interacted item identifier by the item recommendation model; Determining the non-interacted item identifier and the interaction prediction rate as candidate item information; Arrange the determined candidate item information in descending order to obtain a candidate item information sequence; Based on the candidate item information sequence, item recommendation information corresponding to the target user is generated.

7. A task-oriented multimodal item recommendation device, comprising: an acquisition unit, configured to acquire an item modality data set and a user-item interaction data set, wherein the item modality data in the item modality data set corresponds to an item in the item set; A construction unit, configured to generate an initial item modal relationship graph matrix set, a user-item interaction matrix, and an inter-item second-order interaction relationship matrix based on the item modal data set and the user-item interaction data set; The first generating and executing unit is configured to generate a training sample set based on the user-item interaction dataset, and execute the following training steps based on the training sample set: Input the pre-generated initial interaction embedding representation set, each item modal embedding representation set, the initial item modal relationship graph matrix set and the user-item interaction matrix into the initial item recommendation model to obtain a high-order user embedding vector set and a high-order item embedding vector set, wherein the initial interaction embedding representation set includes an initial user embedding vector set and an initial item embedding vector set, and the initial item recommendation model includes a first graph neural network submodule, a second graph neural network submodule, a multi-head attention submodule and an item embedding representation fusion layer; Determine, based on the high-order user embedding vector set and the high-order item embedding vector set, an interaction prediction value corresponding to each training sample in at least one training sample included in the training sample set; Generate an item recommendation loss value based on an initial item modal relationship graph matrix set, the second-order interaction relationship matrix between items, an interaction prediction value corresponding to each training sample in the at least one training sample, and a corresponding sample label through a pre-designed information constraint loss function and a recommendation loss function, wherein the information constraint loss function is used to minimize the information difference between each initial item modal relationship graph matrix and the second-order interaction relationship matrix between items; In response to determining that the item recommendation loss value is less than a preset loss threshold, determining the trained initial item recommendation model as the item recommendation model; The second generating and sending unit is configured to generate item recommendation information corresponding to the target user by using the item recommendation model, and send the item recommendation information to the browsing terminal corresponding to the target user.

8. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images