Training method, recommendation method and related products

Abstract

The invention relates to the technical field of deep learning, and particularly discloses a training method, a recommendation method and a related product, and the training method comprises the steps: obtaining an initial object representation of a target object and an initial object representation of a target object; based on an interaction relationship between the target object and the target article and an article multi-modal similarity relationship between different target articles, updating the initial article representation to obtain a first updated article representation, the article multi-modal similarity relationship comprising a similarity relationship determined based on the representation of the target article in at least one predetermined modal; based on the interaction relationship and an object multi-modal similarity relationship between different target objects, updating the initial object representation to obtain a first updated object representation, the object multi-modal similarity relationship comprising a similarity relationship determined based on the representation of the target object in at least one predetermined modal; and training a target model based on the first updated article characterization, the first updated object characterization and the article multi-modal characterization.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a training method, a recommendation method and related products. Background Art

[0002] By recommending items that the object is interested in to the object, the items that the object is interested in can be presented to the object. Thanks to its powerful performance, deep learning models are increasingly being used in recommendation scenarios. Specifically, after obtaining item representations based on deep learning models and / or object representations based on deep learning models, it can be determined based on the item representations and / or object representations whether the object is interested in the item, and further it can be determined whether the item is an item to be recommended to the object. Therefore, how to train a deep learning model for obtaining item representations and / or object representations is of great significance. Summary of the invention

[0003] The present application provides a training method, a recommendation method and related products, wherein the related products include a recommendation device, a training device, an electronic device, a computer-readable storage medium and a computer program product.

[0004] In a first aspect, a training method is provided, the method comprising:

[0005] obtaining an initial object representation of a target object and an initial item representation of a target item;

[0006] Based on the interactive relationship between the target object and the target item, and the item multimodal similarity relationship between different target items, the initial item representation is updated to obtain a first updated item representation, wherein the item multimodal similarity relationship includes a similarity relationship determined based on the representation of the target item in at least one predetermined modality;

[0007] Based on the interaction relationship and the object multimodal similarity relationship between different target objects, the initial object representation is updated to obtain a first updated object representation, wherein the object multimodal similarity relationship includes a similarity relationship determined based on the representation of the target object in the at least one predetermined modality;

[0008] A target model is trained based on the first updated item representation, the first updated object representation and the multimodal representation of the item, wherein the multimodal representation of the item includes a representation of the target item in at least one predetermined modality, and the target model is used to obtain a target item representation of the target item and / or a target object representation of the target object.

[0009] In combination with any embodiment of the present application, the updating of the initial item representation based on the interactive relationship between the target object and the target item, and the multimodal similarity relationship between different target items, to obtain a first updated item representation includes:

[0010] Acquire first graph data, the first graph data comprising a first object node and a first item node, information carried by the first object node comprising the initial object representation, information carried by the first item node comprising the initial item representation, and edges between nodes in the first graph data being determined based on the interaction relationship;

[0011] Acquire second graph data, where the second graph data includes a second item node, information carried by the second item node includes the initial item representation, and edges between nodes in the second graph data are determined based on the multimodal similarity relationship of the items;

[0012] Processing the first graph data using a graph convolutional network to obtain processed first graph data;

[0013] Processing the second graph data using the graph convolutional network to obtain processed second graph data;

[0014] The first updated item representation is obtained based on the representation carried by the first item node in the processed first graph data and the representation carried by the second item node in the processed second graph data, wherein the first item node and the second item node are both nodes corresponding to the target item.

[0015] In combination with any implementation manner of the present application, the at least one predetermined modality includes a text modality and a visual modality, and the acquiring the second image data includes:

[0016] Acquire a first initial visual representation of the target object and a first initial textual representation of the target object;

[0017] Determining similarities between the first initial visual representations of different target objects to obtain a first visual similarity;

[0018] Determining similarities between the first initial text representations of different target objects to obtain first text similarities;

[0019] Based on the first visual similarity and the first textual similarity, edges of the second graph data are determined.

[0020] In combination with any embodiment of the present application, the updating of the initial object representation based on the interaction relationship and the object multimodal similarity relationship between different target objects to obtain a first updated object representation includes:

[0021] Acquire third graph data, the third graph data including a second object node, information carried by the second object node including the initial object representation, and edges of the nodes in the third graph data are determined based on the multimodal similarity relationship of the objects;

[0022] Processing the third graph data using the graph convolutional network to obtain processed third graph data;

[0023] The first updated object representation is obtained based on the representation carried by the first object node in the processed first graph data and the representation carried by the second object node in the processed third graph data, and the first object node and the second object node are both nodes corresponding to the target object.

[0024] In combination with any implementation manner of the present application, the obtaining of the third image data includes:

[0025] Acquire a second initial visual representation of the target object and a second initial textual representation of the target object, wherein the second initial visual representation is obtained based on a visual representation of an object that has interacted with the target object, and the second initial textual representation is obtained based on a textual representation of the object that has interacted with the target object;

[0026] Determining the similarity between the second initial visual representations of different target objects to obtain a second visual similarity;

[0027] Determining the similarity between the second initial text representations of different target objects to obtain a second text similarity;

[0028] Based on the second visual similarity and the second textual similarity, edges of the third graph data are determined.

[0029] In combination with any implementation manner of the present application, the processing the first graph data using a graph convolutional network to obtain the processed first graph data includes:

[0030] The graph convolutional network is used to process nodes connected by some edges in the first graph data to obtain the processed first graph data.

[0031] In combination with any embodiment of the present application, the training of the target model based on the first updated item representation, the first updated object representation, and the item multimodal representation includes:

[0032] determining an interaction similarity between the first updated item representation and the first updated object representation;

[0033] Based on the interaction similarity, an interaction loss is obtained; when the target item and the target object have interacted before, the interaction loss is negatively correlated with the interaction similarity; when the target item and the target object have not interacted before, the interaction loss is positively correlated with the interaction similarity;

[0034] The target model is trained based on the interaction loss and the multimodal representation of the item.

[0035] In combination with any embodiment of the present application, the multimodal representation of the object includes a first initial visual representation of the target object and a first initial text representation of the target object, and training the target model based on the interaction loss and the multimodal representation of the object includes:

[0036] Using the target model, respectively perform a first conversion on the first initial visual representation, the first initial text representation, the first updated item representation, and the first updated object representation to obtain a first converted visual representation, a first converted text representation, a first converted item representation, and a first converted object representation, wherein the first initial visual representation corresponds to the first converted visual representation, the first initial text representation corresponds to the first converted text representation, the first updated item representation corresponds to the first converted item representation, and the first updated object representation corresponds to the first converted object representation;

[0037] Using the target model, a second conversion is performed on the first initial visual representation, the first initial text representation, the first updated item representation, and the first updated object representation, respectively, to obtain a second converted visual representation, a second converted text representation, a second converted item representation, and a second converted object representation, wherein the first conversion and the second conversion are different conversion processes, the first initial visual representation corresponds to the second converted visual representation, the first initial text representation corresponds to the second converted text representation, the first updated item representation corresponds to the second converted item representation, and the first updated object representation corresponds to the second converted object representation;

[0038] Obtaining a multi-view loss based on the first converted visual representation, the first converted text representation, the first converted item representation, the first converted object representation, the second converted visual representation, the second converted text representation, the second converted item representation, and the second converted object representation;

[0039] The target model is trained based on the interaction loss and the multi-view loss.

[0040] In combination with any embodiment of the present application, the multi-view loss includes one or more of the following: a first multi-view loss, a second multi-view loss, and a third multi-view loss;

[0041] The first multi-view loss is obtained based on the first transformed visual representation, the first transformed textual representation, the second transformed visual representation, and the second transformed textual representation;

[0042] The second multi-view loss is obtained based on the first converted item representation, the first converted object representation, the second converted item representation and the second converted object representation;

[0043] The third multi-view loss is obtained based on the first converted visual representation, the first converted text representation and the second converted item representation.

[0044] In combination with any embodiment of the present application, before using the target model to perform the first conversion and the second conversion on the first initial visual representation, the first initial text representation, the first updated item representation and the first updated object representation, respectively, the first initial visual representation, the first initial text representation, the first updated item representation and the first updated object representation are respectively converted via the conversion module of the target model so that the dimensions of the converted representations are the same.

[0045] In combination with any embodiment of the present application, the method further includes:

[0046] In the process of training the target model, updating the multimodal representation of the object to obtain an updated multimodal representation;

[0047] Obtaining a modality consistency loss based on a difference between the updated multimodal representation and the multimodal features of the item;

[0048] The target model is trained based on the modality consistency loss.

[0049] In a second aspect, a recommendation method is provided, the method comprising:

[0050] Acquire a target item representation of a target item and a target object representation of a target object, wherein the target item representation and the target object representation are obtained by processing an updated item representation and an updated object representation using at least part of a module of a target model, wherein the target model is trained according to the first aspect and any embodiment thereof, and the updated item representation and the updated object representation are updated during the training process of the target model;

[0051] Determining a target similarity between the target item representation and the target object representation;

[0052] When the target similarity is greater than or equal to the recommendation threshold, the target item is determined to be an item recommended to the target object.

[0053] In a third aspect, a training device is provided, the training device comprising:

[0054] an acquisition unit, configured to acquire an initial object representation of a target object and an initial item representation of a target item;

[0055] an updating unit, configured to update the initial item representation based on the interaction relationship between the target object and the target item, and the item multimodal similarity relationship between different target items, to obtain a first updated item representation, wherein the item multimodal similarity relationship includes a similarity relationship determined based on the representation of the target item in at least one predetermined modality;

[0056] The updating unit is configured to update the initial object representation based on the interaction relationship and the object multimodal similarity relationship between different target objects to obtain a first updated object representation, wherein the object multimodal similarity relationship includes a similarity relationship determined based on the representation of the target object in the at least one predetermined modality;

[0057] A training unit is used to train a target model based on the first updated item representation, the first updated object representation and the multimodal representation of the item, wherein the multimodal representation of the item includes the representation of the target item in at least one predetermined modality, and the target model is used to obtain a target item representation of the target item and / or a target object representation of the target object.

[0058] In combination with any implementation manner of the present application, the updating unit is specifically used for:

[0059] Acquire first graph data, the first graph data comprising a first object node and a first item node, information carried by the first object node comprising the initial object representation, information carried by the first item node comprising the initial item representation, and edges between nodes in the first graph data being determined based on the interaction relationship;

[0060] Acquire second graph data, where the second graph data includes a second item node, information carried by the second item node includes the initial item representation, and edges between nodes in the second graph data are determined based on the multimodal similarity relationship of the items;

[0061] Processing the first graph data using a graph convolutional network to obtain processed first graph data;

[0062] Processing the second graph data using the graph convolutional network to obtain processed second graph data;

[0063] The first updated item representation is obtained based on the representation carried by the first item node in the processed first graph data and the representation carried by the second item node in the processed second graph data, wherein the first item node and the second item node are both nodes corresponding to the target item.

[0064] In combination with any implementation manner of the present application, the at least one predetermined modality includes a text modality and a visual modality, and the acquisition unit is specifically configured to:

[0065] Acquire a first initial visual representation of the target object and a first initial textual representation of the target object;

[0066] Determining similarities between the first initial visual representations of different target objects to obtain a first visual similarity;

[0067] Determining similarities between the first initial text representations of different target objects to obtain first text similarities;

[0068] Based on the first visual similarity and the first textual similarity, edges of the second graph data are determined.

[0069] In combination with any implementation manner of the present application, the updating unit is specifically used for:

[0070] Acquire third graph data, the third graph data including a second object node, information carried by the second object node including the initial object representation, and edges of the nodes in the third graph data are determined based on the multimodal similarity relationship of the objects;

[0071] Processing the third graph data using the graph convolutional network to obtain processed third graph data;

[0072] The first updated object representation is obtained based on the representation carried by the first object node in the processed first graph data and the representation carried by the second object node in the processed third graph data, and the first object node and the second object node are both nodes corresponding to the target object.

[0073] In combination with any implementation manner of the present application, the acquisition unit is specifically used to:

[0074] Acquire a second initial visual representation of the target object and a second initial textual representation of the target object, wherein the second initial visual representation is obtained based on a visual representation of an object that has interacted with the target object, and the second initial textual representation is obtained based on a textual representation of the object that has interacted with the target object;

[0075] Determining the similarity between the second initial visual representations of different target objects to obtain a second visual similarity;

[0076] Determining the similarity between the second initial text representations of different target objects to obtain a second text similarity;

[0077] Based on the second visual similarity and the second textual similarity, edges of the third graph data are determined.

[0078] In combination with any implementation manner of the present application, the updating unit is specifically used for:

[0079] The graph convolutional network is used to process nodes connected by some edges in the first graph data to obtain the processed first graph data.

[0080] In conjunction with any implementation manner of the present application, the training unit is specifically used for:

[0081] determining an interaction similarity between the first updated item representation and the first updated object representation;

[0082] Based on the interaction similarity, an interaction loss is obtained; when the target item and the target object have interacted before, the interaction loss is negatively correlated with the interaction similarity; when the target item and the target object have not interacted before, the interaction loss is positively correlated with the interaction similarity;

[0083] The target model is trained based on the interaction loss and the multimodal representation of the item.

[0084] In conjunction with any embodiment of the present application, the multimodal representation of the object includes a first initial visual representation of the target object and a first initial text representation of the target object, and the training unit is specifically used to:

[0085] Using the target model, respectively perform a first conversion on the first initial visual representation, the first initial text representation, the first updated item representation, and the first updated object representation to obtain a first converted visual representation, a first converted text representation, a first converted item representation, and a first converted object representation, wherein the first initial visual representation corresponds to the first converted visual representation, the first initial text representation corresponds to the first converted text representation, the first updated item representation corresponds to the first converted item representation, and the first updated object representation corresponds to the first converted object representation;

[0086] Using the target model, a second conversion is performed on the first initial visual representation, the first initial text representation, the first updated item representation, and the first updated object representation, respectively, to obtain a second converted visual representation, a second converted text representation, a second converted item representation, and a second converted object representation, wherein the first conversion and the second conversion are different conversion processes, the first initial visual representation corresponds to the second converted visual representation, the first initial text representation corresponds to the second converted text representation, the first updated item representation corresponds to the second converted item representation, and the first updated object representation corresponds to the second converted object representation;

[0087] Obtaining a multi-view loss based on the first converted visual representation, the first converted text representation, the first converted item representation, the first converted object representation, the second converted visual representation, the second converted text representation, the second converted item representation, and the second converted object representation;

[0088] The target model is trained based on the interaction loss and the multi-view loss.

[0089] In combination with any embodiment of the present application, the multi-view loss includes one or more of the following: a first multi-view loss, a second multi-view loss, and a third multi-view loss;

[0090] The first multi-view loss is obtained based on the first transformed visual representation, the first transformed textual representation, the second transformed visual representation, and the second transformed textual representation;

[0091] The second multi-view loss is obtained based on the first converted item representation, the first converted object representation, the second converted item representation and the second converted object representation;

[0092] The third multi-view loss is obtained based on the first converted visual representation, the first converted text representation and the second converted item representation.

[0093] In combination with any embodiment of the present application, before using the target model to perform the first conversion and the second conversion on the first initial visual representation, the first initial text representation, the first updated item representation and the first updated object representation, respectively, the first initial visual representation, the first initial text representation, the first updated item representation and the first updated object representation are respectively converted via the conversion module of the target model so that the dimensions of the converted representations are the same.

[0094] In combination with any embodiment of the present application, the updating unit is further used to update the multimodal representation of the object during the training of the target model to obtain an updated multimodal representation;

[0095] The training unit is further configured to obtain a modality consistency loss based on a difference between the updated multimodal representation and the multimodal feature of the object;

[0096] The training unit is further used to train the target model based on the modality consistency loss.

[0097] In a fourth aspect, a recommendation device is provided, the recommendation device comprising:

[0098] an acquisition unit, configured to acquire a target item representation of a target item and a target object representation of a target object, wherein the target item representation and the target object representation are obtained by processing an updated item representation and an updated object representation using at least part of a module of a target model, wherein the target model is trained according to the first aspect and any one of its embodiments, and the updated item representation and the updated object representation are updated during the training process of the target model;

[0099] A determination unit, configured to determine a target similarity between the target item representation and the target object representation;

[0100] The determining unit is configured to determine that the target item is an item recommended to the target object when the target similarity is greater than or equal to a threshold.

[0101] In a fifth aspect, an electronic device is provided, comprising: a processor and a memory, the memory being used to store computer program code, the computer program code comprising computer instructions, and when the processor executes the computer instructions, the electronic device executes the first aspect and any one of its implementations described above; when the processor executes the computer instructions, the electronic device may execute the second aspect described above.

[0102] In a sixth aspect, another electronic device is provided, comprising: a processor, a sending device, an input device, an output device and a memory, wherein the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the first aspect and any one of its implementation modes described above; when the processor executes the computer instructions, the electronic device may execute the second aspect described above.

[0103] In the seventh aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to execute the first aspect and any one of its implementation modes described above; when the program instructions are executed by the processor, the processor is caused to execute the second aspect described above.

[0104] In an eighth aspect, a computer program product is provided, which includes a computer program or instructions, and when the computer program or instructions are run on a computer, the computer is caused to execute the above-mentioned first aspect and any one of its embodiments; when the computer program or instructions are run on a computer, the computer is caused to execute the above-mentioned second aspect.

[0105] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application.

[0106] In the present application, after acquiring the initial object representation of the target object and the initial item representation of the target item, the training device updates the initial item representation based on the interaction relationship between the target object and the target item, and the multimodal similarity relationship between different target items, to obtain a first updated item representation, wherein the multimodal similarity relationship of items includes a similarity relationship determined based on the representation of the target item in at least one predetermined modality. By updating the initial item representation based on the above-mentioned interaction relationship, the initial item representation of the target item can be updated using the representation of the target object that has interacted with the target item, thereby enriching the information carried in the initial item representation of the target item for determining whether the object is interested in the target item. By updating the initial item representation based on the above-mentioned multimodal similarity relationship of items, the multimodal representation of the target item can be used to update the initial item representation, thereby realizing implicit utilization of the multimodal information of the target item.

[0107] After obtaining the initial object representation and the initial item representation, the training device also updates the initial object representation based on the interaction relationship and the object multimodal similarity relationship between different target objects to obtain a first updated object representation, wherein the object multimodal similarity relationship includes a similarity relationship determined based on the representation of the target object in at least one predetermined modality. When updating the initial object representation based on the above-mentioned interaction relationship, the initial object representation can be updated using the representation of the target item that has interacted with the target object, thereby enriching the information carried in the initial object representation of the target object for determining the items of interest to the target object. When updating the initial object representation based on the above-mentioned object multimodal similarity relationship, the initial object representation can be updated using the multimodal representation of the target object, thereby realizing implicit utilization of the multimodal information of the target object.

[0108] After obtaining the first updated item representation and the first updated object representation, the training device trains the target model based on the first updated item representation, the first updated object representation and the item multimodal representation, wherein the item multimodal representation includes the representation of the target item in at least one predetermined mode, and the target model is used to obtain the target item representation of the target item and / or the target object representation of the target object. In the training process, the training device trains the target model based on the item multimodal representation, and can realize the training of the target model using the multimodal information of the target item carried by the item multimodal representation, thereby realizing the display and utilization of the multimodal information of the target item. The first updated item representation and the first updated object representation are both obtained by implicitly utilizing the multimodal information, so the training device trains the target model based on implicitly utilizing and explicitly utilizing the multimodal information of the target item and the multimodal information of the target object, which can improve the training effect of the target model, thereby improving the accuracy of the information carried by the target item representation output by the target model for determining whether the object is interested in the target item, and / or improving the accuracy of the information carried by the target object representation output by the target model for determining the object of interest to the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0109] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0110] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0111] Figure 1 A flowchart of a training method provided in an embodiment of the present application;

[0112] Figure 2 A schematic diagram of a training framework provided in an embodiment of the present application;

[0113] Figure 3 A flowchart of a recommended method provided in an embodiment of the present application;

[0114] Figure 4 A schematic diagram of the structure of a training device provided in an embodiment of the present application;

[0115] Figure 5 A schematic diagram of a recommended device provided in an embodiment of the present application;

[0116] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0117] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0118] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0119] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0120] The embodiment of the present application provides a training method and a recommendation method, wherein the execution subject of the training method is a training device, wherein the training device can be any electronic device that can execute the technical solution disclosed in the embodiment of the method of the present application. Optionally, the training device can be one of the following: a computer, a server.

[0121] It should be understood that the method embodiment of the present application can also be implemented by a processor executing a computer program code. The following describes the embodiment of the present application in conjunction with the drawings in the embodiment of the present application. Figure 1 , Figure 1 A flowchart of a training method provided in an embodiment of the present application.

[0122] 101. Obtain an initial object representation of a target object and an initial item representation of a target item.

[0123] In the embodiment of the present application, the target object can be any object. The target item can be any item. Optionally, the items include: notes, images, videos, texts, audios, and commodities. The initial object representation is the representation (embedding) of the target object, and the initial item representation is the representation of the target item. Optionally, the information in the initial object representation and the information in the initial item representation are both obtained by random initialization. For example, the initial object representation and the initial item representation are both vectors, and the values ​​of the elements in the vectors are random values.

[0124] 102. Based on the interactive relationship between the target object and the target item, and the multimodal similarity relationship between different target items, the initial item representation is updated to obtain a first updated item representation.

[0125] In the embodiment of the present application, the interaction relationship between the target object and the target item includes that the target object has interacted with the target item and that the target object has not interacted with the target item. The target object has interacted with the target item, that is, the target object has interacted with the target item, which means that the target object is interested in the target item, wherein the interactive behavior includes: clicking, liking, commenting, forwarding, and collecting.

[0126] In an embodiment of the present application, the multimodal similarity relationship of items includes a similarity relationship determined based on the representation of the target item in at least one predetermined modality, wherein the similarity relationship includes two target items being similar and two target items being dissimilar. In one possible implementation, the number of predetermined modalities is 1, and the multimodal similarity relationship of items includes a similarity relationship determined based on the similarity of the representation of the target item in the predetermined modality. For example, if the predetermined modality is text, then the multimodal similarity relationship of items includes a similarity relationship between target items determined based on the similarity between the text representations of the target items, such as if the similarity is greater than or equal to the item threshold, indicating that the target items are similar, and if the similarity is less than the item threshold, indicating that the target items are dissimilar. In another possible implementation, the number of predetermined modalities is greater than 1, and the multimodal similarity relationship of items includes a similarity relationship determined based on the multimodal similarity of items, wherein the multimodal similarity of items is a similarity determined by weighted summing the similarities of the representations of the target items in different modalities. For example, at least one predetermined modality includes text and image, and the multimodal similarity of items can be obtained by weighted summing the similarity between the text representations of the target items and the similarity between the image representations of the target items. Based on the multimodal similarity of the items, the similarity relationship of the target items can be determined. If the multimodal similarity of the items is greater than or equal to the item threshold, it means that the target items are similar. If the multimodal similarity of the items is less than the item threshold, it means that the target items are not similar.

[0127] The training device updates the initial item representation based on the above-mentioned interaction relationship and the multimodal similarity relationship of the items. The initial item representation can be updated using the representation of the target object that has interacted with the target item and the multimodal representation of the target item. Among them, by using the representation of the target object that has interacted with the target item to update the initial item representation, the information carried in the representation of the target item for determining whether the object is interested in the target item can be enriched. For the convenience of expression, the information used to determine whether the object is interested in the target item will be referred to as item recommendation information below. By using the multimodal representation of the target item to update the initial item representation, the multimodal information carried by the multimodal representation can be used to update the initial item representation, thereby realizing implicit utilization of the multimodal information of the item.

[0128] 103. Based on the interaction relationship and the object multimodal similarity relationship between different target objects, update the initial object representation to obtain a first updated object representation.

[0129] In an embodiment of the present application, the object multimodal similarity relationship includes a similarity relationship determined based on the representation of the target object in at least one predetermined modality, wherein the similarity relationship includes two target objects being similar and two target objects being dissimilar. In one possible implementation, the number of predetermined modalities is 1, and the object multimodal similarity relationship includes a similarity relationship determined based on the similarity of the representation of the target object in the predetermined modality. For example, if the predetermined modality is text, then the object multimodal similarity relationship includes a similarity relationship between target objects determined based on the similarity between the text representations of the target objects, such as if the similarity is greater than or equal to the object threshold, indicating that the target objects are similar, and if the similarity is less than the object threshold, indicating that the target objects are dissimilar. In another possible implementation, the number of predetermined modalities is greater than 1, and the object multimodal similarity relationship includes a similarity relationship determined based on the object multimodal similarity, wherein the object multimodal similarity is a similarity determined by weighted summing the similarities of the representations of the target objects in different modalities. For example, at least one predetermined modality includes text and image, and the object multimodal similarity can be obtained by weighted summing the similarity between the text representations of the target objects and the similarity between the image representations of the target objects. Based on the object multimodal similarity, the similarity relationship of the target object can be determined. For example, if the object multimodal similarity is greater than or equal to the object threshold, it means that the target objects are similar. If the object multimodal similarity is less than the object threshold, it means that the target objects are not similar.

[0130] The training device updates the initial object representation based on the above-mentioned interaction relationship and the multimodal similarity relationship of the objects. The initial object representation can be updated using the representation of the target item that has interacted with the target object and the multimodal representation of the target object. Among them, updating the initial object representation using the representation of the target item that has interacted with the target object can enrich the information carried in the representation of the target object for determining the items of interest to the target object. For the convenience of expression, the information used to determine the items of interest to the target object will be referred to as object recommendation information below. And updating the initial object representation using the multimodal representation of the target object can realize the updating of the initial object representation using the multimodal information carried by the multimodal representation, thereby realizing the implicit use of the multimodal information of the object.

[0131] 104. Train a target model based on the first updated item representation, the first updated object representation, and the item multimodal representation.

[0132] In an embodiment of the present application, the multimodal representation of the item includes a representation of the target item in at least one predetermined mode. For example, if at least one predetermined mode includes text, then the multimodal representation of the item is a text representation of the target item. The target model is used to obtain a target item representation of the target item and / or a target object representation of the target object, wherein the target item representation carries item recommendation information, the target object representation carries an object recommendation representation, the target item representation is used to determine an object of interest to the target item, and the target object representation is used to determine an item of interest to the target object. In one possible implementation, the target model is used to obtain a target item representation of the target item, in another possible implementation, the target model is used to obtain a target object representation of the target object, and in yet another possible implementation, the target model is used to obtain a target item representation of the target item and a target object representation of the target object.

[0133] In step 104, the training device trains the target model based on the multimodal representation of the object, and can train the target model using the multimodal information of the target object carried by the multimodal representation of the object, thereby achieving explicit utilization of the multimodal information of the target object.

[0134] In one possible implementation, the training device determines the interaction similarity between the first updated item representation and the first updated object representation. Based on the interaction similarity, an interaction loss is obtained. In the case where the target item and the target object have interacted, the interaction loss is negatively correlated with the interaction similarity. In the case where the target item and the target object have not interacted, the interaction loss is positively correlated with the interaction similarity. Based on the interaction loss and the multimodal representation of the item, the target model is trained.

[0135] In this implementation, in the process of training the target model based on the interaction loss, by reducing the interaction loss, the interaction similarity can be reduced when the target item and the target object have not interacted, and the interaction similarity can be increased when the target item and the target object have interacted. This can have the effect of shortening the distance between the target object and the target item that have interacted, and increasing the distance between the target object and the target item that have not interacted. This can improve the accuracy of the item recommendation information carried by the target item representation obtained by the target model, and improve the accuracy of the object recommendation information carried by the target object representation obtained by the target model. When training the target model based on the multimodal representation of the item, the multimodal information of the target item carried by the multimodal representation of the item can be used to train the target model, thereby improving the accuracy of the information carried by the target item representation obtained by the target model and improving the accuracy of the information carried by the target object representation obtained by the target model.

[0136] Optionally, the multimodal representation of the object includes a first initial visual representation of the target object and a first initial text representation of the target object. In an embodiment of the present application, the visual representation is a representation of visual data, and the text representation is a representation of text, wherein the visual data includes: images and videos. In one possible implementation, the target object is a target document, and the target document includes visual data and text. Accordingly, the first initial visual representation of the target document includes a first initial visual representation and a first initial text representation. The training device implements "training a target model based on interaction loss and multimodal representation of objects" by performing the following steps: using the target model to perform a first conversion on the first initial visual representation, the first initial text representation, the first updated object representation, and the first updated object representation, respectively, to obtain a first converted visual representation, a first converted text representation, a first converted object representation, and a first converted object representation, wherein the first initial visual representation corresponds to the first converted visual representation, the first initial text representation corresponds to the first converted text representation, the first updated object representation corresponds to the first converted object representation, and the first updated object representation corresponds to the first converted object representation. The target model is used to perform a second conversion on the first initial visual representation, the first initial text representation, the first updated item representation and the first updated object representation, respectively, to obtain a second converted visual representation, a second converted text representation, a second converted item representation and a second converted object representation, wherein the first conversion and the second conversion are different conversion processes, the first initial visual representation corresponds to the second converted visual representation, the first initial text representation corresponds to the second converted text representation, the first updated item representation corresponds to the second converted item representation, and the first updated object representation corresponds to the second converted object representation. Based on the first converted visual representation, the first converted text representation, the first converted item representation, the first converted object representation, the second converted visual representation, the second converted text representation, the second converted item representation and the second converted object representation, a multi-view loss is obtained. Based on the interaction loss and the multi-view loss, the target model is trained. Optionally, the target model implements the second conversion of the representation by randomly inactivating the representation.

[0137] Since the first conversion and the second conversion are different conversion processes, different representations can be obtained by performing the first conversion and the second conversion on the same representation, and the information carried by the representation obtained by the first conversion and the information carried by the representation obtained by the second conversion are both the information carried by the representation before the conversion. Therefore, based on the representation obtained by the first conversion and the representation obtained by the second conversion, the accuracy of the information carried by the representation before the conversion can be measured from different angles. Therefore, based on the representation obtained by the first conversion and the representation obtained by the second conversion, the training device can improve the training effect without using negative samples to train the target model, that is, improve the accuracy of the information carried by the target object representation obtained by the target model and the accuracy of the information carried by the target item representation.

[0138] Based on this, the training device uses the target model to perform a first conversion on the first initial visual representation to obtain a first converted visual representation, uses the target model to perform a first conversion on the first initial text representation to obtain a first converted text representation, uses the target model to perform a first conversion on the first updated item representation to obtain a first converted item representation, and uses the target model to perform a first conversion on the first updated object representation to obtain a first converted object representation. Use the target model to perform a second conversion on the first initial visual representation to obtain a second converted visual representation, use the target model to perform a second conversion on the first initial text representation to obtain a second converted text representation, use the target model to perform a second conversion on the first updated item representation to obtain a second converted item representation, and use the target model to perform a second conversion on the first updated object representation to obtain a second converted object representation. Then, based on the first converted visual representation, the first converted text representation, the first converted item representation, the first converted object representation, the second converted visual representation, the second converted text representation, the second converted item representation, and the second converted object representation, a multi-view loss is obtained, and the target model is trained based on the interaction loss and the multi-view loss.

[0139] As an optional implementation, the multi-view loss includes one or more of the following: a first multi-view loss, a second multi-view loss, and a third multi-view loss. The first multi-view loss is obtained based on the first conversion visual representation, the first conversion text representation, the second conversion visual representation, and the second conversion text representation; the second multi-view loss is obtained based on the first conversion item representation, the first conversion object representation, the second conversion item representation, and the second conversion object representation; and the third multi-view loss is obtained based on the first conversion visual representation, the first conversion text representation, and the second conversion item representation. Optionally, the multi-view loss is obtained by weighted summing the first multi-view loss, the second multi-view loss, and the third multi-view loss.

[0140] In one possible implementation, the training device obtains the first multi-view loss through the following steps: determining a first cross-similarity between a first transformed visual representation and a second transformed textual representation; determining a second cross-similarity between the second transformed visual representation and the first transformed textual representation; and obtaining the first multi-view loss based on the first cross-similarity and the second cross-similarity.

[0141] In this implementation, the first conversion visual representation and the second conversion visual representation both contain partial information of the first initial visual representation, and the first conversion text representation and the second conversion text representation both contain partial information of the first initial text representation. Therefore, the first cross-similarity between the first conversion visual representation and the second conversion text representation is determined, that is, the first cross-similarity between the partial information of the first initial visual representation and the partial information of the first initial text representation. Since the determination of the first cross-similarity only uses the partial information of the first initial visual representation and the partial information of the first initial text representation, in order to make the first cross-similarity large, it is necessary to make the item recommendation information carried by the first initial visual representation and the item recommendation information carried by the first initial text representation accurate. Similarly, the determination of the second cross-similarity also only uses the partial information of the first initial visual representation and the partial information of the first initial text representation. Therefore, in order to make the second cross-similarity large, it is necessary to make the item recommendation information carried by the first initial visual representation and the item recommendation information carried by the first initial text representation accurate. Finally, based on the first cross-similarity and the second cross-similarity, the first multi-view loss is obtained. Since the target model is trained based on the first multi-view loss without using negative samples, the target item and the target object have interacted, so the similarity between the first initial visual representation of the target item and the second initial visual representation of the target object, and the similarity between the first initial text representation of the target item and the second initial text representation of the target object should be as large as possible. Therefore, in this implementation, the first multi-view loss is negatively correlated with the first cross-similarity and the second cross-similarity. In order to reduce the first multi-view loss, it is necessary to increase the first cross-similarity and the second cross-similarity, that is, to increase the accuracy of the item recommendation information carried by the first initial visual representation and the item recommendation information carried by the first initial text representation. Furthermore, training the target model based on the first multi-view loss can make the accuracy of the item recommendation information carried by the representation of the target item in different modalities obtained by the target model high, thereby improving the accuracy of the item recommendation information carried by the representation of the target item obtained by the target model.

[0142] In one possible implementation, the training device obtains the second multi-view loss through the following steps: determining a third cross-similarity between the first converted object representation and the second converted object representation; determining a fourth cross-similarity between the first converted object representation and the second converted object representation; and obtaining the second multi-view loss based on the third cross-similarity and the fourth cross-similarity.

[0143] In this implementation, the first conversion object representation and the second conversion object representation both contain partial information of the first initial object representation, and the first conversion item representation and the second conversion item representation both contain partial information of the first initial item representation. Therefore, the third cross-similarity between the first conversion object representation and the second conversion item representation is determined, which is the third cross-similarity between the partial information of the first initial object representation and the partial information of the first initial item representation. Since the third cross-similarity is determined by only using partial information of the first initial object representation and partial information of the first initial item representation, in order to make the third cross-similarity large, the object recommendation information carried by the first initial object representation and the item recommendation information carried by the first initial item representation need to be accurate and high. Similarly, the fourth cross-similarity is determined by only using partial information of the first initial object representation and partial information of the first initial item representation. Therefore, in order to make the fourth cross-similarity large, the object recommendation information carried by the first initial object representation and the item recommendation information carried by the first initial item representation need to be accurate and high. Finally, based on the third cross-similarity and the fourth cross-similarity, the second multi-view loss is obtained. Since the target model is trained based on the second multi-view loss without using negative samples, the target item and the target object have interacted, so the similarity between the first initial object representation of the target item and the second initial object representation of the target object, and the similarity between the first initial item representation of the target item and the second initial item representation of the target object should be as large as possible. Therefore, in this implementation, the second multi-view loss is negatively correlated with the third cross-similarity and the fourth cross-similarity. In order to reduce the second multi-view loss, it is necessary to increase the third cross-similarity and the fourth cross-similarity, that is, to increase the accuracy of the object recommendation information carried by the first initial object representation and the item recommendation information carried by the first initial item representation. Furthermore, training the target model based on the second multi-view loss can improve the accuracy of the item recommendation information carried by the representation of the target item obtained by the target model and the accuracy of the object recommendation information carried by the representation of the target object obtained by the target model.

[0144] In one possible implementation, the training device obtains the third multi-view loss through the following steps: determining the fifth cross-similarity between the first converted visual representation and the second converted object representation; determining the sixth cross-similarity between the first converted text representation and the second converted object representation; and obtaining the third multi-view loss based on the fifth cross-similarity and the sixth cross-similarity.

[0145] In this implementation, the second converted item representation includes partial information of the first initial item representation, the first converted visual representation includes partial information of the first initial visual representation, and the first converted text representation includes partial information of the first initial text representation. Therefore, the fifth cross-similarity between the first converted visual representation and the second converted item representation is determined, which is the fifth cross-similarity between the partial information of the first initial visual representation and the partial information of the first initial item representation. Since the fifth cross-similarity is determined by only using partial information of the first initial visual representation and partial information of the first initial item representation, in order to make the fifth cross-similarity large, the item recommendation information carried by the first initial visual representation and the item recommendation information carried by the first initial item representation need to be accurate and high. Similarly, the sixth cross-similarity is determined by only using partial information of the first initial text representation and partial information of the first initial item representation. Therefore, in order to make the sixth cross-similarity large, the item recommendation information carried by the first initial text representation and the item recommendation information carried by the first initial item representation need to be accurate and high. Finally, based on the fifth cross-similarity and the sixth cross-similarity, the third multi-view loss is obtained. Since the first initial text representation, the first visual representation, and the first initial item representation are all representations of the target item, the similarity between the first initial visual representation and the first initial item representation, and the similarity between the first initial text representation and the first initial item representation should be as large as possible. Therefore, in this implementation, the third multi-view loss is negatively correlated with the fifth cross-similarity and the sixth cross-similarity. In order to reduce the third multi-view loss, it is necessary to increase the fifth cross-similarity and the sixth cross-similarity, that is, to increase the accuracy of the item recommendation information carried by the first initial visual representation, the item recommendation information carried by the first initial text representation, and the item recommendation information carried by the first initial item representation. Furthermore, by training the target model based on the third multi-view loss, the accuracy of the item recommendation information carried by the representation of the target item obtained by the target model can be improved.

[0146] As an optional implementation, the target model implements the first conversion through a target module, wherein the target module has trainable parameters, wherein the trainable parameters are parameters that can be updated through training. The parameters of the target module can be updated by training the target model, so that the accuracy of the item recommendation information carried by the target item representation of the target item output by the target module is high, and the accuracy of the object recommendation information carried by the target object representation of the target object output by the target module is high. Optionally, the target model is a fully connected layer (FC).

[0147] As an optional implementation, before the training device uses the target model to perform the first conversion and the second conversion on the first initial visual representation, the first initial text representation, the first updated item representation and the first updated object representation, respectively, the training device converts the first initial visual representation, the first initial text representation, the first updated item representation and the first updated object representation through the conversion module of the target model so that the dimensions of the converted representations are the same. Specifically, after the first initial visual representation, the first initial text representation, the first updated item representation and the first updated object representation are input into the target model, the first initial visual representation, the first initial text representation, the first updated item representation and the first updated object representation are converted through the conversion module of the target model so that the first initial visual representation, the first initial text representation, the first updated item representation and the first updated object representation have the same dimension. Optionally, the conversion module is FC.

[0148] As an optional implementation, optionally, the training device performs the following steps in the process of training the target model based on the interaction loss and the multimodal representation of the object: determine the reference similarity between the multimodal representation of the object and the first updated object representation, and obtain the reference loss based on the reference similarity. In the case where the target item and the target object have interacted, the reference loss is negatively correlated with the reference similarity. In the case where the target item and the target object have not interacted, the reference loss is positively correlated with the reference similarity. Based on the interaction loss and the reference loss, the target model is trained. In this implementation, the training device obtains the reference loss based on the interaction relationship between the target object and the target item using the multimodal representation of the object, and then trains the target model based on the interaction loss and the reference loss, thereby improving the training effect.

[0149] In an embodiment of the present application, after acquiring the initial object representation of the target object and the initial item representation of the target item, the training device updates the initial item representation based on the interaction relationship between the target object and the target item, and the multimodal similarity relationship between different target items, to obtain a first updated item representation, wherein the multimodal similarity relationship of items includes a similarity relationship determined based on the representation of the target item in at least one predetermined modality. By updating the initial item representation based on the above-mentioned interaction relationship, the initial item representation of the target item can be updated using the representation of the target object that has interacted with the target item, thereby enriching the information carried in the initial item representation of the target item for determining whether the object is interested in the target item. By updating the initial item representation based on the above-mentioned multimodal similarity relationship of items, the multimodal representation of the target item can be used to update the initial item representation, thereby realizing implicit utilization of the multimodal information of the target item.

[0150] After obtaining the initial object representation and the initial item representation, the training device also updates the initial object representation based on the interaction relationship and the object multimodal similarity relationship between different target objects to obtain a first updated object representation, wherein the object multimodal similarity relationship includes a similarity relationship determined based on the representation of the target object in at least one predetermined modality. When updating the initial object representation based on the above-mentioned interaction relationship, the initial object representation can be updated using the representation of the target item that has interacted with the target object, thereby enriching the information carried in the initial object representation of the target object for determining the items of interest to the target object. When updating the initial object representation based on the above-mentioned object multimodal similarity relationship, the initial object representation can be updated using the multimodal representation of the target object, thereby realizing implicit utilization of the multimodal information of the target object.

[0151] After obtaining the first updated item representation and the first updated object representation, the training device trains the target model based on the first updated item representation, the first updated object representation and the item multimodal representation, wherein the item multimodal representation includes the representation of the target item in at least one predetermined mode, and the target model is used to obtain the target item representation of the target item and / or the target object representation of the target object. In the training process, the training device trains the target model based on the item multimodal representation, and can realize the training of the target model using the multimodal information of the target item carried by the item multimodal representation, thereby realizing the display and utilization of the multimodal information of the target item. The first updated item representation and the first updated object representation are both obtained by implicitly utilizing the multimodal information, so the training device trains the target model based on implicitly utilizing and explicitly utilizing the multimodal information of the target item and the multimodal information of the target object, which can improve the training effect of the target model, thereby improving the accuracy of the information carried by the target item representation output by the target model for determining whether the object is interested in the target item, and / or improving the accuracy of the information carried by the target object representation output by the target model for determining the object of interest to the target object.

[0152] As an optional implementation, the training device further performs the following steps:

[0153] 2001. In the process of training the above-mentioned target model, the multimodal representation of the above-mentioned object is updated to obtain an updated multimodal representation.

[0154] Specifically, during the training process of the target model, the training device updates the multimodal representation of the object based on the interaction loss and the multi-view loss to obtain an updated multimodal representation. In one possible implementation, the multimodal representation of the object includes a first initial visual representation and a first initial text representation. The training device updates the first initial visual representation based on the interaction loss and the multi-view loss to obtain an updated visual representation. The training device updates the first initial text representation based on the interaction loss and the multi-view loss to obtain an updated visual text representation. At this time, updating the multimodal representation includes updating the visual representation and updating the text representation.

[0155] 2002. Based on the difference between the updated multimodal representation and the multimodal features of the object, a modal consistency loss is obtained.

[0156] Since the multimodal representation of an item includes the original modal information, and the updating of the multimodal representation of an item may result in the loss of the original modal information, after the modal consistency loss is obtained based on the difference between the updated multimodal representation and the multimodal features of the item, the modal consistency loss can indicate how much original modal information is lost after the multimodal representation of the item is updated. Specifically, the greater the difference, the more original modal information is lost after the multimodal representation of the item is updated, so the modal consistency loss is positively correlated with the difference. In this way, during the training process of the target model, by reducing the modal consistency loss, the difference between the updated multimodal representation and the multimodal features of the item can be reduced, thereby reducing the original modal information lost due to the update of the multimodal representation of the item.

[0157] In one possible implementation, the multimodal representation of the object includes a first initial visual representation and a first initial text representation, and the updated multimodal representation includes an updated visual representation and an updated text representation. The training device determines the visual difference between the first initial visual representation and the updated visual representation, and determines the text difference between the first initial text representation and the updated text representation. Based on the visual difference and the text difference, a modal consistency loss is obtained, wherein the modal consistency loss is positively correlated with both the visual difference and the text difference.

[0158] Since both the first initial visual representation and the first initial text representation include item recommendation information, and the first initial visual representation and the first initial text representation may result in the loss of item recommendation information after being updated, after the modal consistency loss is obtained based on the visual difference and the text difference, the modal consistency loss can indicate how much item recommendation information the first initial visual representation and the first initial text representation have lost after being updated. Specifically, the greater the visual difference, the more item recommendation information the first initial visual representation has lost after being updated, and the greater the text difference, the more item recommendation information the first initial text representation has lost after being updated, so the modal consistency loss is positively correlated with the difference. In this way, during the training process of the target model, by reducing the modal consistency loss, the difference between the updated visual representation and the first initial visual features and the difference between the updated text representation and the first initial text representation can be reduced, thereby reducing the item recommendation information lost by the first initial visual representation and the first initial text representation due to the update.

[0159] 2003. The target model is trained based on the modality consistency loss.

[0160] By executing step 2003, the accuracy of the item recommendation information carried by the target item representation output by the target model can be made higher. Optionally, after obtaining the interaction loss, multi-view loss and modality consistency loss, the training device trains the target model based on the interaction loss, multi-view loss and modality consistency loss.

[0161] As an optional implementation, the training device performs the following steps during the execution of step 102:

[0162] 3001. Obtain first image data.

[0163] In the embodiment of the present application, graph data includes nodes, nodes correspond to data, and graph data includes the relationship between the data corresponding to the nodes. For example, if the data corresponding to the nodes is a representation of an object or a representation of an item, then the relationship between the data corresponding to the nodes includes the interactive relationship between the item and the node. Optionally, the graph data includes an undirected graph.

[0164] The first graph data includes a first object node and a first item node. The information carried by the first object node includes an initial object representation, and the information carried by the first item node includes an initial item representation. The edges between the nodes in the first graph data are determined based on the interaction relationship. Since in the first graph data, the information carried by the first object node includes an initial object representation, and the initial object representation is a representation of the target object, the first object node is a node corresponding to the target object. Similarly, the first item node is a node corresponding to the target item. In the first graph data, if the object corresponding to the first object node has interacted with the item corresponding to the first item node, there is a first edge between the first object node and the first item node. Conversely, if there is no first edge between the first object node and the first item node, it means that the object corresponding to the first object node has not interacted with the item corresponding to the first item node.

[0165] 3002. Obtain second image data.

[0166] In an embodiment of the present application, the second graph data includes a second item node, and the information carried by the second item node includes an initial item representation. The edges between the nodes in the second graph data are determined based on the multimodal similarity relationship of the items. Specifically, in the second graph data, the existence of a second edge between two item nodes indicates that the target items corresponding to the two item nodes are similar. Optionally, the target items corresponding to the two item nodes are similar, which means that the similarity between the initial item representations carried by the two second item nodes is greater than or equal to the item similarity threshold. Optionally, for any second item node, determine the item similarity between the initial item representation carried by the second item node and other initial item representations, determine the initial item representation corresponding to the largest k item similarities as a candidate item representation, and connect the second item node to the node corresponding to the candidate item representation to obtain a second edge.

[0167] In one possible implementation, the second graph data is obtained by the following steps: obtaining a first initial visual representation of the target item and a first initial text representation of the target item. Based on the first initial visual representation and the first initial text representation, obtaining an item multimodal similarity between the two initial item representations. Determining an item multimodal similarity relationship based on the item multimodal similarity. Based on the item multimodal similarity relationship, obtaining the second graph data.

[0168] Optionally, the training device obtains the second graph data by performing the following steps: obtaining a first initial visual representation of the target object and a first initial text representation of the target object; determining the similarity between the first initial visual representations of different target objects to obtain a first visual similarity; determining the similarity between the first initial text representations of different target objects to obtain a first text similarity; and determining the edges of the second graph data based on the first visual similarity and the first text similarity.

[0169] Optionally, by weighted summing the first visual similarity and the first textual similarity, the multimodal similarity of the item is obtained, and the edge of the second graph data is determined based on the multimodal similarity of the item. In a possible implementation, when it is determined based on the multimodal similarity of the item that the two target items corresponding to the multimodal similarity of the item are similar, it is determined to establish a second edge between the second item nodes corresponding to the two target items. Optionally, in the second graph data, the edge weight of the second edge is positively correlated with the multimodal similarity of the item. Wherein, by processing the second graph data, information can be transmitted in the node, thereby realizing the update of the representation carried by the node. Specifically, by weighted summing the representation carried by the second item node to be updated and the representation carried by the second item node connected to the second item node to be updated, the update of the representation carried by the second item node to be updated is realized, and the edge weight of the second edge can be used to determine the weight of the representation carried by the second item node connected to the second item node to be updated, and the weight is positively correlated with the edge weight of the second edge.

[0170] Optionally, for any first initial visual representation of a target object, its similarity with other first initial visual representations is determined to obtain a first initial visual matrix. For any first initial text representation of a target object, its similarity with other first initial text representations is determined to obtain a first initial text matrix. v Represents the first initial visual matrix of the target object, denoted by S t represents the first initial text matrix of the target item, then for each target item S v and S t First, sparseness is performed to obtain the first sparse visual matrix and the first sparse text matrix. Specifically, sparseness is achieved by retaining the k vectors with the largest similarity in the matrix. Then, the first sparse visual matrix and the first sparse text matrix are regularized to obtain the first regularized visual matrix and the first regularized text matrix respectively. The first regularized visual matrix and the first regularized text matrix are weighted summed to obtain the multimodal similarity of the items.

[0171] 3003. Use a graph convolutional network (GCN) to process the first graph data to obtain processed first graph data.

[0172] By using GCN to process the first graph data, the information carried by the nodes in the first graph data can be transmitted through the first edge to obtain the processed first graph data, and then the information carried by each node in the first graph data can be updated. Since in the first graph data, when the target object corresponding to the first object node and the target item corresponding to the first item node have interacted, there is a first edge between the first object node and the first item node, so the information carried by the nodes in the first graph data is transmitted through the first edge, and when the target object and the target item have interacted, the information carried by the first initial representation of the target object can be transmitted to the first item node corresponding to the target item, so that the information carried by the first item node is updated. In this way, when the target object and the target item have interacted, the initial object representation of the target item can be used to update the initial item representation of the target item.

[0173] In one possible implementation, the training device processes the nodes connected by some edges in the first graph data using GCN to obtain the processed first graph data. Optionally, some first edges in the first graph data are removed to obtain the fourth graph data. The fourth graph data is processed using GCN to obtain the processed first graph data. Optionally, the fourth graph data is obtained by randomly removing some first edges in the first graph data. Specifically, during the training process, the first edges removed in different training cycles (epochs) are different.

[0174] In actual recommendation scenarios, the object may interact with items that it is not interested in. The interaction relationship between the object and the item is noisy. For example, the item is a note. After clicking on the note, the object finds that the content of the note is not of interest to him, and then closes the note. In other words, the note clicked by the object is not the note that the object is interested in, but there has been an interaction between the object and the note. At this time, there is noise in the interaction relationship between the object and the item. The training process is based on whether the target object and the target item have interacted, that is, the training process is based on the interaction relationship between the target object and the target item. Therefore, in order to reduce the impact of the noise in the interaction relationship on the training effect, after constructing the first graph data based on the interaction relationship between the target object and the target item, the fourth graph data is obtained by removing part of the first edge in the first graph data, which can simulate the noise and thus improve the training effect.

[0175] 3004. Use the GCN to process the second graph data to obtain processed second graph data.

[0176] By processing the second graph data using GCN, the information carried by the nodes in the second graph data can be transmitted through the second edge to obtain the processed second graph data, and then the information carried by each node in the second graph data can be updated. Since in the second graph data, when the target items corresponding to two second item nodes are similar, there is a second edge between the two second item nodes, therefore, the information carried by the nodes in the second graph data is transmitted through the second edge, so that the initial item representations of similar target items can be fused, and then the initial item representation of the target item can be updated. Because the second edge of the second graph data is determined based on the multimodal similarity relationship of the items, and the multimodal similarity relationship of the items is obtained using the multimodal representation of the target item, by processing the second graph data using GCN, the information carried by the nodes in the second graph data can be transmitted through the second edge, and the multimodal information carried by the multimodal representation of the target item can be used to update the initial item representation of the target item.

[0177] 3005. Based on the representation carried by the first item node in the processed first graph data and the representation carried by the second item node in the processed second graph data, obtain the first updated item representation.

[0178] In the embodiment of the present application, the first item node and the second item node are both nodes corresponding to the target item. In one possible implementation, the representation carried by the first item node in the processed first graph data and the representation carried by the second item node in the processed second graph data are concatenated to obtain a first updated item representation.

[0179] As an optional implementation, the training device performs the following steps during the execution of step 103:

[0180] 4001. Obtain the third image data.

[0181] The third graph data includes a second object node, and the information carried by the second object node includes an initial object representation. The edges of the nodes in the third graph data are determined based on the multimodal similarity relationship of the objects. Specifically, in the third graph data, the existence of a third edge between two object nodes indicates that the target objects corresponding to the two object nodes are similar. Optionally, the target objects corresponding to the two object nodes are similar, which means that the similarity of the initial object representations carried by the two second object nodes is greater than or equal to the object similarity threshold. Optionally, for any second object node, the object similarity between the initial object representation carried by the second object node and other initial object representations is determined, and the initial object representation corresponding to the largest k object similarities is determined as a candidate object representation, and the second object node is connected to the node corresponding to the candidate object representation to obtain a third edge.

[0182] In one possible implementation, the third graph data is obtained by the following steps: obtaining a second initial visual representation of the target object and a second initial text representation of the target object. Based on the second initial visual representation and the second initial text representation, obtaining an object multimodal similarity between the two initial object representations. Determining an object multimodal similarity relationship based on the object multimodal similarity. Based on the object multimodal similarity relationship, obtaining the third graph data.

[0183] Optionally, the training device obtains the third graph data by performing the following steps: obtaining a second initial visual representation of the target object and a second initial text representation of the target object; determining the similarity between the second initial visual representations of different target objects to obtain a second visual similarity; determining the similarity between the second initial text representations of different target objects to obtain a second text similarity; and determining the edges of the third graph data based on the second visual similarity and the second text similarity.

[0184] In an embodiment of the present application, the second initial visual representation is obtained based on the visual representation of the item that has interacted with the target object, and the second initial text representation is obtained based on the text representation of the item that has interacted with the target object. For example, the document that has interacted with the target object includes visual data and text. In this case, the second initial visual representation is the representation of the visual data, and the second initial text representation is the representation of the text. For another example, the documents that have interacted with the target object include document a and document b, wherein both document a and document b include visual data and text. In this case, the second initial visual representation is the average of the representation of the visual data of document a and the representation of the visual data of document b, and the second initial text representation is the average of the representation of the text of document a and the representation of the text of document b.

[0185] Optionally, by weighted summing the second visual similarity and the second textual similarity, the object multimodal similarity is obtained, and the edge of the third graph data is determined based on the object multimodal similarity. In a possible implementation, when the two target objects corresponding to the object multimodal similarity are determined to be similar based on the object multimodal similarity, it is determined to establish a third edge between the second object nodes corresponding to the two target objects. Optionally, in the third graph data, the edge weight of the third edge is positively correlated with the object multimodal similarity. Among them, by processing the third graph data, information can be transmitted in the node, thereby realizing the update of the representation carried by the node. Specifically, by weighted summing the representation carried by the second object node to be updated and the representation carried by the second object node connected to the second object node to be updated, the update of the representation carried by the second object node to be updated is realized, and the edge weight of the third edge can be used to determine the weight of the representation carried by the second object node connected to the second object node to be updated, and the weight is positively correlated with the edge weight of the third edge.

[0186] Optionally, for any second initial visual representation of a target object, its similarity with other second initial visual representations is determined to obtain a second initial visual matrix. For any second initial text representation of a target object, its similarity with other second initial text representations is determined to obtain a second initial text matrix. v Represents the second initial visual matrix of the target object, denoted by R t represents the second initial text matrix of the target object, then for each target object's R v and R t First, the second sparse visual matrix and the second sparse text matrix are obtained by sparseness. Specifically, the sparseness is achieved by retaining the k vectors with the largest similarity in the matrix. Then, the second sparse visual matrix and the second sparse text matrix are regularized to obtain the second regularized visual matrix and the second regularized text matrix respectively. The second regularized visual matrix and the second regularized text matrix are weighted summed to obtain the multimodal similarity of the object.

[0187] 4002. Use GCN to process the third graph data to obtain processed third graph data.

[0188] By processing the third graph data using GCN, the information carried by the nodes in the third graph data can be transmitted through the third edge to obtain the processed third graph data, and then the information carried by each node in the third graph data can be updated. Since in the third graph data, when the target objects corresponding to two second object nodes are similar, there is a third edge between the two second object nodes, therefore, the information carried by the nodes in the third graph data is transmitted through the third edge, which can achieve fusion between the initial object representations of similar target objects, and then the initial object representation of the target object is updated. Because the third edge of the third graph data is determined based on the multimodal similarity relationship of the object, and the multimodal similarity relationship of the object is obtained using the multimodal representation of the target object, by processing the third graph data using GCN, the information carried by the nodes in the third graph data is transmitted through the third edge, which can achieve the update of the initial object representation of the target object using the multimodal information carried by the multimodal representation of the target object.

[0189] 4003. Obtain a first updated object representation based on the representation carried by the first object node in the processed first graph data and the representation carried by the second object node in the processed third graph data.

[0190] In the embodiment of the present application, the first object node and the second object node are both nodes corresponding to the target object. In one possible implementation, the representation carried by the first object node in the processed first graph data and the representation carried by the second object node in the processed third graph data are concatenated to obtain a first updated object representation.

[0191] Based on the object target training process provided in the embodiment of the present application, the embodiment of the present application also provides a training method, in which the recommended model is FC, and the training method can be used to train FC and obtain the target object representation of the target object and the target object representation of the target object. Figure 2 , Figure 2 A schematic diagram of a training framework provided in an embodiment of the present application.

[0192] like Figure 2 As shown, first obtain the initial object representation of the target object and the initial item representation of the target item. Then, based on the interactive relationship between the target object and the target item, determine the first edge of the first graph data, and use the initial object representation as the representation of the first object node in the first graph data, and use the initial item representation as the representation of the first item node in the first graph data. Determine the second edge of the second graph data based on the multimodal similarity relationship between different target items, and use the initial item representation as the representation of the second item node in the second graph data. Determine the second edge of the second graph data based on the multimodal similarity relationship between different target objects, and use the initial object representation as the representation of the second object node in the second graph data. Since the edges in the second graph data and the third graph data are determined using multimodal information, the second graph data and the third graph data both carry multimodal information, that is, the second graph data and the third graph data are both multimodal graph data. If U represents the target object and I represents the target item, then the first graph data can be represented as UI graph data (UI graph), the second graph data can be represented as multimodal II graph data (MultiModal II graph), and the third graph data can be represented as multimodal UU graph data (MultiModal UU graph). Optionally, GCN is used to process the nodes connected by some edges in the first graph data to obtain the processed first graph data, then some of the first edges in the first graph data are randomly masked (Masked), so the first graph data can be represented as masked UI graph data (Masked UI graph).

[0193] Then, the first graph data is processed using a lightweight graph convolutional network (LightGCN) to obtain the first transfer object representation of each target object (i.e. Figure 2 In ) and the first transferred item representation of each target item (i.e. Figure 2 In ). By processing the second graph data using a lightweight graph convolutional network, the second transfer item representation of each target item can be obtained (i.e. Figure 2 In ). By processing the third graph data using a lightweight graph convolutional network, the second transfer object representation of each target object can be obtained (i.e. Figure 2 In ). Then, the first transfer object representation of each target object and the second transfer object representation of each target object are concatenated to obtain the first updated object representation of each target object (i.e. Figure 2 h in u ), the first transferred item representation of each target item and the second transferred item representation of each target item are concatenated to obtain the first updated item representation of each target item (i.e. Figure 2 h in i ).

[0194] exist Figure 2 In the training process, the first transferred item representation of each target item includes the updated visual representation of each target item and the updated text representation of each target item. Figure 2 In the , the fully connected layer converts the first initial visual representation or updated visual representation of each target object into an intermediate converted visual representation of each target object (i.e. Figure 2 In ), the fully connected layer converts the first initial text representation or updated text representation of each target item into an intermediate converted text representation of each target item (i.e. Figure 2 In ).

[0195] The intermediate converted visual representation of each target item, the intermediate converted textual representation of each target item, the first updated object representation of each target object, and the first updated item representation of each target item are input to the multi-view generator. Figure 2 As shown in FIG, after the representation is input to the multi-view generator, the input representation is processed using a fully connected layer (FC) and random dropout, respectively. Specifically, the input representation is processed using FC to convert the input representation into a first conversion representation (i.e., Figure 2 In ), random inactivation of the input representation can randomly discard the information in the input representation to obtain the second transformed representation (i.e. Figure 2 In ). The intermediate transformed visual representation of each target object is input into the multi-view generator to obtain the first transformed visual representation (i.e. Figure 2 In ) and the second transformed visual representation (i.e. Figure 2 In ). The intermediate conversion text representation of each target item is input into the multi-view generator to obtain the first conversion text representation (i.e. Figure 2 In ) and the second transformed text representation (i.e. Figure 2 In ). The first updated object representation of each target object is input into the multi-view generator to obtain the first transformed object representation (i.e. Figure 2 In ) and the second conversion object representation (i.e. Figure 2 In ). The first updated item representation of each target item is input into the multi-view generator to obtain the first transformed item representation (i.e. Figure 2 In ) and the second conversion item representation (i.e. Figure 2 In ).

[0196] After obtaining the first initial visual representation, the first initial text representation, the updated visual representation, and the updated text representation, the multimodal consistency loss can be calculated by the following formula:

[0197]

[0198] Among them, L MFP is the multimodal consistency loss, I is the set of target items, i∈I represents any target item in the target item set. M represents the set of visual representations and textual representations, m∈M represents m is a visual representation or a textual representation. ∑ represents the sum. In the case where m is a visual representation, represents updating the visual representation, represents the first initial visual representation. In the case where m is a text representation, represents updating the text representation, represents the first initial text representation. In formula (1), the parameters of the first initial visual representation and the parameters of the first initial text representation are not updated. ‖·‖2 represents the L2 norm.

[0199] In getting h u and h i After that, the interaction loss can be calculated by the following formula:

[0200]

[0201] Among them, L lAis the interaction loss, D represents the set of target objects and target items, (u,i,j)∈D represents the target objects and target items that have interacted in D, and the target objects and target items that have not interacted in D. Specifically, (u,i)∈D represents the target objects and target items that have interacted in D, and (u,j)∈D represents the target objects and target items that have not interacted in D. N represents the number of object-item pairs in D, and the object-item pair includes a target item and a target object. ∑ represents the sum. log(·) represents the logarithmic function with base 10. σ(·) represents the function that maps the values ​​in the brackets to numbers between 0 and 1, and σ(·) is a monotonically increasing function. h u The transpose of h i is the first transferred item representation of the target item that has interacted with the target object, h j A first transferred item representation representing a target item that has not interacted with the target object.

[0202] In getting After that, the first multi-view loss can be calculated by the following formula:

[0203]

[0204] Among them, L mm is the first multi-view loss, I is the set of target items, i∈I represents any target item in the target item set. M represents the set of visual representations and textual representations, m∈M represents m is a visual representation or a textual representation. In the case where m is a visual representation, represents the first transformed visual representation, Represents the second transformed visual representation, at this time for for When m is a text representation, represents the first transformed text representation, Represents the second transformed text representation, for for C(·) represents the negative cosine similarity. ∑ represents the sum.

[0205] In getting After that, the second multi-view loss can be calculated by the following formula:

[0206]

[0207] Among them, L imis the second multi-view loss, I is the set of target items, i∈I represents any target item in the target item set. M represents the set of visual representations and textual representations, m∈M represents m is a visual representation or a textual representation. In the case where m is a visual representation, Represents the first conversion visual representation, at this time for When m is a text representation, Represents the first converted text representation, at this time for represents the second transformed item representation. C(·) represents the negative cosine similarity. ∑ represents the sum.

[0208] In getting After that, the third multi-view loss can be calculated by the following formula:

[0209]

[0210] Among them, L ui is the third multi-view loss, D represents the set of target objects and target items, (u,i)∈D represents the target objects and target items that have interacted in D. ∑ represents the summation. C(·) represents the negative cosine similarity. represents the first conversion object representation, represents the second conversion item representation, represents the first conversion item representation, Represents the second transformed object representation.

[0211] After calculating the second multi-view loss, the first multi-view loss, and the third multi-view loss, the multi-view loss can be calculated by the following formula:

[0212] L EA =L ui +β(L im +L mm )…Formula (6)

[0213] Among them, L EA is the multi-view loss, and β is a non-negative number less than 1.

[0214] After calculating the multimodal consistency loss, the interaction loss, and the multi-view loss, the total loss can be obtained by weighted summing the multimodal consistency loss, the interaction loss, and the multi-view loss. When the total loss is less than or equal to the convergence threshold, the initial object representation is determined to be the target object representation, and the initial item representation is determined to be the target item representation. When the total loss is greater than the convergence threshold, the parameters of the fully connected layer, the parameters of the initial object representation, and the parameters of the initial item representation are updated based on the total loss.

[0215] See also Figure 3 , Figure 3 A flowchart of a recommendation method provided in an embodiment of the present application is provided, wherein the execution subject of the recommendation method is a recommendation device.

[0216] 301. Obtain a target item representation of a target item and a target object representation of a target object.

[0217] In the embodiment of the present application, the target item representation and the target object representation are obtained by processing the updated item representation and the updated object representation using at least part of the modules of the target model, wherein the target model is based on Figure 1 The updated item representation and the updated object representation are obtained by training using the training method shown, and are updated during the training process of the target model. Specifically, during the training process of the target model, the updated object representation is obtained by updating the initial object representation, and the updated item representation is obtained by updating the initial item representation.

[0218] In one possible implementation, based on Figure 1 In the case where the training method shown in the figure updates the initial item representation to obtain the updated item representation, the target item representation is obtained by transforming the updated item representation using the transformation module in the target model. Figure 1 When the training method shown updates the initial object representation to obtain the updated object representation, the target object representation is obtained by transforming the updated object representation using the transformation module in the target model.

[0219] In another possible implementation, based on Figure 1 In the case where the training method shown in the figure updates the initial item representation to obtain the updated item representation, the target item representation is obtained by the following steps: using the conversion module of the target model to convert the updated item representation to obtain a second updated item representation. Using the target model to perform a first conversion on the second updated item representation to obtain the target item representation. Figure 1 In the case where the training method shown updates the initial object representation to obtain the updated object representation, the target object representation is obtained by the following steps: using the conversion module of the target model to convert the updated object representation to obtain a second updated object representation. Using the target model to perform a first conversion on the second updated object representation to obtain the target object representation.

[0220] 302. Determine the target similarity between the target item representation and the target object representation.

[0221] 303. When the target similarity is greater than or equal to a recommendation threshold, determine that the target item is an item recommended to the target object.

[0222] Since the target item representation carries the item recommendation information of the target item, and the target object representation carries the object recommendation information of the target object, the greater the target similarity between the target object representation and the target item representation, the higher the degree of interest of the target object in the target item. Therefore, the recommendation device can determine that the target item is an item to be recommended to the target object when the target similarity is large. In the embodiment of the present application, the recommendation device determines whether the target similarity is large or small based on the recommendation threshold. Specifically, if the target similarity is greater than or equal to the recommendation threshold, it means that the target similarity is large. Conversely, if the target similarity is less than the recommendation threshold, it means that the target similarity is small. Therefore, the recommendation device determines that the target item is an item to be recommended to the target object when the target similarity is greater than or equal to the recommendation threshold.

[0223] In the embodiment of the present application, both the updated item representation and the updated object representation are updated during the training process of the target model, so the accuracy of the item recommendation information carried by the updated item representation and the accuracy of the object recommendation information carried by the updated object representation are both high. Because the target item representation and the target object representation are obtained by processing using at least part of the modules of the target model, and the target model is trained based on the training method described above, the accuracy of the item recommendation information carried by the target item representation and the accuracy of the object recommendation information carried by the target object representation are both high. Therefore, after obtaining the target item representation of the target item and the target object representation of the target object, the recommendation device determines the target similarity between the target item representation and the target object representation, and when the target similarity is greater than or equal to the recommendation threshold, determines that the target item is an item recommended to the target object, which can improve the accuracy of determining the items recommended to the target object.

[0224] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0225] The method of the embodiment of the present application is described in detail above, and the device of the embodiment of the present application is provided below.

[0226] See also Figure 4 , Figure 4 A structural diagram of a training device provided in an embodiment of the present application, wherein the training device 1 comprises: an acquisition unit 11, an update unit 12, and a training unit 13, specifically:

[0227] An acquisition unit 11, configured to acquire an initial object representation of a target object and an initial item representation of a target item;

[0228] An updating unit 12 is configured to update the initial item representation based on the interaction relationship between the target object and the target item, and the item multimodal similarity relationship between different target items, to obtain a first updated item representation, wherein the item multimodal similarity relationship includes a similarity relationship determined based on the representation of the target item in at least one predetermined modality;

[0229] The updating unit 12 is used to update the initial object representation based on the interaction relationship and the object multimodal similarity relationship between different target objects to obtain a first updated object representation, wherein the object multimodal similarity relationship includes a similarity relationship determined based on the representation of the target object in the at least one predetermined modality;

[0230] The training unit 13 is used to train a target model based on the first updated item representation, the first updated object representation and the item multimodal representation, wherein the item multimodal representation includes the representation of the target item in at least one predetermined modality, and the target model is used to obtain the target item representation of the target item and / or the target object representation of the target object.

[0231] In combination with any implementation manner of the present application, the updating unit 12 is specifically configured to:

[0232] Acquire first graph data, the first graph data comprising a first object node and a first item node, information carried by the first object node comprising the initial object representation, information carried by the first item node comprising the initial item representation, and edges between nodes in the first graph data being determined based on the interaction relationship;

[0233] Acquire second graph data, where the second graph data includes a second item node, information carried by the second item node includes the initial item representation, and edges between nodes in the second graph data are determined based on the multimodal similarity relationship of the items;

[0234] Processing the first graph data using a graph convolutional network to obtain processed first graph data;

[0235] Processing the second graph data using the graph convolutional network to obtain processed second graph data;

[0236] The first updated item representation is obtained based on the representation carried by the first item node in the processed first graph data and the representation carried by the second item node in the processed second graph data, wherein the first item node and the second item node are both nodes corresponding to the target item.

[0237] In combination with any embodiment of the present application, the at least one predetermined modality includes a text modality and a visual modality, and the acquisition unit 11 is specifically configured to:

[0238] Acquire a first initial visual representation of the target object and a first initial textual representation of the target object;

[0239] Determining similarities between the first initial visual representations of different target objects to obtain a first visual similarity;

[0240] Determining similarities between the first initial text representations of different target objects to obtain first text similarities;

[0241] Based on the first visual similarity and the first textual similarity, edges of the second graph data are determined.

[0242] In combination with any implementation manner of the present application, the updating unit 12 is specifically configured to:

[0243] Acquire third graph data, the third graph data including a second object node, information carried by the second object node including the initial object representation, and edges of the nodes in the third graph data are determined based on the multimodal similarity relationship of the objects;

[0244] Processing the third graph data using the graph convolutional network to obtain processed third graph data;

[0245] The first updated object representation is obtained based on the representation carried by the first object node in the processed first graph data and the representation carried by the second object node in the processed third graph data, and the first object node and the second object node are both nodes corresponding to the target object.

[0246] In combination with any embodiment of the present application, the acquisition unit 11 is specifically used for:

[0247] Acquire a second initial visual representation of the target object and a second initial textual representation of the target object, wherein the second initial visual representation is obtained based on a visual representation of an object that has interacted with the target object, and the second initial textual representation is obtained based on a textual representation of the object that has interacted with the target object;

[0248] Determining the similarity between the second initial visual representations of different target objects to obtain a second visual similarity;

[0249] Determining the similarity between the second initial text representations of different target objects to obtain a second text similarity;

[0250] Based on the second visual similarity and the second textual similarity, edges of the third graph data are determined.

[0251] In combination with any implementation manner of the present application, the updating unit 12 is specifically configured to:

[0252] The graph convolutional network is used to process nodes connected by some edges in the first graph data to obtain the processed first graph data.

[0253] In combination with any implementation manner of the present application, the training unit 13 is specifically used for:

[0254] determining an interaction similarity between the first updated item representation and the first updated object representation;

[0255] Based on the interaction similarity, an interaction loss is obtained; when the target item and the target object have interacted before, the interaction loss is negatively correlated with the interaction similarity; when the target item and the target object have not interacted before, the interaction loss is positively correlated with the interaction similarity;

[0256] The target model is trained based on the interaction loss and the multimodal representation of the item.

[0257] In conjunction with any embodiment of the present application, the multimodal representation of the object includes a first initial visual representation of the target object and a first initial text representation of the target object. The training unit 13 is specifically configured to:

[0258] Using the target model, respectively perform a first conversion on the first initial visual representation, the first initial text representation, the first updated item representation, and the first updated object representation to obtain a first converted visual representation, a first converted text representation, a first converted item representation, and a first converted object representation, wherein the first initial visual representation corresponds to the first converted visual representation, the first initial text representation corresponds to the first converted text representation, the first updated item representation corresponds to the first converted item representation, and the first updated object representation corresponds to the first converted object representation;

[0259] Using the target model, a second conversion is performed on the first initial visual representation, the first initial text representation, the first updated item representation, and the first updated object representation, respectively, to obtain a second converted visual representation, a second converted text representation, a second converted item representation, and a second converted object representation, wherein the first conversion and the second conversion are different conversion processes, the first initial visual representation corresponds to the second converted visual representation, the first initial text representation corresponds to the second converted text representation, the first updated item representation corresponds to the second converted item representation, and the first updated object representation corresponds to the second converted object representation;

[0260] Obtaining a multi-view loss based on the first converted visual representation, the first converted text representation, the first converted item representation, the first converted object representation, the second converted visual representation, the second converted text representation, the second converted item representation, and the second converted object representation;

[0261] The target model is trained based on the interaction loss and the multi-view loss.

[0262] In combination with any embodiment of the present application, the multi-view loss includes one or more of the following: a first multi-view loss, a second multi-view loss, and a third multi-view loss;

[0263] The first multi-view loss is obtained based on the first transformed visual representation, the first transformed textual representation, the second transformed visual representation, and the second transformed textual representation;

[0264] The second multi-view loss is obtained based on the first converted item representation, the first converted object representation, the second converted item representation and the second converted object representation;

[0265] The third multi-view loss is obtained based on the first converted visual representation, the first converted text representation and the second converted item representation.

[0266] In combination with any embodiment of the present application, before using the target model to perform the first conversion and the second conversion on the first initial visual representation, the first initial text representation, the first updated item representation and the first updated object representation, respectively, the first initial visual representation, the first initial text representation, the first updated item representation and the first updated object representation are respectively converted via the conversion module of the target model so that the dimensions of the converted representations are the same.

[0267] In combination with any embodiment of the present application, the updating unit 12 is further used to update the multimodal representation of the object during the training of the target model to obtain an updated multimodal representation;

[0268] The training unit 13 is further configured to obtain a modality consistency loss based on a difference between the updated multimodal representation and the multimodal feature of the object;

[0269] The training unit 13 is further used to train the target model based on the modality consistency loss.

[0270] In an embodiment of the present application, after acquiring the initial object representation of the target object and the initial item representation of the target item, the training device updates the initial item representation based on the interaction relationship between the target object and the target item, and the multimodal similarity relationship between different target items, to obtain a first updated item representation, wherein the multimodal similarity relationship of items includes a similarity relationship determined based on the representation of the target item in at least one predetermined modality. By updating the initial item representation based on the above-mentioned interaction relationship, the initial item representation of the target item can be updated using the representation of the target object that has interacted with the target item, thereby enriching the information carried in the initial item representation of the target item for determining whether the object is interested in the target item. By updating the initial item representation based on the above-mentioned multimodal similarity relationship of items, the multimodal representation of the target item can be used to update the initial item representation, thereby realizing implicit utilization of the multimodal information of the target item.

[0271] After obtaining the initial object representation and the initial item representation, the training device also updates the initial object representation based on the interaction relationship and the object multimodal similarity relationship between different target objects to obtain a first updated object representation, wherein the object multimodal similarity relationship includes a similarity relationship determined based on the representation of the target object in at least one predetermined modality. When updating the initial object representation based on the above-mentioned interaction relationship, the initial object representation can be updated using the representation of the target item that has interacted with the target object, thereby enriching the information carried in the initial object representation of the target object for determining the items of interest to the target object. When updating the initial object representation based on the above-mentioned object multimodal similarity relationship, the initial object representation can be updated using the multimodal representation of the target object, thereby realizing implicit utilization of the multimodal information of the target object.

[0272] After obtaining the first updated item representation and the first updated object representation, the training device trains the target model based on the first updated item representation, the first updated object representation and the item multimodal representation, wherein the item multimodal representation includes the representation of the target item in at least one predetermined mode, and the target model is used to obtain the target item representation of the target item and / or the target object representation of the target object. In the training process, the training device trains the target model based on the item multimodal representation, and can realize the training of the target model using the multimodal information of the target item carried by the item multimodal representation, thereby realizing the display and utilization of the multimodal information of the target item. The first updated item representation and the first updated object representation are both obtained by implicitly utilizing the multimodal information, so the training device trains the target model based on implicitly utilizing and explicitly utilizing the multimodal information of the target item and the multimodal information of the target object, which can improve the training effect of the target model, thereby improving the accuracy of the information carried by the target item representation output by the target model for determining whether the object is interested in the target item, and / or improving the accuracy of the information carried by the target object representation output by the target model for determining the object of interest to the target object.

[0273] See also Figure 5 , Figure 5 This is a schematic diagram of a structure of a recommendation device provided in an embodiment of the present application. The recommendation device 2 includes: an acquisition unit 21 and a determination unit 22. Specifically:

[0274] an acquisition unit 21, configured to acquire a target item representation of a target item and a target object representation of a target object, wherein the target item representation and the target object representation are obtained by processing an updated item representation and an updated object representation using at least part of a module of a target model, wherein the target model is trained according to the first aspect and any embodiment thereof, and the updated item representation and the updated object representation are updated during the training process of the target model;

[0275] A determination unit 22, configured to determine a target similarity between the target item representation and the target object representation;

[0276] The determining unit 22 is configured to determine that the target item is an item recommended to the target object when the target similarity is greater than or equal to a threshold.

[0277] In the embodiment of the present application, both the updated item representation and the updated object representation are updated during the training process of the target model, so the accuracy of the item recommendation information carried by the updated item representation and the accuracy of the object recommendation information carried by the updated object representation are both high. Because the target item representation and the target object representation are obtained by processing using at least part of the modules of the target model, and the target model is trained based on the training method described above, the accuracy of the item recommendation information carried by the target item representation and the accuracy of the object recommendation information carried by the target object representation are both high. Therefore, after obtaining the target item representation of the target item and the target object representation of the target object, the recommendation device determines the target similarity between the target item representation and the target object representation, and when the target similarity is greater than or equal to the recommendation threshold, determines that the target item is an item recommended to the target object, which can improve the accuracy of determining the items recommended to the target object.

[0278] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0279] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device 3 includes a processor 31 and a memory 32. Optionally, the electronic device 3 also includes an input device 33 and an output device 34. The processor 31, the memory 32, the input device 33 and the output device 34 are coupled via a connector, and the connector includes various interfaces, transmission lines or buses, etc., which are not limited in the embodiments of the present application. It should be understood that in each embodiment of the present application, coupling refers to mutual connection in a specific manner, including direct connection or indirect connection through other devices, for example, it can be connected through various interfaces, transmission lines, buses, etc.

[0280] The processor 31 may include one or more processors, for example, one or more central processing units (CPUs). When the processor is a CPU, the CPU may be a single-core CPU or a multi-core CPU. Optionally, the processor 31 may be a processor group consisting of multiple CPUs, and the multiple processors are coupled to each other via one or more buses. Optionally, the processor may also be other types of processors, etc., which are not limited in the embodiments of the present application.

[0281] The memory 32 can be used to store computer program instructions and various computer program codes including the program code for executing the solution of the present application. Optionally, the memory includes but is not limited to random access memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM), or portable read only memory (CD-ROM), which is used for related instructions and data.

[0282] The input device 33 is used to input data and / or signals, and the output device 34 is used to output data and / or signals. The input device 33 and the output device 34 can be independent devices or an integrated device.

[0283] It can be understood that in the embodiment of the present application, the memory 32 can be used not only to store relevant instructions, but also to store relevant data. The embodiment of the present application does not limit the specific data stored in the memory.

[0284] Understandably, Figure 6 Only a simplified design of an electronic device is shown. In practical applications, the electronic device may also include other necessary components, including but not limited to any number of input / output devices, processors, memories, etc., and all electronic devices that can implement the embodiments of the present application are within the protection scope of the present application.

[0285] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0286] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. Those skilled in the art can also clearly understand that the descriptions of the various embodiments of the present application have different focuses. For the convenience and brevity of description, the same or similar parts may not be repeated in different embodiments. Therefore, for parts not described or not described in detail in a certain embodiment, refer to the records of other embodiments.

[0287] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0288] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0289] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0290] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0291] A person skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by a computer program to instruct the relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned method embodiments. The aforementioned storage medium includes: a read-only memory (ROM) or a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

Claims

1. A training method, characterized in that: The method comprises: obtaining an initial object representation of a target object and an initial item representation of a target item; Based on the interaction relationship between the target object and the target item, and the item multimodal similarity relationship between different target items, the initial item representation is updated to obtain a first updated item representation, wherein the item multimodal similarity relationship includes a similarity relationship determined based on the representation of the target item in at least one predetermined modality; Based on the interaction relationship and the object multimodal similarity relationship between different target objects, the initial object representation is updated to obtain a first updated object representation, wherein the object multimodal similarity relationship includes a similarity relationship determined based on the representation of the target object in the at least one predetermined modality; A target model is trained based on the first updated item representation, the first updated object representation and the multimodal representation of the item, wherein the multimodal representation of the item includes a representation of the target item in at least one predetermined modality, and the target model is used to obtain a target item representation of the target item and / or a target object representation of the target object.

2. The method according to claim 1, characterized in that The updating of the initial item representation based on the interactive relationship between the target object and the target item, and the item multimodal similarity relationship between different target items, to obtain a first updated item representation, includes: Acquire first graph data, the first graph data comprising a first object node and a first item node, information carried by the first object node comprising the initial object representation, information carried by the first item node comprising the initial item representation, and edges between nodes in the first graph data being determined based on the interaction relationship; Acquire second graph data, where the second graph data includes a second item node, information carried by the second item node includes the initial item representation, and edges between nodes in the second graph data are determined based on the multimodal similarity relationship of the items; Processing the first graph data using a graph convolutional network to obtain processed first graph data; Processing the second graph data using the graph convolutional network to obtain processed second graph data; The first updated item representation is obtained based on the representation carried by the first item node in the processed first graph data and the representation carried by the second item node in the processed second graph data, wherein the first item node and the second item node are both nodes corresponding to the target item.

3. The method according to claim 2, characterized in that The at least one predetermined mode includes a text mode and a visual mode, and the obtaining of the second image data includes: Acquire a first initial visual representation of the target object and a first initial textual representation of the target object; Determining similarities between the first initial visual representations of different target objects to obtain a first visual similarity; Determining similarities between the first initial text representations of different target objects to obtain first text similarities; Based on the first visual similarity and the first textual similarity, edges of the second graph data are determined.

4. The method according to claim 2 or 3, characterized in that: The updating of the initial object representation based on the interaction relationship and the object multimodal similarity relationship between different target objects to obtain a first updated object representation includes: Acquire third graph data, the third graph data including a second object node, information carried by the second object node including the initial object representation, and edges of the nodes in the third graph data are determined based on the multimodal similarity relationship of the objects; Processing the third graph data using the graph convolutional network to obtain processed third graph data; The first updated object representation is obtained based on the representation carried by the first object node in the processed first graph data and the representation carried by the second object node in the processed third graph data, and the first object node and the second object node are both nodes corresponding to the target object.

5. The method according to claim 4, characterized in that The obtaining of the third image data comprises: Acquire a second initial visual representation of the target object and a second initial textual representation of the target object, wherein the second initial visual representation is obtained based on a visual representation of an object that has interacted with the target object, and the second initial textual representation is obtained based on a textual representation of the object that has interacted with the target object; Determining the similarity between the second initial visual representations of different target objects to obtain a second visual similarity; Determining the similarity between the second initial text representations of different target objects to obtain a second text similarity; Based on the second visual similarity and the second textual similarity, edges of the third graph data are determined.

6. The method according to claim 2 or 3, characterized in that: The using a graph convolutional network to process the first graph data to obtain the processed first graph data includes: The graph convolutional network is used to process nodes connected by some edges in the first graph data to obtain the processed first graph data.

7. The method according to any one of claims 1 to 3, characterized in that The training of the target model based on the first updated item representation, the first updated object representation and the item multimodal representation includes: determining an interaction similarity between the first updated item representation and the first updated object representation; Based on the interaction similarity, an interaction loss is obtained; when the target item and the target object have interacted before, the interaction loss is negatively correlated with the interaction similarity; when the target item and the target object have not interacted before, the interaction loss is positively correlated with the interaction similarity; The target model is trained based on the interaction loss and the multimodal representation of the item.

8. The method according to claim 7, characterized in that The multimodal representation of the object includes a first initial visual representation of the target object and a first initial textual representation of the target object. The training of the target model based on the interaction loss and the multimodal representation of the object includes: Using the target model, respectively perform a first conversion on the first initial visual representation, the first initial text representation, the first updated item representation, and the first updated object representation to obtain a first converted visual representation, a first converted text representation, a first converted item representation, and a first converted object representation, wherein the first initial visual representation corresponds to the first converted visual representation, the first initial text representation corresponds to the first converted text representation, the first updated item representation corresponds to the first converted item representation, and the first updated object representation corresponds to the first converted object representation; Using the target model, a second conversion is performed on the first initial visual representation, the first initial text representation, the first updated item representation, and the first updated object representation, respectively, to obtain a second converted visual representation, a second converted text representation, a second converted item representation, and a second converted object representation, wherein the first conversion and the second conversion are different conversion processes, the first initial visual representation corresponds to the second converted visual representation, the first initial text representation corresponds to the second converted text representation, the first updated item representation corresponds to the second converted item representation, and the first updated object representation corresponds to the second converted object representation; Obtaining a multi-view loss based on the first converted visual representation, the first converted text representation, the first converted item representation, the first converted object representation, the second converted visual representation, the second converted text representation, the second converted item representation, and the second converted object representation; The target model is trained based on the interaction loss and the multi-view loss.

9. The method according to claim 8, characterized in that The multi-view loss includes one or more of the following: a first multi-view loss, a second multi-view loss, and a third multi-view loss; The first multi-view loss is obtained based on the first transformed visual representation, the first transformed textual representation, the second transformed visual representation, and the second transformed textual representation; The second multi-view loss is obtained based on the first converted item representation, the first converted object representation, the second converted item representation and the second converted object representation; The third multi-view loss is obtained based on the first converted visual representation, the first converted text representation and the second converted item representation.

10. The method according to claim 8, characterized in that Before using the target model to perform the first conversion and the second conversion on the first initial visual representation, the first initial text representation, the first updated item representation and the first updated object representation, respectively, the first initial visual representation, the first initial text representation, the first updated item representation and the first updated object representation are converted respectively via a conversion module of the target model so that the dimensions of the converted representations are the same.

11. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: In the process of training the target model, updating the multimodal representation of the object to obtain an updated multimodal representation; Obtaining a modality consistency loss based on a difference between the updated multimodal representation and the multimodal features of the item; The target model is trained based on the modality consistency loss.

12. A recommendation method, characterized in that: The method comprises: Obtaining a target item representation of a target item and a target object representation of a target object, wherein the target item representation and the target object representation are obtained by processing an updated item representation and an updated object representation using at least part of a module of a target model, wherein the target model is trained according to the method of any one of claims 1 to 11, and the updated item representation and the updated object representation are updated during the training process of the target model; Determining a target similarity between the target item representation and the target object representation; When the target similarity is greater than or equal to the recommendation threshold, the target item is determined to be an item recommended to the target object.

13. A training device, characterized in that: The training device comprises: an acquisition unit, configured to acquire an initial object representation of a target object and an initial item representation of a target item; an updating unit, configured to update the initial item representation based on the interaction relationship between the target object and the target item, and the item multimodal similarity relationship between different target items, to obtain a first updated item representation, wherein the item multimodal similarity relationship includes a similarity relationship determined based on the representation of the target item in at least one predetermined modality; The updating unit is configured to update the initial object representation based on the interaction relationship and the object multimodal similarity relationship between different target objects to obtain a first updated object representation, wherein the object multimodal similarity relationship includes a similarity relationship determined based on the representation of the target object in the at least one predetermined modality; A training unit is used to train a target model based on the first updated item representation, the first updated object representation and the multimodal representation of the item, wherein the multimodal representation of the item includes the representation of the target item in at least one predetermined modality, and the target model is used to obtain a target item representation of the target item and / or a target object representation of the target object.

14. A recommendation device, characterized in that: The recommended devices include: an acquisition unit, configured to acquire a target item representation of a target item and a target object representation of a target object, wherein the target item representation and the target object representation are obtained by processing an updated item representation and an updated object representation using at least a part of a module of a target model, wherein the target model is trained according to the method according to any one of claims 1 to 11, and the updated item representation and the updated object representation are updated during the training process of the target model; A determination unit, configured to determine a target similarity between the target item representation and the target object representation; The determining unit is configured to determine that the target item is an item recommended to the target object when the target similarity is greater than or equal to a threshold.

15. An electronic device, characterized in that: include: A processor and a memory, wherein the memory is used to store computer program code, wherein the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the method as claimed in any one of claims 1 to 11, or the electronic device executes the method as claimed in claim 12.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the processor is caused to execute the method described in any one of claims 1 to 11, or the processor is caused to execute the method described in claim 12.

17. A computer program product, characterized in that The computer program product includes a computer program or instructions; when the computer program or instructions are run on a computer, the computer is enabled to execute the method according to any one of claims 1 to 11, or the computer is enabled to execute the method according to claim 12.

Images