Data recommendation and classification method, feature fusion model and electronic device

Abstract

This application provides a data recommendation and classification method, a feature fusion model, and an electronic device. The data recommendation method includes: determining target data; determining at least one related data among multiple data sets that is relevant to the target data based on the correlation information between data sets; fusing the feature information of the target data and the feature information of the at least one related data set to obtain a fused feature of the target data; and recommending at least one piece of recommended data to a user based on the fused feature of the target data. The technical solution provided in this application fuses the feature information of the target data with the feature information of at least one related data set to obtain the fused feature of the target data. Then, based on the fused feature that incorporates the feature information of other related data sets, data is recommended to the user, which helps to improve the diversity of recommendations and incorporates posterior information.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data recommendation and classification method, a feature fusion model, and an electronic device. Background Technology

[0002] Data-driven recommendation systems, such as text recommendations, video recommendations, and music recommendations, are now widely used. Currently, data-driven recommendations typically analyze user preferences based on historical data and then make recommendations accordingly. Alternatively, they may recommend relevant text, videos, or music based on the user's current browsing, viewing, or playing activity.

[0003] In existing technologies, data recommendation solutions only recommend similar data to users, resulting in insufficient recommendation diversity. Summary of the Invention

[0004] This application provides a data recommendation and classification method, feature fusion model, and electronic device that solves or at least partially solves the above-mentioned problems.

[0005] In one embodiment of this application, a data recommendation method is provided. The method includes:

[0006] Determine the target data;

[0007] Based on the correlation information between the data, at least one related data among multiple data is determined that is related to the target data;

[0008] By fusing the feature information of the target data and the feature information of at least one associated data, the fused feature of the target data is obtained;

[0009] Based on the fusion features of the target data, at least one piece of recommended data is recommended to the user.

[0010] In another embodiment of this application, a data recommendation method is also provided. The method includes:

[0011] In response to user actions on the interactive interface, output the first multimedia data;

[0012] Determine at least one second multimedia data related to the first multimedia data;

[0013] By fusing the feature information of the first multimedia data and the feature information of at least one second multimedia data, the fused feature of the first multimedia data is obtained;

[0014] Based on the fusion characteristics of the first multimedia data, recommended data is determined;

[0015] When the output conditions are met, the recommended data is output to the user.

[0016] In yet another embodiment of this application, a data classification method is also provided. The method includes:

[0017] Determine at least one second multimedia data associated with the first multimedia data;

[0018] By fusing the feature information of the first multimedia data and the feature information of at least one second multimedia data, the fused feature of the first multimedia data is obtained;

[0019] Based on the fusion characteristics of the first multimedia data, the category to which the first multimedia data belongs is determined.

[0020] In another embodiment of this application, a feature fusion model based on a relationship graph is provided. This feature fusion model based on a relationship graph includes:

[0021] A node sampling module is used to sample nodes in a relation graph for a first node in the relation graph, so as to obtain at least one second node related to the first node.

[0022] The node feature information determination module is used to configure corresponding embedding features for the first node and the at least one second node based on the multimodal information of the first node, the multimodal information of the at least one second node, and the edge information between the first node and the at least one second node; determine the feature information of the first node based on the multimodal information of the first node and the embedding features corresponding to the first node; and determine the feature information of the at least one second node based on the multimodal information of the at least one second node and the embedding features corresponding to the at least one second node.

[0023] The feature fusion module is used to input the feature information of the first node and the feature information of at least one second node into the feature fusion model, and execute the feature fusion model to obtain the fused features of the first node;

[0024] The optimization module is used to perform graph reconstruction tasks and masked node feature reconstruction tasks based on the fusion features of the first sample nodes, and obtain the execution results corresponding to each task; and optimize the parameters in the feature fusion model based on the execution results corresponding to each task.

[0025] In another embodiment of this application, an electronic device is also provided. The electronic device includes a memory and a processor; wherein the memory is used to store a program; the processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the above-described data recommendation methods; or to implement the steps in the above-described short video recommendation methods.

[0026] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a computer, can implement the steps in the above-described data recommendation methods; or implement the steps in the above-described short video recommendation methods.

[0027] This application also provides a computer program product. The computer program product includes a computer program that, when executed by a computer, enables the computer to implement the steps in the aforementioned data recommendation methods; or to implement the steps in the aforementioned short video recommendation methods.

[0028] In the technical solutions provided in the various embodiments of this application, the feature information of the target data is fused with the feature information of at least one related data to obtain the fused feature of the target data. Then, based on the fused feature that incorporates the feature information of other related data, data is recommended to the user, which helps to improve the diversity of recommendations and brings the integration of posterior information. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 A flowchart illustrating a data recommendation method provided in an embodiment of this application is shown;

[0031] Figure 2 This illustration shows a user clicking on video C to play it in full screen, and then displaying recommended videos through corresponding operations, as mentioned in this embodiment.

[0032] Figure 3 An example of the relationship graph mentioned in the embodiments of this application and a schematic diagram of node sampling are shown;

[0033] Figure 4 A schematic diagram illustrating the principle of the node feature information determination process in an embodiment of this application is shown;

[0034] Figure 5 An example diagram of the encoding module mentioned in an embodiment of this application is shown;

[0035] Figure 6 It shows Figure 5 A schematic diagram of the principle of a single-layer encoder in a computer.

[0036] Figure 7A schematic diagram of the PMGT training process is shown;

[0037] Figure 8 A flowchart illustrating a data recommendation method provided in another embodiment of this application is shown;

[0038] Figure 9a A flowchart illustrating a short video recommendation method provided in another embodiment of this application is shown;

[0039] Figure 9b A flowchart illustrating a data classification method provided in an embodiment of this application is shown;

[0040] Figure 10 A schematic diagram of the feature fusion model provided in an embodiment of this application is shown;

[0041] Figure 11 A flowchart illustrating the training method of a feature fusion model provided in an embodiment of this application is shown.

[0042] Figure 12 A schematic diagram of the structure of a data recommendation system provided in an embodiment of this application is shown;

[0043] Figure 13 A schematic diagram of the structure of a data recommendation device provided in an embodiment of this application is shown;

[0044] Figure 14 A schematic diagram of the structure of a data recommendation device provided in another embodiment of this application is shown;

[0045] Figure 15 A schematic diagram of the structure of a short video recommendation device provided in an embodiment of this application is shown;

[0046] Figure 16 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0047] Before introducing the various embodiments of this application, a brief explanation of the terms or concepts that will appear below will be provided to aid understanding.

[0048] Multimedia data refers to data that combines multiple media types (such as text, graphics, images, sound, video, etc.).

[0049] A modality (or modal) can refer to any source or form of information. For example, human senses of touch, hearing, vision, and smell; information media such as sound, video, and text; and various sensors such as radar, infrared, and accelerometers. Each of these can be considered a modality. Furthermore, a modality can have a very broad definition; for example, two different languages ​​can be considered two modalities, or even datasets collected under two different circumstances can be considered two modalities, and so on. In this article, modality has two meanings: one refers to features; the other refers to media types. Multimodal information refers to data that contains multiple feature representations or multiple media types. For example, the multimodal information of a short video may include, but is not limited to, visual information (e.g., video frame information), text information (e.g., video title), and audio information (e.g., background music). The multimodal information of a webpage may include text information and image information (illustrations within text), etc.

[0050] A relational graph (or homogeneous graph) is a graph in which nodes represent data (short videos, web pages, music, articles, etc.), edges indicate the correlation between nodes, edge weights indicate the magnitude of the correlation, and nodes have multiple modal features.

[0051] Pre-training is the process of training a model using training samples and training tasks, and then saving the trained model or the model's output representation for use in downstream tasks.

[0052] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. In some processes described in the specification, claims, and accompanying drawings of this application, multiple operations appearing in a specific order are included. These operations may be executed in any order or in parallel. Operation numbers such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different modules, models, devices, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types. In addition, the following embodiments are only some embodiments of this application, not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0053] Short video applications are currently very popular. Users can upload their own short videos or watch other users' short videos through the application. The content of these short videos can include user-generated comedy videos, product promotion videos, knowledge-sharing videos, and so on. While a user is watching a short video, the server recommends other short videos, automatically playing the recommended video after the current one finishes playing. As long as the user doesn't close the short video playback interface, the application will continuously play the short videos recommended by the server. However, current technology recommends the next short video based on the currently playing video. This means that current technology only recommends short videos similar to the currently playing video. For example, a user might find that after watching a "clay crafting video," the next recommended video is likely to be a crafting video or a clay product promotion video, even though the user might want to see other content. At this point, the user becomes uninterested in the recommended videos, demonstrating the low diversity of recommendations provided by current technology.

[0054] This application provides technical solutions in the following embodiments to enhance recommendation diversity. The example of short video recommendation illustrates the rationale behind the proposed technical solutions. In essence, the technical solutions provided in this application can be applied not only to short video recommendations but also to product recommendations, text recommendations, music recommendations, image recommendations, and so on. Furthermore, the core of the technical solutions provided in the following embodiments of this application lies in the following: in addition to the features of the target data (such as the short video currently being watched by the user, the music currently being listened to, etc.), other data related to the target data (such as short videos the user has watched in the past, music previously listened to, short videos watched in the past by other users who have also watched the same short video, etc.) are also integrated. The integrated features are used as the fusion features of the target data. These fusion features of the target data can be used for data recommendation and also for data classification (such as product classification, short video classification, text classification, music classification, etc.).

[0055] Figure 1 A flowchart illustrating a data recommendation method according to an embodiment of this application is shown. The execution entity of the method provided in this embodiment can be a server or a client; this embodiment does not specifically limit this. Figure 12 In the illustrated embodiment, server 22 can be a server, a server cluster, a virtual server built on a server or cluster, or the cloud, etc. Of course, the execution subject of some steps in the method provided in this embodiment can be a server, and the execution subject of some steps can be a client (e.g., ...). Figure 12In the illustrated embodiment, 21 corresponds to the client. For example, the execution entity corresponding to steps 101 and 102 below can be a client, and the execution entity corresponding to steps 103 and 104 below can be a server. Alternatively, the execution entity corresponding to step 101 below can be a client, and the execution entity corresponding to steps 102-104 below can be a server. Alternatively, the execution entity corresponding to steps 101-103 below is a client, and the execution entity corresponding to step 104 below can be a server; and so on. Specifically, the data recommendation method includes:

[0056] 101. Determine the target data;

[0057] 102. Based on the correlation information between the data, determine at least one related data among multiple data that is related to the target data;

[0058] 103. By fusing the feature information of the target data and the feature information of at least one associated data, the fused feature of the target data is obtained;

[0059] 104. Based on the fusion features of the target data, recommend at least one piece of recommended data to the user.

[0060] In the above 101, the target data can be any one type of data, or any combination of multiple types of data, including text, images, videos, and audio, viewed by the user through the client. This embodiment does not limit this. For example, Figure 2 In the scenario shown, a user opens a data page of an application, which displays multiple short videos. The user clicks to open one of the short videos, and is redirected to the playback interface of that short video, such as playing video C in full screen. At this time, the short video C playing in full screen is identified as the target data in this embodiment.

[0061] In step 102 above, the multiple data can be determined based on historical data related to the user. For example, data on the user's historical browsing, playback, commenting, favorites, adding to cart, or adding to playlists, etc. In this embodiment, the multiple data may include all data from the user's historical browsing, playback, commenting, favorites, adding to cart, or adding to playlists; or the multiple data may only include a subset of data selected from all the user's historical browsing, playback, commenting, favorites, adding to cart, or adding to playlists. This embodiment does not specifically limit the selection rules.

[0062] Furthermore, the multiple data sets can also be determined based on historical data of other users associated with the user. These associated users can include: multiple users who have viewed the same video, the same webpage, etc.; multiple users who have played the same music; multiple users who have given the same or similar evaluations of the same data (such as videos, webpages, music, or images); multiple users who have favorited the same videos or products, etc. This embodiment does not limit this. For example, the multiple data sets can be determined from the historical data of at least one associated user. A portion of the data can be extracted from all the historical browsing, playback, commenting, and favorites data of the at least one associated user as the multiple data sets.

[0063] Of course, the multiple data can also be determined by combining historical data related to the user and historical data of at least one associated user associated with the user. This embodiment does not limit this.

[0064] Two pieces of data that are related possess the aforementioned association information. In one feasible instance, the two pieces of data are associated when they share the same tag. In this case, the association information may include: the number of times the two pieces of data share the same tag. Two pieces of data may also have a sequential relationship based on user behavior; for example, if a user watches the first short video and then clicks to watch the second, the two short videos are associated. In this case, the association information may further include: an identifier indicating a continuous playback relationship between the two pieces of data. Another example is when two pieces of data exhibit the same user behavior; for instance, if a user saves both the first and second audio files, the two audio files are associated. In this case, the association information may further include: an identifier indicating the same user behavior between the two pieces of data and a user behavior identifier; and so on. This embodiment can further explore other aspects of association, which are not listed here. In specific implementation, the association can be determined according to the actual scenario.

[0065] In the above 103, the feature information corresponding to different types of data will also be different. For example, if the target data only contains text data, then the feature information of the target data may only include text features extracted from the text data. If the target data only contains audio data, then the feature information of the target data may only include audio features extracted from the audio data. Similarly, if the target data only contains image data, then the feature information of the target data may only include image features extracted from the image data. However, when the target data is multimedia data, such as short videos or movies, this type of data may include, but is not limited to: text information (such as short video titles, movie names, etc.), video information, audio information, etc. For multimedia data, its feature information needs to be obtained by fusing features extracted from each of the above modalities.

[0066] Similarly, when the associated data contains only one modality, its feature information is the feature extracted from that modality; however, when the associated data contains multiple modalities, its feature information needs to be obtained by fusing the features extracted from each modality separately.

[0067] As can be seen, when the target data and associated data contain multiple modal data, the "fusion" in step 103 of this embodiment includes two steps: one step is the fusion of the multimodal features of the target data and the fusion of the multimodal features of the associated data; the other step is the fusion of the feature information of the target data and the feature information of at least one associated data.

[0068] In this embodiment, the feature information of at least one related data is integrated into the feature information of the target data. The purpose is to allow the target data to have its own features as well as the features of the related data, so as to bring in the posterior information and thus better serve the data recommendation.

[0069] It should be noted here that, for better integration, the "fusion" step described above can also incorporate features representing the relationships between the target data and its associated data. This means that the target data not only possesses its own features and features from its associated data, but also implicitly contains features representing the relationships between the target data and its associated data, such as location-based and role-based embedding features. This will be explained in more detail below; please refer to the relevant content therein.

[0070] As mentioned above, the fusion features of the target data in step 104 not only possess their own features but also include features from related data. Furthermore, they can also include features representing the relationship between the target data and each related data point. In step 104, data recommendation based on these fusion features can effectively improve the diversity of recommendations, recommending not only data with features similar to the target data but also data similar to related data.

[0071] In practical implementation, the fusion features of the target data can be used as input to the recommendation model. The recommendation model has two stages: a recall stage, in which at least one candidate data can be recalled directly based on the fusion features of the target data; and a ranking stage, in which a ranking operation is performed based on the fusion features of the target data and the features of each candidate data. Further, the recommendation model includes two models corresponding to different stages, such as a recall model corresponding to the recall stage and a ranking model corresponding to the ranking stage. The fusion features of the target data are first input to the recall model to recall at least one candidate data; then, the fusion features of the target data and the features of at least one candidate data are used as input to the ranking model to obtain the ranking result. The top N candidate data in the ranking result can be used as recommended data to the user. N can be 1, 2, 5, 10, or more; this embodiment does not limit this.

[0072] In summary, the technical solution provided in this embodiment first determines at least one related data that is related to the target data based on the correlation information between the data; then, the feature information of the target data is fused with the feature information of at least one related data, and the fused feature is used as the fused feature of the target data, so as to recommend data to the user based on the fused feature. It can be seen that in this embodiment, when recommending based on the target data, because the fused feature not only has the features of the target data, but also the features of the related data, and further, it can also have the features of the relationship between the target data and each related data, the feature used as the recommendation reference is not a single feature of the target data. The recommendation result can include recommended data similar to the target data, and can also include data similar to the related data, etc. Therefore, the solution provided in this embodiment can effectively improve the diversity of recommendations.

[0073] See Figure 3 One feasible technical solution, as shown, involves constructing a relationship graph based on the correlation information between multiple data points after identifying them, and then sampling nodes based on the relationship graph to sample at least one related data point. For example, in this embodiment, step 102, "determining at least one related data point among the multiple data points that is related to the target data based on the correlation information between the data points," can be implemented using the following steps:

[0074] 1021. Construct a relationship graph based on the correlation information between data; wherein the relationship graph includes multiple nodes and edge information reflecting the relationship between nodes; different nodes correspond to different data, and the node corresponding to the target data in the relationship graph is the target node;

[0075] 1022. For the target node, sample the nodes in the relationship graph to sample at least one associated node related to the target node; wherein, the data corresponding to the associated node is the associated data related to the target data.

[0076] See Figure 3 The diagram shown assumes that node h0 corresponds to the target data in this embodiment. h1~h 16 This refers to multiple data sets in this embodiment. An edge in the relationship graph, i.e., a line connecting two nodes, represents a relationship between the two nodes. Edge information can include the association information between the data corresponding to the two nodes. Taking the example of each node corresponding to different short videos in the relationship graph, the short videos have tag information. If two short videos corresponding to two nodes have a certain number (e.g., 1, 2, 5 or more) of the same tags, it indicates that the two nodes are associated, and an "edge" is established between the two nodes. Edge information can be the edge weight, which is related to the number of common tags of the two short videos corresponding to the two nodes. The features of a node are the features of the short video corresponding to the node, such as the multimodal features of the short video. It should be added here that: to alleviate the problem of large variance in the number of tags, a scaling operation can be added in this embodiment.

[0077] The above 1022 can be implemented using a corresponding sampling algorithm. One such sampling algorithm is illustrated below: "For the target node, sample the nodes in the relationship graph to sample at least one associated node related to the target node," including:

[0078] S1. Obtain the target number of times and the number of samples;

[0079] S2. In a single sampling process, the target node is used as the sampling origin, and at least one neighboring node in the relationship graph that is adjacent to the origin is sampled.

[0080] S3. Determine whether the number of samples has reached the target number;

[0081] S4. When the number of samplings is not greater than the target number, proceed to the next sampling iteration; in the next sampling iteration, take any one of the at least one neighboring nodes as the sampling origin, and sample at least one neighboring node in the relationship graph that is adjacent to the origin;

[0082] S5. When the number of samplings is greater than the target number, determine the number of neighboring nodes sampled from the neighboring nodes sampled in the sampling iteration of the target number as associated nodes.

[0083] This embodiment proposes an efficient parallel sampling algorithm. Specifically, starting from the target node, iteratively samples a fixed number of nodes at a fixed depth, where the sampling probability of a node is proportional to the edge weight. Figure 3 The above process will be explained using the relationship diagram shown as an example to facilitate understanding.

[0084] like Figure 3 The diagram shown assumes that the target node is h0, the number of target iterations (same as the sampling depth) is 2, and the number of samples is 4.

[0085] Step 11: One sampling iteration, B = 1; sampling origin = h0.

[0086] Step 12: Obtain the neighbor nodes of the sampling origin to get the neighbor node set.

[0087] Among them, h i This is the sampling origin. For example, Figure 3 In the above, when the sampling origin = h0, the neighboring nodes of h0 include: h8, h9, h1, h4, h 14 Obtain the set of neighbor nodes

[0088] Step 13: Next sampling iteration, B = B + 1.

[0089] Step 14: Determine that B is not greater than 2, proceed to the second sampling iteration, and replace the target node with... Each node in the graph is taken as the sampling origin, and at least one neighboring node adjacent to each sampling origin in the sampling relationship graph, or, from Randomly sample a subset of nodes, such as h9, h1, and h4, and use these three nodes as the sampling origin. Sample at least one neighboring node adjacent to each sampling origin in the graph to obtain the neighbor node set. Then return to step 13 until B is greater than the target number (i.e., 2).

[0090] The following example replaces the target node with Taking each node in the array as an example, the specific set of neighbor nodes obtained is as follows:

[0091] Target node = h8, obtained from neighboring nodes

[0092] Target node = h9, obtained from neighboring nodes

[0093] Target node = h1, obtained from neighboring nodes

[0094] Target node = h4, obtained from neighboring nodes

[0095] Step 15: When B is greater than 2, determine the number of neighboring nodes (e.g., 4) as associated nodes from all neighboring nodes sampled in the two sampling iterations.

[0096] That is, from as well as In the sample, four neighboring nodes are selected as associated nodes, such as h1, h2, h3 and h4.

[0097] As can be seen from the examples above, duplicate nodes are inevitable in the obtained set of neighbor nodes. For example... as well as h4, h6 and h 14 Both were sampled twice. In one feasible technical solution, in steps S5 and 15 above, the associated nodes can be determined based on the occurrence count (or frequency) of each neighbor node and the distance of each neighbor node from the target node. That is, in this embodiment, step S5, "determining the number of neighbor nodes sampled from the neighbor nodes sampled in the target number of sampling iterations as associated nodes" may include:

[0098] S51. Obtain the occurrence count of each neighbor node in the neighbor node set collected in each sampling iteration;

[0099] S52. Obtain the distance between each neighbor node in the neighbor node set collected in each sampling iteration and the target node;

[0100] S53. Based on the occurrence frequency of each neighbor node and the distance of each neighbor node from the target node, determine the number of neighbor nodes sampled from the sampling iterations of the target number as associated nodes.

[0101] More specifically, the product of the number of times a neighboring node appears (or its frequency) and the distance of that neighboring node from the target node can be calculated, and this product is used as the importance. All neighboring nodes sampled from the target number of sampling iterations are then sorted according to their importance.

[0102] For example, select the top-k nodes as the associated nodes of the target node. Here, top-k equals the number of samples.

[0103] Again, as mentioned above Figure 3 The example shown illustrates that all neighbor nodes sampled from the target number of sampling iterations include: h8, h9, h1, h4, h 14 h7, h6, h3, h 10 ,h2 h15 ,h 16 ,h 14 ,h 13 ,h 12 .in,

[0104] h8 appears once, and is 1 jump away from h0; correspondingly, the importance of h8 is 1 * 1 jump.

[0105] h4 appears twice, and is 1 jump away from h0; correspondingly, the importance of h4 is 2 * 1 jump.

[0106] h2 appears once, and is 2 jumps away from h0; correspondingly, the importance of h4 is 1 * 2 jumps; and so on.

[0107] Similarly, the importance scores of each of the other neighboring nodes can be calculated in turn.

[0108] It should be noted that each node mentioned above corresponds to one piece of data. The associated nodes sampled using the above method correspond to one piece of associated data.

[0109] Based on the above, when sampling multiple related data points associated with the target data, the method provided in this embodiment may further include the following steps:

[0110] S6. Determine the importance of multiple related data;

[0111] S7. Configure corresponding embedding features for the multiple related data according to their importance;

[0112] S8. Configure embedding features for the target data;

[0113] S9. Obtain the features of at least one modality of the target data and the features of at least one modality of any of the multiple associated data;

[0114] S10. Determine the feature information of the target data based on the features of at least one modality of the target data and the embedding features corresponding to the target data;

[0115] S11. Determine the feature information of the associated data based on the features of at least one modality of any of the multiple associated data and the embedding features corresponding to the associated data.

[0116] For the calculation of the importance of related data in S6 above, please refer to the content above.

[0117] In S7 above, multiple related data can be sorted according to their importance. In this embodiment, the embedding features configured for each related data may include position features and role features. For example, a position feature based on the position identifier of the related data in the sorting (e.g., first, second, or third) can be configured for that related data. Relative to the target data, the related data plays a related role, not a primary role; therefore, a role feature based on the "related role" can be configured for the related data. The role features of multiple related data can be the same, all being "related role". In specific implementation, the first role identifier (e.g., 1) can be associated with the primary role (i.e., the role corresponding to the target data), and the second role identifier (e.g., 0) can be associated with the related role (i.e., a non-primary role).

[0118] In S8 above, to facilitate subsequent calculations, this embodiment can concatenate the feature information of the target data with the feature information of multiple related data to form a combined feature. As mentioned above, sorting multiple related data according to their importance yields a data sequence, in which the related data are arranged in the sorting order. During concatenation, the target data can be added to the data sequence, either at the beginning or the end. At this point, the target data has a position identifier (e.g., first, last, e.g., fifth) in the data sequence; thus, a position feature based on the position identifier corresponding to its location can be configured for the target data. Furthermore, this step can also configure a role feature based on the "primary role" for the target data.

[0119] In S9 above, multimodal information (including features of multiple modalities) can be found in the explanation above. For example, short videos contain rich modal information, including but not limited to: video frame information, title text information, and audio frame information, etc. Using multimodal information can bring several benefits. For instance, multimodal information can mine the content representation of data, improving recall relevance; in scenarios where user historical behavior is relatively sparse, multimodal information can effectively boost the exposure of less popular data. Therefore, effectively integrating multimodal information into recommendation scenarios has very high application value.

[0120] The multimodal features mentioned above are extracted from the multimodal information. Taking short videos as an example, title features can be extracted from the title information; short video frame features can be extracted from the video frames; and audio frame features can be extracted from the audio information. Specifically, the title features extracted from the short video title can be considered as features corresponding to the first modality; the video frame features extracted from the short video frames can be considered as features corresponding to the second modality; and the audio frame features extracted from the short video audio frames can be considered as features corresponding to the third modality. In practice, the TF-IDF method or the BERT network can be used to extract title features from the title information of the short video; the Inception-v4 network can be used to extract features from each frame of the video, and then the average value of all frame features can be taken to obtain the visual features of the short video.

[0121] In one specific implementation, the embedding features configured for the target data and the embedding features configured for the at least one associated data mentioned above may include, but are not limited to, location features and role features. When the target data in this embodiment has multimodal features, correspondingly, step S10 in this embodiment, "determining the feature information of the target data based on the multimodal features of the target data and the embedding features corresponding to the target data," may include:

[0122] S1001. Determine the corresponding weights for each modality of the multimodal features in the target data;

[0123] S1002. Determine the content features of the target data based on the multimodal features of the target data and the weight of each modal feature in the multimodal features;

[0124] S1003. Aggregate the content features of the target data and the location and role features corresponding to the target data to obtain the feature information of the target data.

[0125] In step S1001 above, the influence weight of features corresponding to different modalities of the target data on the target data can be calculated first. Then, based on the influence weight of features corresponding to different modalities on the target data, the weight of each modality-related feature in the multimodal features of the target data can be determined. This embodiment does not limit the algorithm for calculating the influence weight; it can be determined by combining user behavior data, or by analyzing the multimodal features of the target data and the multimodal features of at least one associated data. Alternatively, this embodiment can use a self-attention mechanism to calculate a score for each modal feature of the target data. The score corresponding to each modal feature can be used as its weight, or the weight can be determined based on the score corresponding to each modal feature. For details on the attention mechanism, please refer to relevant literature; this embodiment will not elaborate further.

[0126] In S1002 above, the content features of the target data can be obtained by weighted summing of the features of all modalities of the target data.

[0127] Based on the embedded features mentioned above, which include location features and role features, in this embodiment S1003, the content features, location features, and role features of the target data can be added and aggregated to obtain the feature information of the target data.

[0128] To make it easier to understand, the following will be combined with... Figure 4 The example shown illustrates the above process. Figure 4 The target data shown is t; the multimodal features of the target data t include: first modality features Second modal features Third modal characteristics The weights corresponding to each modality feature can be determined through the attention mechanism, such as the first modality feature. The corresponding weight is Second modal features Corresponding weights Third modal characteristics Corresponding weights Content features M of target data t t :

[0129]

[0130] Figure 4 P shown in t R represents the location features of the target data t. t The role characteristics of target data t. Feature information of target data t.

[0131]

[0132] Similarly, the characteristic information of associated data can also be determined using the above method.

[0133] Furthermore, step 103 in this embodiment, "fusing the feature information of the target data and the feature information of at least one associated data to obtain the fused feature of the target data," may specifically include:

[0134] 1031. Determine the feature similarity between the feature information of the target data and the feature information of at least one associated data;

[0135] 1032. Based on feature similarity, determine the attention weight corresponding to the at least one associated data;

[0136] 1033. Based on the attention weights corresponding to the at least one associated data, the feature information of the target data and the feature information of the at least one associated data are fused and encoded to obtain the fused features of the target data.

[0137] The method for determining the similarity of feature information of two data in step 1031 above is not limited in this embodiment. For example, the feature similarity of two data can be determined by calculating the distance (such as cosine distance) between the feature information of two data.

[0138] The reason for determining the attention weights corresponding to each associated data based on feature similarity in step 1032 of this embodiment is to focus on associated nodes with high attention weights during feature fusion through an attention mechanism. Normally, it's easier to focus on associated nodes that are similar to the target node, which reduces the diversity of subsequent recommendations. To improve diversity, this embodiment adds an attention mechanism to enhance diversity, namely, increasing the attention weights of associated nodes with low feature similarity. Specifically, embodiment 1032 can be defined as follows: Arranging the multiple associated data in descending order according to their respective feature similarities; sequentially configuring attention weights for each associated data in descending order, wherein the attention weights corresponding to later associated data are higher than those corresponding to earlier associated data.

[0139] Furthermore, the solution provided in this embodiment can use a Transformer (which can be translated as deformer, transformer, or converter). A Transformer is a model whose main component is attention. For example, Figure 5 As shown, the Transformer may include an encoding module. The encoding module includes multiple encoding layers (Encoders), each layer containing two sub-layers (e.g., ...). Figure 6 (As shown). For example, Figure 5In the example shown, the Transformer can include 6 layers of Encoders. Figure 6 As shown, the attention mechanism mentioned above can be set in each encoder layer, or it can be set in some encoder layers, namely, increasing the attention weights of related nodes with low feature similarity. Multi-head attention is composed of multiple self-attention mechanisms. Figure 6 As can be seen, the Encoder block contains a Multi-Head Attention layer, as well as Add and Norm layers. Add represents residual connections to prevent network degradation; Norm represents Layer Normalization, used to normalize the activation values ​​of each layer. In this embodiment, for... Figure 6 The specific structure of the feedforward network in this embodiment is not limited and can be found in relevant literature. Step 103, "fusing the feature information of the target data and the feature information of at least one associated data to obtain the fused feature of the target data," can be implemented using a Transformer. Specifically, a diversity-enhancing attention mechanism is added to the Transformer's attention mechanism, i.e., increasing the attention weight between data with dissimilar features. Therefore, the diversity and richness of the representation can be improved through the encoding module in the Transformer.

[0140] like Figure 6 As shown, the determination of attention weights in the above steps can be achieved using the multi-head attention mechanism in the encoding layer of the Transformer. The concept of multi-head attention is actually quite simple: multiple sets of Q, K, and V can be defined, each focusing on different related data with low feature similarity. For example... Figure 6 As shown, the multi-head attention mechanism can be implemented in an encoder layer within a Transformer. This encoder module contains multiple encoder layers, which is the feature fusion model discussed below.

[0141] That is, step 103 of this application embodiment, "fusing the feature information of the target data and the feature information of at least one associated data to obtain the fused feature of the target data," may include the following steps:

[0142] 1034. Obtain the feature fusion model;

[0143] 1035. Input the feature information of the target data and the feature information of at least one associated data into the feature fusion model, and execute the feature fusion model to obtain the fused features of the target data;

[0144] The feature fusion model is obtained through a pre-training process. As can be seen from the above, the feature fusion model in step 1034 can be a Transformer, and more specifically, the encoding module in a Transformer.

[0145] Furthermore, the method provided in this embodiment may further include a pre-training step of the feature fusion model. That is, the method provided in this embodiment may further include the following steps:

[0146] 105. Obtain a sample graph, wherein the sample graph includes sample nodes and edge information reflecting the relationships between sample nodes;

[0147] 106. For the first sample node in the sample graph, sample nodes in the sample graph are sampled to sample at least one second sample node related to the first sample node;

[0148] 107. Based on the features of at least one modality of the first sample node and the features of at least one modality of the at least one second sample node, determine the feature information of the first sample node and the feature information of the at least one second sample node, respectively.

[0149] 108. Input the feature information of the first sample node and the feature information of the at least one second sample node into the feature fusion model to obtain the fusion feature of the first sample node and the fusion feature of the at least one second sample node;

[0150] 109. Based on the fusion features of the first sample node and the fusion features of the at least one second sample node, perform graph reconstruction task and masked sample node feature reconstruction task to obtain the execution results corresponding to each task;

[0151] 110. Optimize the parameters in the feature fusion model based on the execution results of each task.

[0152] like Figure 7 As shown, this embodiment provides a framework structure diagram of a feature fusion system based on a relationship graph. See also... Figure 7 As shown, the feature fusion system includes: a node sampling layer, a node feature information determination layer, a feature fusion layer, and an optimization layer. The feature fusion model in this embodiment may include... Figure 7 The feature fusion layer in the middle. This feature fusion layer can be... Figure 5 The multi-layered structure shown refers to a multi-layered coding system, where each coding layer includes... Figure 6The multiple sub-layers shown.

[0153] See Figure 7 As shown, the training process described above in this embodiment may include, but is not limited to, the following four parts:

[0154] Part (1), Construction of the Relationship Diagram

[0155] Part (2), Node Sampling and Determination of Node Feature Information

[0156] Part (3), Feature Fusion Model

[0157] Part (4) Optimize the parameters in the feature fusion model based on the reconstruction task.

[0158] The construction process of the relationship graph in part (1) is not limited in this embodiment. For example, based on the labels included in the data corresponding to all sample nodes, when two data packets contain one, two, three or more identical labels, edge information (such as...) can be established for the nodes corresponding to the two data packets respectively. Figure 7 The connection between two nodes in the graph (the line connecting them); edge information can include edge weights, which can be the number of common labels. Through the above process, a graph can be established as follows: Figure 7 The diagram shown is a relational graph. This graph is used for training the feature fusion model. To distinguish it from the "relationship graph" mentioned above, the relationship graph used for training here will be called a "sample graph." It should be added that in practical applications, historical user behavior data can also be used. For example, if a user browses data A and also browses data B, it indicates that data A and data B are related. Edge information between the nodes corresponding to the two data points can be established to construct a relational graph.

[0159] In part (2), after the sample graph is constructed, for the first sample node h0 in the sample graph (i.e., any sample node in the sample graph), at least one second sample node related to the first sample node h0 is sampled. For example Figure 7 In the example shown, the second sample nodes associated with the first sample node h0 are: h1 h2 h3 h4.

[0160] The first sample node and at least one second sample node are combined to form a node sequence. The position (or order) of the at least one second sample node in the node sequence can be determined based on its importance. For example, the importance of each second sample node can be calculated based on the frequency at which it is sampled and its distance from the first sample node (e.g., 1 hop or 2 hops). For details on the sampling frequency and importance calculation of the second sample nodes, please refer to the relevant content above; it will not be repeated here.

[0161] The first sample node can be either at the beginning or at the end of the ordered nodes. For example... Figure 7 In the example shown, the node sequence is {h0 h1 h2 h3 h4}.

[0162] After assembling the node sequence, corresponding embedding features can be configured for each node in the sequence (such as the first sample node and at least one second sample node) based on the position of each node in the sequence and the corresponding role attribute of each node (e.g., the first sample node has the protagonist attribute, and the second sample node has the association attribute related to the first sample node). It should be noted that the configuration of embedding features can be found in the relevant content above.

[0163] Subsequently, the feature information of the first sample node can be determined based on the features of at least one modality of the first sample node and the corresponding embedding features. That is, the feature information of the first sample node contains not only feature terms corresponding to at least one modality, but also embedding features. Similarly, the feature information of the associated data is determined based on at least one modality feature of the associated data and the corresponding embedding features. For details on determining the feature information of sample nodes, please refer to the relevant content above. Figure 4 This will not be elaborated upon here.

[0164] In part (3), the feature information of the first sample node obtained in part (2) and the feature information of at least one second sample node are input into the feature fusion model. After executing the feature fusion model, the fused features of the first sample node are obtained. In addition, the feature fusion model can also obtain the fused features of the second sample node.

[0165] In part (4), the fusion features of the first sample node and the fusion features of the second sample node obtained in part (3) are used to perform the reconstruction task. The reconstruction task may include: graph structure reconstruction task (i.e. Figure 7 The task of reconstructing the features of GSR (Gross Reconstruction System) and masked nodes (i.e., GSR) Figure 7The graph structure reconstruction task can be simply understood as: the graph structure reconstructed based on the fusion features of each sample node obtained in part (3) is compared with the graph structure of the sample graph, and the parameters in the feature fusion model are optimized by using this reconstruction difference. The feature reconstruction task of masked nodes can be simply understood as: masking some nodes in the node sequence (e.g., masking one node), decoding the feature information of the masked node through the decoding process; optimizing the parameters in the feature fusion model based on the difference between the decoded feature information of the masked node and the actual feature information of the masked node. The feature information of the masked node in the node sequence can be set to 0. In the specific training and optimization process, the graph structure reconstruction task and the feature reconstruction task of masked nodes can be represented by corresponding objective functions; the values ​​of the objective functions corresponding to the graph structure reconstruction task and the feature reconstruction task of masked nodes can be calculated based on the fusion features of each sample node obtained in part (3); the parameters in the feature fusion model are optimized by combining the values ​​of the objective functions corresponding to these two tasks.

[0166] In this embodiment, the objective function corresponding to each task may be referred to as a loss function in some specific embodiments. The specific expression of the objective function is not limited in this embodiment. Specific function expressions will be listed below in conjunction with specific scenarios. Furthermore, the optimization process of parameters in the feature fusion model is not specifically limited in this embodiment; please refer to the relevant literature for implementation details.

[0167] Repeat steps (2), (3), and (4) until training is complete. For example, after performing the reconstruction task in step (4), if the difference between the reconstructed graph structure and the sample graph structure meets the first preset requirement, and the difference between the reconstructed feature information of the masked node and the actual feature information of the node meets the first preset requirement, then training is considered complete. Essentially, this means minimizing the differences in the graph structure and the differences in the feature information of the masked node. Once the goal of minimizing the differences is achieved, training is complete.

[0168] Multimodal information from data has proven effective in improving accuracy in recommendation scenarios. For example, short videos, which include video frame features, title features, and audio frame features, can have their content representation mined for recommendation, improving recall relevance. In scenarios where user behavior is sparse, multimodal information can effectively boost the exposure of less popular short videos. In this embodiment, such as... Figure 7 The training process of the feature fusion system based on relational graphs shown can be understood as: learning node representations by considering the multimodal features of sample nodes (i.e., data) and the relationships between nodes (such as edge information).

[0169] The technical solution provided in this embodiment will be described below with reference to specific implementation schemes. The relationship diagram mentioned in this document can be represented as: G = (ν, ξ), to provide a unified view of the multimodal features of nodes and their relationships. Here, ν represents a node, and ξ represents the multimodal features of each node h. Let h represent the i-th modal feature of node h. Node h has m modal features. In the following text, h represents the target node for which sampling is performed. The target node is the node used as the sampling reference in a single sampling process. In a single sampling process, the sampling is performed on the neighboring nodes of the target node.

[0170] The MCN algorithm mentioned above can be represented as follows:

[0171] Input: Relationship graph G, sampling step B, sampling depth K, sampling size The number of associated nodes, S;

[0172] Output: Sampling sequence of associated nodes C B ;

[0173]

[0174] In each sampling process, only neighboring nodes at the same depth relative to the target node h are sampled; therefore, the sampling depth K is equivalent to the target number mentioned above. Alternatively, in another feasible example, in each sampling process, neighboring nodes within a sampling depth K (inclusive) of the target node h are sampled. For example, if the sampling depth K = 2, the sampled set of neighboring nodes includes: neighboring nodes 1 hop away from the target node h, and neighboring nodes 2 hops away from the target node h. In the relationship graph G, N is used... h Represents the single-hop neighbor nodes (or single-hop associated nodes) of h, and denoted by ω. ht This represents the edge information between nodes h and t (such as the edge weight, where ω). ht >0. For node h, use C. h Let G be the associated nodes sampled by the MCN sampling algorithm. Given a relationship graph G and the associated nodes of each node, the goal of the relationship graph-based feature fusion system is to obtain node representations that can capture the multimodal features of nodes and node relationships. Then, the learned node representations can be directly applied to downstream tasks. These downstream tasks can include, but are not limited to, data recommendation, data classification (such as video classification, image classification, etc.), etc.

[0175] The feature fusion system based on the relationship graph in this embodiment, such as Figure 7The system framework example shown includes nodes whose feature information participating in the fusion may contain multimodal information. Therefore, the feature fusion system based on the relational graph in this embodiment can also be called a "pre-training graph transformer with multimodal side information for recommendation", abbreviated as PMGT.

[0176] For each node h in the relationship graph G, there are related nodes in the relationship graph G, which may be one or more. In this embodiment, the feature information of the related nodes is fused into the feature information of node h, which helps to enrich the representation of node h, that is, the fused feature of node h.

[0177] The MCN algorithm mentioned above includes multiple sampling iterations. Each sampling iteration samples the associated nodes related to the target node at a predefined sampling depth K, targeting the origin node. Let... This indicates that the set of associated nodes was sampled during the (k-1)th sampling iteration. For For each node t in the sampling iteration, during the k-th step, the single-hop neighbors of node t are replaced with Nt. The probability of node t'∈Nt being sampled is related to the edge information ω between nodes t and t'. tt It is directly proportional. It is worth noting that a node may... The term appears multiple times. In the MCN sampling algorithm, when selecting associated nodes from the set of associated nodes sampled in multiple sampling iterations, the following two points can be considered: 1) the sampling frequency of the nodes; 2) the number of sampling steps (or distance) between the target node h and the nodes in the set of associated nodes, that is, how many hops it takes between the nodes in the set of associated nodes and the target node h.

[0178] For each node t∈V\h, in specific implementation, the importance of t to the target node h in the kth sampling step (k≤K) can be calculated using the following formula (1):

[0179]

[0180] in, This indicates that t appears in The number of times in. Let be the importance of node t in the k-th sampling step. That is, node t is considered more relevant to the target node h if t is sampled more frequently and it has a smaller sampling step to the target node h. The importance of node t to the target node h is defined by the following formula (2):

[0181]

[0182] Among them, S t Let t be the importance of node t to target node h. Then, based on the importance of each sampled node t, all nodes in V\h can be sorted in descending order; and the top S (e.g., 2, 5, 6 or more) nodes can be selected as the associated nodes with the target node h.

[0183] Once the associated nodes are sampled, the target node h and its associated nodes C can be concatenated. h It can be represented as a sequence of nodes: I h = [h, h1, h2, ... h s ]. h j Let be the j-th associated node, and 1≤j≤S. Let t∈I be the node with respect to t. h The corresponding content feature M t :

[0184]

[0185]

[0186] α t =softmax[tanh(X)] t W s +b s ]

[0187]

[0188] and Representing node sequence I respectively h The weight matrix and bias term of the i-th mode of node t. and b s ∈R 1×m This represents the weight matrix and bias terms of the attention mechanism. It is an aggregation operation. The feature represents the i-th mode of node t.

[0189] A node in node sequence I h The position of a node in I reflects its importance to the target node h. As discussed above, the multiple associated nodes related to the target node h are ranked based on the importance of each associated node. Therefore, it can be considered that a node in I... h The order within the nodes is important in learning node representations. The following are embedded position IDs (i.e., sequence numbers P). t =P-Embedding[p(t)], (4)

[0190] (Identification) is used to identify nodes in node sequence I. h The sequence information.

[0191] Where p(t) represents the position of node t in node sequence I. h The location ID in the middle. This represents the location feature of node t based on its location ID embedding.

[0192] The goal of training PMGT is to derive the fused features (i.e., the representation of a node) of an input node based on its feature information and that of its associated nodes. Intuitively, the target node and its associated nodes should play different roles during pre-training. To identify these role differences, this embodiment adds the following role-based embeddings to each node t∈I. h .

[0193] R t =R-Embedding[r(t)], (5)

[0194] Where r(t) represents the role corresponding to node t. This represents the role characteristics of node t based on its own role. In specific implementations, the role of the target node can be set to "primary role" or "target", and the role of the associated node can be set to "associated role", "context" or "non-primary role", etc. This embodiment does not limit this.

[0195] The second algorithm involved in this application is a model training algorithm or a model optimization algorithm. For details, please refer to the following:

[0196] Taking PMGT as an example again, the algorithm is expressed as follows:

[0197] Input: Sample graph (e.g., relational graph G), multimodal features of all nodes in graph G.

[0198] Output: The parameter set Θ of PMGT and the pre-trained node representations.

[0199]

[0200]

[0201] like Figure 7 As shown, the above training process specifically trains the encoding modules in the Transformer. The main purpose is to optimize the encoding modules (the parameters in the Encoder of each encoding layer). In PMGT, the encoding module can be called a transformer-based graphical encoder.

[0202] Essentially, it uses a transformer to simulate the interaction between nodes and their associated nodes. Given a node fusion feature H at layer (l-1), l-1 The output of the l-th layer of the converter is defined as follows:

[0203]

[0204] in, Let denote the weight matrix, and FFN(.) be the feedforward network. Here, for convenience, other networks are omitted in equation (7).

[0205] For a target node h, there may be some sampled nodes in Ch whose representation is similar to that of h. It is assumed that all sampled associated nodes are related to the target node. The graphical encoder in this embodiment is expected to capture the diversity of associated nodes, which requires focusing on associated nodes that are related to the target node but not very similar to it. To achieve this goal, this embodiment designs an attention mechanism that promotes diversity and incorporates it into the network framework of the converter.

[0206]

[0207]

[0208]

[0209]

[0210] in, It is the weight matrix. E∈R (S+1)×(S+1) S is a matrix where all its elements are 1. ||S||2∈R (S+1)×1 Let S denote the l2 row norm of S, where I ∈ R. (S+1)×(S+1) Let U represent the identity matrix. The greater the similarity between two different nodes, the smaller their attention weight in U1. The purpose of adding I to the definition of U1 is to include the node's own information. β is a constant (0 ≤ β ≤ 1) balancing the contributions of the two attention weights.

[0211] The output H is obtained from the last layer of the encoder. l After that, you can obtain For simplicity, we will use h to represent the target node h (i.e., the fusion feature of the target node h). Then, H l It will be used in subsequent pre-training work, that is, during the optimization process.

[0212] In the optimization process, this embodiment achieves optimization through a reconstruction task. The PMGT model is pre-trained with the following two objectives: 1) graph structure reconstruction and 2) masked node feature reconstruction. To ensure that the learned node representations can capture the graph structure, this embodiment defines the following loss function:

[0213]

[0214] Where σ(·) is the sigmoid function, and Pn and Q represent the negative sampling distribution and the number of negative samples, respectively.

[0215] The task of masked node feature reconstruction focuses on capturing multimodal features in the learned node representations. This application embodiment designs a masked node feature reconstruction task, aiming to reconstruct multimodal features from node sequence I. h The characteristics of the masked node are reconstructed from other non-masked nodes.

[0216] In this embodiment, no masking operation is applied to the target node h. Instead, list I is randomly selected. h 20% of the nodes in \h are masked. If node t is selected, t is replaced with: (1) 80% of the time it is a [masked] node, (2) 10% of the time it is a random node, and (3) 10% of the time it is an invariant node t. The feature of the [masked] node is set to 0. Then, the node sequence I after the masked nodes is... h The fused features of unmasked nodes and the features of masked nodes (set to 0) are input into the Decoder in the Transformer to output the reconstructed feature information of the masked nodes in the node sequence. For the masked node feature reconstruction task, this application defines a corresponding feature reconstruction loss function, as follows:

[0217]

[0218] Among them, M h Represents node sequence I h The set of masking nodes in the middle, For H L The fusion feature (node ​​representation) of node t in the data. It is the weight matrix for feature reconstruction of the i-th mode; It is the feature of the i-th mode of node t.

[0219] The model parameters of PMGT can be learned by minimizing the combined objective function, that is, by minimizing the combined objective function shown in the following formula (11).

[0220]

[0221] When applying a pre-trained PMGT to downstream tasks (such as data recommendation tasks, data classification (such as video classification, image classification, etc.), the fusion features of the data output by the PMGT can be directly input into the downstream task; or, in order to be suitable for the scenario corresponding to the downstream task, the pre-trained PMGT can be adjusted and trained using the sample data of the scenario.

[0222] Specifically, the reconstruction task in this embodiment is designed to further capture the topological relationships and node feature information of the graph. The reconstruction task may include two parts: one is a graph structure reconstruction task, which aims to ensure that the fused features of adjacent nodes (or node representations) are similar, while the fused features of non-adjacent nodes are far apart, in the fused features of the target nodes obtained by the encoding module. The other is a masked node feature reconstruction task, which aims to ensure that the fused features of the nodes obtained by the encoding module can be used by the decoding module to recover the various modal features of the nodes, thereby allowing the pre-trained short video representation to contain modal information.

[0223] It should be noted that for information on the decoding module (Decoder), please refer to the relevant literature; this article does not limit the scope of the information.

[0224] The pre-trained PMGT in this embodiment can be integrated into a recommendation model for a recommendation scenario. Using user behavior data, the PMGT can be further trained to maximize performance. Alternatively, the pre-trained PMGT can output fused features of the target data, which can then be input into a recommendation model to obtain corresponding recommendation data.

[0225] Traditional methods for fusing multimodal information only extract the modal representations of the target data (such as short videos) and then concatenate and fuse them. Recall based on this representation only retrieves data similar to its own modality. This application, however, fuses not only the multimodal features of the target data but also the multimodal features of at least one related data point. This allows for recall based on the fused features, which not only retrieve data similar to the target data but also data related to it.

[0226] Based on the above, the core of this application can be summarized as follows: using a relationship graph to establish the correlation between data, guiding the data to integrate its own multimodal information, and capturing the topological relationships of the relationship graph to further integrate the features of the relational data, thereby helping with recommendations.

[0227] In practice, when applied to specific recommendation scenarios, multiple relationship graphs can be established between users and short videos, between users, and between short videos to directly capture various correlation information. This allows for multi-dimensional pre-training of feature fusion models (such as PMGT).

[0228] The data recommendation method provided in the above embodiments can be applied to multimedia data recommendation scenarios. For example, it can be used for recommending short videos in social applications, songs in music apps, music videos in music apps, products in e-commerce apps, and product promotion videos in e-commerce apps. Multimedia data can include text, images, audio, video, and other data. That is, such as... Figure 8 As shown, another embodiment of this application provides a data recommendation method, which includes:

[0229] 201. In response to the user's operation on the interactive interface, output the first multimedia data;

[0230] 202. Determine at least one second multimedia data related to the first multimedia data;

[0231] 203. Merge the feature information of the first multimedia data and the feature information of at least one second multimedia data to obtain the fused feature of the first multimedia data;

[0232] 204. Based on the fusion features of the first multimedia data, determine the recommended data;

[0233] 205. When the output conditions are met, output the recommended data to the user.

[0234] The executing entity for each step of the method provided in this embodiment can be, for example, such as... Figure 12 The server 22 in the illustrated embodiment; or the execution entity of some steps in the method can be, for example, the execution entity of such steps. Figure 12 In the server-side component 22, the execution entity for some steps can be... Figure 12 Client 21 in the middle.

[0235] In the above 201, the user's operation on the interactive interface can be a "click" operation or a "swipe" operation on a short video playback control on the interface, etc., and this embodiment does not limit this.

[0236] In one feasible technical solution, step 202 of this embodiment, "determining at least one second multimedia data related to the first multimedia data," may include:

[0237] 2021. Based on historical data related to the user, determine at least one second multimedia data.

[0238] 2022. Obtain the multimodal features of the first multimedia data, the multimodal features of at least one second multimedia data, and the user's behavioral data;

[0239] 2023. Based on the multimodal features of the first multimedia data, the multimodal features of at least one second multimedia data, and the user's behavioral data, determine the association information between the multimedia data;

[0240] 2024. Construct a relationship graph based on the association information between the multimedia data; wherein the relationship graph includes multiple nodes and edge information reflecting the relationship between nodes; the relationship graph includes: a first node corresponding to the first multimedia data, and at least one second node corresponding to the at least one second multimedia data;

[0241] 2025. For the first node, the nodes in the relationship graph are sampled to sample at least one associated node related to the first node, wherein the data corresponding to the associated node is the second multimedia data related to the first multimedia data.

[0242] In a specific, implementable embodiment, step 203, "fusing the feature information of the first multimedia data and the feature information of at least one second multimedia data to obtain the fused feature of the first multimedia data," includes:

[0243] 2031. Configure corresponding embedding features for the first multimedia data and the at least one second multimedia data respectively;

[0244] 2032. Based on the multimodal features of the first multimedia data and the embedding features corresponding to the first multimedia data, determine the feature information of the first multimedia data;

[0245] 2033. Based on the multimodal features of the at least one second multimedia data and the embedding features corresponding to the at least one second multimedia data, determine the feature information of the at least one second multimedia data;

[0246] 2034. Input the feature information of the first multimedia data and the feature information of at least one second multimedia data into the feature fusion model, and execute the feature fusion model to obtain the fused features of the first multimedia data;

[0247] The feature fusion model is obtained through a pre-training process.

[0248] Specifically, the feature fusion model described above can be the encoding module in PMGT mentioned above. Furthermore, steps 2025 and 2031-2034 can be implemented using PMGT. Based on the content related to PMGT mentioned above... Figure 7 As shown in the structure, after the constructed relationship graph and target nodes (i.e., the nodes corresponding to the first multimedia data) are input into PMGT, PMGT can complete the acquisition of associated nodes (i.e., at least one related second multimedia data), the determination of the feature information of the first multimedia data and at least one second multimedia data, the fusion of the feature information of the first multimedia data and at least one second multimedia data, and finally output the fused feature of the first multimedia data.

[0249] In the above 204, since the fusion feature of the first multimedia data not only incorporates information corresponding to its own features, but also incorporates information corresponding to features of at least one second multimedia data, when making recommendations based on the fusion feature, in addition to recalling recommended data similar to the first multimedia data, it can also recall recommended data that is dissimilar to the first multimedia data but related to it. Therefore, the method provided in this embodiment can effectively improve the diversity of data recommendations.

[0250] The aforementioned 205 may include, but is not limited to: the output condition being met when the first multimedia data playback is complete, or the output condition being met when a user triggers a switching operation. For example, if a user selects a short video for full-screen playback, the output condition is met once the short video finishes playing. Alternatively, the user performs a "swipe up" operation on a short video full-screen playback interface (e.g., ...). Figure 2 As shown in the figure, the output condition is satisfied.

[0251] Once the output conditions are met, the determined recommended data can be output on the user's client device, such as displayed on the screen, played, or played through the speaker.

[0252] The technical solutions provided in this application will be applied to short video recommendation scenarios and multimedia data classification scenarios. For example... Figure 9a A flowchart illustrating an embodiment for a short video recommendation scenario is shown. Figure 9a As shown, the short video recommendation method includes:

[0253] 301. Display the first short video;

[0254] 302. Determine at least one second short video related to the first short video;

[0255] 303. Merge the feature information of the first short video and the feature information of at least one second short video to obtain the fused feature of the first short video;

[0256] 304. Determine the recommended objects based on the fusion characteristics of the first short video;

[0257] 305. When the display conditions are met, the recommended objects are displayed.

[0258] For example, Figure 9b An example of the data classification method is shown. For example... Figure 9b As shown, the data classification method includes:

[0259] 301' Determine at least one second multimedia data related to the first multimedia data;

[0260] 302'. Merge the feature information of the first multimedia data and the feature information of at least one second multimedia data to obtain the fused feature of the first multimedia data;

[0261] 303'. Based on the fusion characteristics of the first multimedia data, determine the category to which the first multimedia data belongs.

[0262] Figure 9b The entity performing each step in the illustrated embodiment may be, for example, as follows: Figure 12 The server 22 in the illustrated embodiment; or the execution entity of some steps in the method can be, for example, the execution entity of such steps. Figure 12 In the server-side component 22, the execution entity for some steps can be... Figure 12 In the case of client 21; or, the client is the main body executing all the steps of the method. The client classifies the multimedia data.

[0263] The category to which the multimedia data belongs can be used when searching for data or recommending applications on the homepage. In one feasible solution, in step 303' above, the fusion features of the first multimedia data can be input into a classification model, and executing the classification model will yield the corresponding category. This classification model can be a deep neural network model, a convolutional neural network model, etc., and this embodiment does not limit it.

[0264] For details on the implementation of each step above, please refer to the corresponding content above, which will not be repeated here.

[0265] like Figure 10As shown, another embodiment of this application provides a feature fusion model based on a relationship graph. This feature fusion model based on a relationship graph may include: a sampling module 11, a feature information determination module 12, a feature fusion module 13, and an optimization module 14. The sampling module 11 is used to sample nodes in the relationship graph for a first node to obtain at least one second node associated with the first node. The feature information determination module 12 is used to configure corresponding embedding features for the first node and the at least one second node based on the multimodal information of the first node, the multimodal information of the at least one second node, and the edge information between the first node and the at least one second node; determine the feature information of the first node based on the multimodal information of the first node and the corresponding embedding feature; and determine the feature information of the at least one second node based on the multimodal information of the at least one second node and the corresponding embedding feature. The feature fusion module 13 is used to input the feature information of the first node and the feature information of the at least one second node into the feature fusion model, and execute the feature fusion model to obtain the fused feature of the first node. The optimization module 14 is used to perform graph reconstruction tasks and masked node feature reconstruction tasks based on the fusion features of the first sample nodes, and obtain the execution results corresponding to each task; and optimize the parameters in the feature fusion model based on the execution results corresponding to each task.

[0266] Accordingly, this application also provides an embodiment of a training method for a feature fusion model based on a relationship graph. The execution entity for each step of the method provided in this embodiment can be a server or a client. Because the model training process has high performance requirements for the device, it is more common for the execution entity in this embodiment to be a server. Figure 11 As shown, the method includes:

[0267] 401. Obtain a sample graph, wherein the sample graph includes sample nodes and edge information reflecting the relationships between sample nodes;

[0268] 402. For the first sample node in the sample graph, sample nodes in the sample graph are sampled to sample at least one second sample node related to the first sample node;

[0269] 403. Based on the multimodal features of the first sample node and the multimodal features of at least one second sample node, determine the feature information of the first sample node and the feature information of the at least one second sample node, respectively.

[0270] 404. Input the feature information of the first sample node and the feature information of at least one second sample node into the feature fusion model to obtain the fused feature of the first sample node;

[0271] 405. Based on the fusion features of the first sample node, perform the graph reconstruction task and the masked sample node feature reconstruction task to obtain the execution results corresponding to each task;

[0272] 406. Optimize the parameters in the feature fusion model based on the execution results of each task.

[0273] The solutions provided in this application can be applied to, for example, Figure 12 In the system architecture shown. For example... Figure 12 As shown, this embodiment provides a service system. The service system includes a client 21 and a server 22. The server 22 can be a server, a server cluster consisting of multiple servers, or a virtual server on a server cluster, etc., and this embodiment does not limit this. Specifically,

[0274] The client 21 is used to display or output target data and send a recommendation request to the server based on the target data, wherein the recommendation request may carry a user identifier.

[0275] The server 22 can have three tasks. One task is to construct a relationship graph. For example, it could focus on a single user and construct a relationship graph related to that user, containing nodes corresponding to data (such as videos, music, and text) from the user's historical browsing, collection, likes, rewards, and following activities. Alternatively, it could focus on users belonging to the same category on the platform and construct a relationship graph related to that category of users, containing nodes corresponding to data (such as videos, music, and text) from the same category of users' historical browsing, collection, likes, rewards, and following activities. Or, it could focus on all the data on the platform and construct a relationship graph containing nodes corresponding to all the data on the platform. The two data points connected by edges in the relationship graph can have relationships such as: having the same label, having a sequential user behavior relationship, having the same user behavior (e.g., both data points were rewarded by the user), etc. The second task is model training, such as the PMGT pre-training process mentioned above. The third task is the recommendation task, which is used to receive recommendation requests sent by the client, obtain the corresponding relationship graph based on the user identifier and target data carried in the recommendation request, then call the pre-trained PMGT, input the relationship graph and target data into the PMGT, and output the fusion features of the target data; subsequently, based on the fusion features of the target data, perform the recommendation task to obtain at least one recommendation data, and feed back the at least one recommendation data to the client device.

[0276] Figure 13An embodiment of a data recommendation device provided in this application is illustrated. As shown in the figure, the data recommendation device includes: a determination module 31, a fusion module 32, and a recommendation module 33. The determination module 31 is used to determine target data and, based on the correlation information between data, to determine at least one associated data among multiple data sets related to the target data. The fusion module 32 is used to fuse the feature information of the target data and the feature information of the at least one associated data set to obtain a fused feature of the target data. The recommendation module 33 is used to recommend at least one piece of recommended data to the user based on the fused feature of the target data.

[0277] Furthermore, when determining at least one related data among multiple data sets that is related to the target data based on the correlation information between the data, the determining module 31 is specifically used for:

[0278] A relationship graph is constructed based on the correlation information between the data; wherein, the relationship graph includes multiple nodes and edge information reflecting the relationships between the nodes; different nodes correspond to different data, and the node corresponding to the target data in the relationship graph is the target node;

[0279] For the target node, the nodes in the relationship graph are sampled to sample at least one associated node related to the target node; wherein, the data corresponding to the associated node is the associated data related to the target data.

[0280] Furthermore, when the determining module samples the nodes in the relationship graph for the target node to sample at least one associated node related to the target node, it is specifically used for:

[0281] Obtain the target number of times and the number of samples;

[0282] In a single sampling process, the target node is used as the sampling origin, and at least one neighboring node in the relationship graph that is adjacent to the sampling origin is sampled.

[0283] Determine whether the number of samples has reached the target number;

[0284] When the number of samplings is not greater than the target number, proceed to the next sampling iteration; in the next sampling iteration, take any one of the at least one neighboring nodes as the sampling origin, and sample at least one neighboring node in the relationship graph that is adjacent to the sampling origin;

[0285] When the number of samplings is greater than the target number, the number of neighboring nodes sampled from the sampling iterations of the target number are determined as associated nodes.

[0286] Furthermore, when the determining module samples multiple related data points associated with the target data, it is also used to: determine the importance of the multiple related data points; configure corresponding embedding features for each of the multiple related data points according to their importance; configure embedding features for the target data; obtain features of at least one modality of the target data and multimodal features of any one of the multiple related data points; determine the feature information of the target data based on the features of at least one modality of the target data and the embedding features corresponding to the target data; and determine the feature information of the related data based on the features of at least one modality of any one of the multiple related data points and the embedding features corresponding to that related data.

[0287] Furthermore, the embedded features include location features and role features, and the target data has multimodal features, which include features of multiple modalities. Accordingly, when the determining module 31 determines the feature information of the target data based on the multimodal features and the corresponding embedded features, it is specifically used for:

[0288] Determine the corresponding weights for each modality of the multimodal features in the target data;

[0289] Based on the multimodal features of the target data and the weight of each modality feature in the multimodal features, the content features of the target data are determined;

[0290] The content features of the target data, along with the location and role features corresponding to the target data, are aggregated to obtain the feature information of the target data.

[0291] Furthermore, when the fusion module 32 fuses the feature information of the target data and the feature information of at least one associated data to obtain the fused features of the target data, it is used to:

[0292] Determine the feature similarity between the feature information of the target data and the feature information of at least one associated data;

[0293] Based on feature similarity, determine the attention weight corresponding to the at least one associated data;

[0294] Based on the attention weights corresponding to the at least one associated data, the feature information of the target data and the feature information of the at least one associated data are fused and encoded to obtain the fused feature of the target data.

[0295] Alternatively, when the fusion module 32 fuses the feature information of the target data and the feature information of at least one associated data to obtain the fused features of the target data, it is used to:

[0296] Obtain a feature fusion model;

[0297] The feature information of the target data and the feature information of at least one associated data are input into the feature fusion model, and the feature fusion model is executed to obtain the fused features of the target data.

[0298] The feature fusion model is obtained through a pre-training process.

[0299] Furthermore, the data recommendation device provided in this embodiment may also include a training module. The training model is used for:

[0300] Obtain a sample graph, wherein the sample graph includes sample nodes and edge information reflecting the relationships between sample nodes;

[0301] For the first sample node in the sample graph, sample nodes in the sample graph are sampled to sample at least one second sample node related to the first sample node;

[0302] Based on the features of at least one modality of the first sample node and the features of at least one modality of the at least one second sample node, the feature information of the first sample node and the feature information of the at least one second sample node are determined respectively.

[0303] The feature information of the first sample node and the feature information of at least one second sample node are input into the feature fusion model to obtain the fused features of the first sample node and the fused features of the at least one second sample node.

[0304] Based on the fusion features of the first sample node and the fusion features of at least one second sample node, a graph reconstruction task and a masked sample node feature reconstruction task are performed to obtain the execution results corresponding to each task.

[0305] Based on the execution results of each task, the parameters in the feature fusion model are optimized.

[0306] It should be noted that any aspects of the data recommendation device provided in this embodiment that are not fully described in detail can be found in the corresponding contents of the above embodiments, and will not be repeated here. Furthermore, in addition to the functions described above, the data recommendation device provided in this embodiment may also include functions corresponding to other parts or all of the steps in the above embodiments; for details, please refer to the corresponding contents of the above embodiments, and will not be repeated here.

[0307] Figure 14 A structural diagram of a data recommendation device provided in another embodiment of this application is shown. Figure 14As shown, the data recommendation device includes: an output module 41, a determination module 42, a fusion module 43, and a recommendation module 44. The output module 41 is used to output first multimedia data in response to user operations on the interactive interface. The determination module 42 is used to determine at least one second multimedia data related to the first multimedia data. The fusion module 43 is used to fuse the feature information of the first multimedia data and the feature information of the at least one second multimedia data to obtain a fused feature of the first multimedia data. The recommendation module 44 is used to determine recommended data based on the fused feature of the first multimedia data. The output module is also used to output the recommended data to the user when the output conditions are met.

[0308] Furthermore, when determining at least one second multimedia data related to the first multimedia data, the determining module 42 is specifically used for:

[0309] Based on historical data related to the user, at least one second multimedia data is determined; the multimodal features of the first multimedia data, the multimodal features of the at least one second multimedia data, and the user's behavioral data are obtained; based on the multimodal features of the first multimedia data, the multimodal features of the at least one second multimedia data, and the user's behavioral data, the association information between the multimedia data is determined; based on the association information between the multimedia data, a relationship graph is constructed; wherein the relationship graph includes multiple nodes and edge information reflecting the relationships between nodes; the relationship graph includes: a first node corresponding to the first multimedia data, and at least one second node corresponding to the at least one second multimedia data; for the first node, the nodes in the relationship graph are sampled to sample at least one associated node related to the first node, wherein the data corresponding to the associated node is the second multimedia data related to the first multimedia data.

[0310] Furthermore, when the fusion module 43 fuses the feature information of the first multimedia data and the feature information of at least one second multimedia data to obtain the fused feature of the first multimedia data, it is specifically used for:

[0311] The following steps are performed: Configure corresponding embedding features for the first multimedia data and the at least one second multimedia data; determine the feature information of the first multimedia data based on its multimodal features and the corresponding embedding features; determine the feature information of the at least one second multimedia data based on its multimodal features and the corresponding embedding features; input the feature information of the first multimedia data and the at least one second multimedia data into a feature fusion model, and execute the feature fusion model to obtain the fused features of the first multimedia data; wherein the feature fusion model is obtained through a pre-training process.

[0312] It should be noted that any aspects of the data recommendation device provided in this embodiment that are not fully described in detail can be found in the corresponding contents of the above embodiments, and will not be repeated here. Furthermore, in addition to the functions described above, the data recommendation device provided in this embodiment may also include functions corresponding to other parts or all of the steps in the above embodiments; for details, please refer to the corresponding contents of the above embodiments, and will not be repeated here.

[0313] Figure 15 A schematic diagram of the structure of a short video recommendation device according to an embodiment of this application is shown. Figure 15 As shown, the device includes: a display module 51, a determination module 52, a fusion module 53, and a recommendation module 54. The display module 51 displays a first short video. The determination module 52 determines at least one second short video related to the first short video. The fusion module 53 fuses the feature information of the first short video and the feature information of the at least one second short video to obtain a fused feature of the first short video. The recommendation module 54 determines recommended objects based on the fused feature of the first short video. The display module is also used to display the recommended objects when display conditions are met.

[0314] It should be noted that any aspects of the short video recommendation device provided in this embodiment that are not fully described in detail can be found in the corresponding contents of the above embodiments, and will not be repeated here. Furthermore, in addition to the functions described above, the short video recommendation device provided in this embodiment may also include functions corresponding to other parts or all of the steps in the above embodiments; for details, please refer to the corresponding contents of the above embodiments, and will not be repeated here.

[0315] Figure 16 A schematic diagram of the structure of an electronic device provided according to an embodiment of this application is shown. Figure 16As shown, the electronic device includes a memory 61 and a processor 62. The memory 61 can be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device. The memory 61 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0316] The memory 61 is used to store one or more computer instructions;

[0317] The processor 62 is coupled to the memory 61 and is used to execute one or more computer instructions stored in the memory 61 to implement the steps in the data recommendation method or short video recommendation method provided in the above embodiments.

[0318] Furthermore, such as Figure 16 As shown, the electronic device also includes other components such as a communication component 63, a power supply component 65, and a display 66. Figure 16 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 16 The components shown.

[0319] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, can implement the steps or functions in the data recommendation method or short video recommendation method provided in the above embodiments.

[0320] This application also provides a computer program product, including a computer program that, when executed by a processor, enables the processor to implement the steps or functions in the data recommendation method or short video recommendation method provided in the above embodiments.

[0321] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0322] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0323] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data recommendation method, characterized in that, include: Identify target data; the target data includes at least one of the following: text data, image data, video data, and audio data; Based on the correlation information between the data, at least one related data among multiple data is determined that is related to the target data; The correlation information between the data includes: the existence of the same label between two data items and the number of times the same label is used; Configure corresponding embedding features for the target data and the at least one associated data respectively; The feature information of the target data is determined based on the multimodal features of the target data and the embedding features corresponding to the target data; Based on the multimodal features of the at least one associated data and the embedding features corresponding to the at least one associated data, the feature information of the at least one associated data is determined; the multimodal features of the associated data or the target data include at least one of text features, audio features, and image features; The feature information of the target data and the feature information of at least one associated data are input into the feature fusion model, and the feature fusion model is executed to obtain the fused features of the target data; wherein, the feature fusion model is obtained through a pre-training process; Based on the fusion features of the target data, at least one piece of recommended data is recommended to the user.

2. The data recommendation method according to claim 1, characterized in that, Based on the correlation information between data, at least one related data is identified from among multiple data sets that is related to the target data, including: A relationship graph is constructed based on the correlation information between the data; wherein, the relationship graph includes multiple nodes and edge information reflecting the relationships between the nodes; different nodes correspond to different data, and the node corresponding to the target data in the relationship graph is the target node; For the target node, the nodes in the relationship graph are sampled to sample at least one associated node related to the target node; wherein, the data corresponding to the associated node is the associated data related to the target data.

3. The data recommendation method according to claim 2, characterized in that, For the target node, sampling is performed on the nodes in the relationship graph to sample at least one associated node related to the target node, including: Obtain the target number of times and the number of samples; In a single sampling process, the target node is used as the sampling origin, and at least one neighboring node in the relationship graph that is adjacent to the sampling origin is sampled. Determine whether the number of samples has reached the target number; When the number of samplings is not greater than the target number, proceed to the next sampling iteration; in the next sampling iteration, take any one of the at least one neighboring nodes as the sampling origin, and sample at least one neighboring node in the relationship graph that is adjacent to the sampling origin; When the number of samplings is greater than the target number, the number of neighboring nodes sampled from the sampling iterations of the target number are determined as associated nodes.

4. The data recommendation method according to claim 3, characterized in that, When sampling multiple related data points associated with the target data, the method further includes: Determine the importance of multiple related data; Based on their importance, configure corresponding embedding features for each of the multiple related data; Configure embedding features for the target data; Obtain features of at least one modality of the target data and multimodal features of any one of the multiple associated data; The feature information of the target data is determined based on the features of at least one modality of the target data and the embedding features corresponding to the target data; The feature information of the associated data is determined based on the features of at least one modality of any of the multiple associated data and the embedding features corresponding to that associated data.

5. The data recommendation method according to claim 4, characterized in that, The embedded features include location features and role features, and the target data has multimodal features, which include features of multiple modalities. as well as Based on the multimodal features of the target data and the corresponding embedding features, the feature information of the target data is determined, including: Determine the corresponding weights for each modality of the multimodal features in the target data; Based on the multimodal features of the target data and the weight of each modality feature in the multimodal features, the content features of the target data are determined; The content features of the target data, along with the location and role features corresponding to the target data, are aggregated to obtain the feature information of the target data.

6. The data recommendation method according to any one of claims 1 to 5, characterized in that, By fusing the feature information of the target data and the feature information of at least one associated data, a fused feature of the target data is obtained, including: Determine the feature similarity between the feature information of the target data and the feature information of at least one associated data; Based on feature similarity, determine the attention weight corresponding to the at least one associated data; Based on the attention weights corresponding to the at least one associated data, the feature information of the target data and the feature information of the at least one associated data are fused and encoded to obtain the fused feature of the target data.

7. The data recommendation method according to any one of claims 1 to 5, characterized in that, Also includes: Obtain a sample graph, wherein the sample graph includes sample nodes and edge information reflecting the relationships between sample nodes; For the first sample node in the sample graph, sample nodes in the sample graph are sampled to sample at least one second sample node related to the first sample node; Based on the features of at least one modality of the first sample node and the features of at least one modality of the at least one second sample node, the feature information of the first sample node and the feature information of the at least one second sample node are determined respectively. The feature information of the first sample node and the feature information of at least one second sample node are input into the feature fusion model to obtain the fused features of the first sample node and the fused features of the at least one second sample node. Based on the fusion features of the first sample node and the fusion features of at least one second sample node, a graph reconstruction task and a masked sample node feature reconstruction task are performed to obtain the execution results corresponding to each task. Based on the execution results of each task, the parameters in the feature fusion model are optimized.

8. A data recommendation method, characterized in that, include: In response to user actions on the interactive interface, output the first multimedia data; Determine at least one second multimedia data related to the first multimedia data; Configure corresponding embedding features for the first multimedia data and the at least one second multimedia data respectively; Based on the multimodal features of the first multimedia data and the embedding features corresponding to the first multimedia data, the feature information of the first multimedia data is determined; Based on the multimodal features of the at least one second multimedia data and the embedding features corresponding to the at least one second multimedia data, the feature information of the at least one second multimedia data is determined; The feature information of the first multimedia data and the feature information of at least one second multimedia data are input into the feature fusion model, and the feature fusion model is executed to obtain the fused features of the first multimedia data; wherein, the feature fusion model is obtained through a pre-training process; Based on the fusion characteristics of the first multimedia data, recommended data is determined; When the output conditions are met, the recommended data is output to the user.

9. The method according to claim 8, characterized in that, Determining at least one second multimedia data associated with the first multimedia data includes: Based on historical data associated with the user, at least one second multimedia data is determined; Acquire the multimodal features of the first multimedia data, the multimodal features of at least one second multimedia data, and the user's behavioral data; Based on the multimodal features of the first multimedia data, the multimodal features of at least one second multimedia data, and the user's behavioral data, the association information between the multimedia data is determined; A relationship graph is constructed based on the association information between the multimedia data; wherein, the relationship graph includes multiple nodes and edge information reflecting the relationship between the nodes; the relationship graph includes: a first node corresponding to the first multimedia data, and at least one second node corresponding to the at least one second multimedia data; For the first node, the nodes in the relationship graph are sampled to sample at least one associated node related to the first node, wherein the data corresponding to the associated node is the second multimedia data related to the first multimedia data.

10. A data classification method, characterized in that, include: Determine at least one second multimedia data associated with the first multimedia data; Configure corresponding embedding features for the first multimedia data and the at least one second multimedia data respectively; Based on the multimodal features of the first multimedia data and the embedding features corresponding to the first multimedia data, the feature information of the first multimedia data is determined; Based on the multimodal features of the at least one second multimedia data and the embedding features corresponding to the at least one second multimedia data, the feature information of the at least one second multimedia data is determined; The feature information of the first multimedia data and the feature information of at least one second multimedia data are input into the feature fusion model, and the feature fusion model is executed to obtain the fused features of the first multimedia data; wherein, the feature fusion model is obtained through a pre-training process; Based on the fusion characteristics of the first multimedia data, the category to which the first multimedia data belongs is determined.

11. A feature fusion system based on a relational graph, characterized in that, include: A node sampling layer is used to sample nodes in a relation graph for a first node in the relation graph to obtain at least one second node associated with the first node. The first node corresponds to the first multimedia data, and the second node corresponds to the second multimedia data; A node feature information determination layer is used to configure corresponding embedding features for the first node and the at least one second node based on the multimodal information of the first node, the multimodal information of the at least one second node, and the edge information between the first node and the at least one second node; and to determine the feature information of the first node based on the multimodal information of the first node and the embedding features corresponding to the first node. Based on the multimodal information of the at least one second node and the embedding features corresponding to the at least one second node, the feature information of the at least one second node is determined; wherein, the multimodal information of the first node or the second node includes at least one of text features, audio features, and image features; A feature fusion layer is used to fuse the feature information of the first node and the feature information of at least one second node to obtain the fused feature of the first node. The optimization layer is used to perform graph reconstruction tasks and masked node feature reconstruction tasks based on the fusion features of the first node, and obtain the execution results corresponding to each task; based on the execution results corresponding to each task, the parameters in the feature fusion model are optimized.

12. An electronic device, characterized in that, The method includes a memory and a processor; wherein the memory is used to store a program; the processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the data recommendation method of any one of claims 1 to 7; or to implement the steps in the data recommendation method of any one of claims 8 to 9; or to implement the steps in the data classification method of claim 10.

Images