Embedding method based on Prompt to enhance Pins

Abstract

This invention discloses an embedding method for Pins based on Prompt enhancement, comprising: constructing a User-Item interaction bipartite graph based on a GCN model; performing embedding learning on nodes in the graph by combining random walk and graph convolution algorithms; generating embedding information for each node by sampling neighbor nodes based on importance and combining edge weight information in the Pinsage network; and enhancing the representation of materials through Prompt; using the PageRank method during the iteration process, adding hard negative samples to enable the model to learn to distinguish samples with different similarity levels; and training the model using a loss function based on the maximum boundary, outputting accurate material embeddings and user interest representations. This invention can more accurately characterize materials and better capture user interests based on user behavior trajectories.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an embedding method based on Prompt-enhanced Pinsage. Background Technology

[0002] Material embedding is a core technology for achieving accurate matching and personalized recommendations. Its core goal is to map discrete material information into low-dimensional dense vector representations, accurately capture the semantic features of materials and the relationships between materials, and combine user behavior data to characterize user interests and preferences, providing support for subsequent matching and recommendation tasks.

[0003] Currently, existing technologies for acquiring material embeddings mostly rely on sequences formed by User-Item (PUI) interaction bipartite graphs, learning them using the sequence modeling approach of Natural Language Processing (NLP). This sequence modeling method primarily focuses on the positional information of the material within the interaction sequence, neglecting the semantic information carried by the material itself. This results in insufficient semantic representation capabilities of the material, making it difficult to accurately depict its core features, and consequently affecting the accuracy of subsequent user interest matching.

[0004] In terms of network structure selection, existing technologies mostly employ Graph Convolutional Networks (GCNs) to learn material embeddings. This involves sampling the neighborhood around a node and dynamically constructing a computational graph from this sampled neighborhood to achieve local convolution. Multiple layers are then stacked to allow nodes to access information from higher-order neighbors, capturing the relationships between nodes. However, GCNs have significant drawbacks in recommendation systems: Firstly, while neural network methods, from shallow models to neural collaborative filtering and Deep Factorization Machines (DeepFM), can gradually model complex user behaviors and large amounts of data input, they struggle to effectively learn high-order structural information from the data. Secondly, GCNs use a transductive learning method, requiring all nodes to participate in training to obtain the corresponding node embeddings. In recommendation systems, new users and new materials (new nodes) are constantly generated, severely limiting the application of GCN models and making it impossible to quickly generate effective embedding representations for new nodes.

[0005] Negative sampling is a key technique for optimizing model performance during training. A common approach is to sample from exposed but unclicked samples. However, the exposed data itself is a result retrieved through some method, meaning users have some degree of preference for that sample; they might simply prefer a particular clicked item among a series of exposed materials. If negative sampling is performed using this approach, the constructed sample set will inherently be biased.

[0006] GCN has demonstrated excellent performance across multiple technical fields, particularly excelling in node representation learning. Pinsage, a graph neural network based on an improvement upon GraphSAGE, combines historical user interaction data with side information from interacting nodes. Furthermore, by leveraging neighborhood sampling and aggregation, it fully utilizes the structural advantages of integrating higher-order information, making it suitable for applications such as recommendation systems where material and user interactions are frequent and data volumes are massive.

[0007] Therefore, how to optimize the Pinsage neural network structure, combine more efficient semantic enhancement methods and reasonable negative sampling strategies to more accurately characterize materials has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0008] To address the problems of insufficient semantic representation of material embeddings, limitations in network structure, and bias in negative sampling, this invention proposes an embedding method based on Prompt-enhanced Pinsage.

[0009] This invention adopts the following technical solution: an embedding method based on Prompt to enhance Pinsage, comprising:

[0010] Constructing a GCN model: Iteratively aggregate neighborhood information and integrate the aggregated information with the representation of the current node, capturing dependencies on the graph through message propagation between nodes;

[0011] Construct a Pinsage network structure based on Prompt enhancement: Based on the GCN model, construct a User-Item interaction bipartite graph, with users and materials as nodes in the graph and the interaction relationship between users and materials as edges; combine random walk and graph convolution algorithms to perform embedding learning on the nodes in the User-Item interaction bipartite graph, and the Pinsage network generates the embedding information of each node by sampling neighbor nodes based on importance and combining the edge weight information, and introduces Prompt to enhance the representation of materials;

[0012] Hard negative sampling: A hierarchical negative sampling strategy is used to process training samples. A portion of samples are randomly sampled outside the range of nodes contained in each minibatch as common negative samples. The PageRank method is used during the iteration process. By adding hard negative samples, the model learns to distinguish samples with different similarities.

[0013] Loss function: The model is trained using a loss function based on the maximum boundary, which maximizes the inner product of positive samples while satisfying the constraints. After the model training is completed, it outputs accurate material embeddings and user interest representations.

[0014] As a preferred option, the GCN model is divided into spectral methods and non-spectral methods based on whether or not a spectral convolution operator is used;

[0015] The spectral method defines a graph convolution operator in the Fourier domain and represents the relationships between nodes through the original graph structure; the non-spectral method aggregates messages from neighbors through an aggregator and merges neighbor nodes and the center node through an updater.

[0016] As a preferred embodiment, in the User-Item interaction bipartite graph, the node information includes node ID information, user nodes introduce features that are important in the ranking model as auxiliary information, and material nodes improve their semantic representation capabilities by using the Prompt enhancement method.

[0017] As a preferred embodiment, the method is applied to a recommendation system in the financial field, where the material nodes include video content materials and individual stock materials.

[0018] As a preferred embodiment, the Prompt enhancement method for the material node is as follows: construct a Prompt input based on the material information according to a preset format, and send it into the fine-tuned BGE model to obtain the title representation of the material, which serves as the semantic enhancement information for the material node.

[0019] As a preferred embodiment, the preset format includes:

[0020] For video content materials, the Prompt input format is: Title is: {Title}__Keywords are: {Word1}&{Word2}&{Word3}&...&{Wordn}; where {Title} is the video title, and {Word1} to {Wordn} are the keywords corresponding to the video;

[0021] For individual stock materials, the Prompt input format is: Stock Name: {Secuname}__Industry: {Industry}, where {Secuname} is the stock name and {Industry} is the industry to which the stock belongs.

[0022] As a preferred embodiment, the Hard negative sampling is specifically implemented as follows:

[0023] Extract samples that are ranked low during model training and use them as Hard negative samples.

[0024] The first round of training uses only simple negative sampling, and Hard negative samples are gradually added in subsequent training.

[0025] During the nth round of training, n-1 Hard negative samples are added to the negative sample set of each sample, enabling the model to learn to distinguish samples with different levels of similarity.

[0026] As a preferred embodiment, the constraint condition is: the inner product of negative samples is smaller than the inner product of positive samples, and the difference exceeds a preset boundary.

[0027] As a preferred option, the material embedding is applied to similar recommendation and / or "you might like" scenarios in recommendation systems in the financial field, as well as to direct recommendation scenarios from users to materials.

[0028] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0029] 1. This invention proposes a highly adaptable and scalable Pinsage mapping method that can quickly adapt to the embedding scenarios of new materials, and can improve the basic material representation capability in the recommendation field.

[0030] 2. When referencing material title embeddings, the method of this invention introduces a Prommp-based approach to optimize semantic expression; at the same time, the use of minbatch during model training can maximize the utilization of GPU memory.

[0031] 3. The method of this invention optimizes the Pinsage neural network structure based on Prompt enhancement to learn the material embedding, which can more accurately characterize the material and better characterize the user's interests based on the user's behavior trajectory. Attached Figure Description

[0032] Figure 1 This is a flowchart of the Embedding method for enhancing Pinsage based on Prompt in this invention;

[0033] Figure 2 This is a schematic diagram of the Pinsage network structure based on Prompt enhancement according to the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the application will be further described in detail below with reference to the accompanying drawings. The described embodiments are only a part of the embodiments involved in this invention. All non-innovative embodiments based on these embodiments by other researchers in the art are within the protection scope of this invention. Furthermore, the step numbers in the embodiments of this invention are only set for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0035] In one embodiment of the present invention, an embedding method based on Prompt-enhanced Pinsage is provided. The Pinsage neural network structure is optimized by Prompt enhancement to learn the material embedding, thereby enabling more accurate characterization of the material. Furthermore, based on the user's behavioral trajectory, the user's interests can be better characterized.

[0036] The Embedding method flow in this embodiment is as follows: Figure 1 As shown, it includes the following steps:

[0037] 1. Constructing the GCN model

[0038] The GCN (Graph Convolutional Network) model is the design basis of the Pinsage neural network structure in this embodiment. It can capture the dependencies on the graph through message passing between nodes, and achieve effective learning of node features by iteratively aggregating neighborhood information and integrating the aggregated information with the representation of the current node.

[0039] GCN models are divided into spectral methods and non-spectral methods based on whether or not they use spectral convolution operators:

[0040] 1) Spectral method: Define graph convolution operators in the Fourier domain (spectral domain). The original graph structure is required to represent the relationships between nodes. Graph convolution calculation is achieved through spectral domain transformation.

[0041] 2) Non-spectral method: It does not rely on the spectral domain features of the original graph structure. It requires the design of an aggregator and an updater. The aggregator is used to aggregate messages from neighbors, and the updater is used to merge neighbor nodes and the central node to complete the update of the node representation.

[0042] 2. Construct a Pinsage network structure based on Prompt enhancement.

[0043] To address the limitation of GCN's fixed graph structure and the difficulty in representing new materials, this embodiment selects the Pinsage neural network as the core network structure. Its core is to add representations for newly added nodes based on the above GCN model. It can generate effective embedding representations for new nodes without all nodes participating in training. At the same time, it combines Prompt enhancement technology to improve the semantic representation capability of materials.

[0044] The specific method is as follows:

[0045] 1) Graph structure construction:

[0046] Using users and items in a recommendation system in the financial field as nodes and the interaction relationship between users and items as edges, a User-Item interaction bipartite graph is constructed as the input graph structure for the Pinsage neural network;

[0047] 2) Node information supplementation: Combining random walk and graph convolution learning node embedding, in addition to basic ID information, User nodes also introduce features that are more important in the ranking model as auxiliary information (Side info) to enrich the feature dimensions of user nodes; Item nodes adopt the Prompt enhancement method to supplement the semantic information of the material itself and improve the semantic representation capability of Embedding.

[0048] 3) Prompt Enhancement: For embedding materials in recommendation systems related to the financial field, a Prompt is introduced to enhance the representation of the materials. In particular, for content-based materials such as videos and individual stock-based materials, the input BGE formats are as follows:

[0049] The title is: {Title}__Keywords are: {Word1} & {Word2} & {Word3} & {Wordn}

[0050] The stock name is: {Secuname}__ and its industry is: {Industry}.

[0051] Then, it is fed into the fine-tuned BGE model to obtain the title characterization of the material.

[0052] 4) Node Embedding Learning: The Pinsage neural network uses importance sampling of neighbor nodes and combines edge weight information (reflecting the interaction strength between users and items) to perform graph convolution calculation and random walk learning on the nodes in the User-Item interaction bipartite graph, aggregating high-order neighborhood information to produce high-quality embedding information for each node in the network structure. This achieves accurate representation of items and can also characterize user interests and preferences by combining user behavior trajectories.

[0053] 5.) Application scenario adaptation: The embedding generated by the network structure in this embodiment can be directly applied to similar recommendation and "you may like" scenarios in the financial field recommendation system, and can also be used for direct recommendation scenarios from user to material, adapting to the personalized recommendation needs of the financial field.

[0054] like Figure 2As shown in the figure, each node in the graph represents a material in the recommended scenario, and A is one of the materials. By connecting the relationships between the nodes in the graph based on the graph network, a graph network is formed. In the graph network, the vector embedding representation of node A can be obtained based on the aggregation method mentioned above.

[0055] 3. Negative sampling

[0056] To address the bias issue of existing negative sampling, this invention designs a hierarchical negative sampling strategy. During the iteration process, it leverages the PageRank approach to balance model training efficiency and performance. The specific implementation is as follows:

[0057] 1) Common negative sample sampling: Outside the range of nodes contained in each minibatch, a portion of samples are randomly sampled as common negative samples to reduce the bias of the sample set, while controlling the size of the sample set and reducing the amount of computation.

[0058] 2) Layered negative sampling: During model training, a layered strategy is used to gradually add Hard negative samples, specifically:

[0059] The first round of training uses only simple negative sampling to help the model parameters converge quickly to a range with lower loss values.

[0060] Hard negative samples are gradually added in subsequent training to teach the model to distinguish between very similar samples and slightly similar samples. The method is as follows: starting from the second round of training, in the nth round of training, n-1 hard negative samples are added to the negative sample set of each sample.

[0061] 3) Definition of Hard Negative Samples: Samples ranked lower in the model training process are extracted as Hard negative samples. These samples have a high semantic similarity to positive samples. By introducing Hard negative samples, the model can learn to distinguish between similar samples and slightly similar samples, thereby improving the model's discrimination and generalization abilities.

[0062] 4. Loss Function

[0063] This invention uses a loss function based on the maximum boundary, which maximizes the inner product of positive samples while ensuring that the inner product of negative samples is smaller than the inner product of positive samples, and that the difference exceeds a predefined boundary.

[0064] As a preferred approach, the loss function for the node embedding representation is defined as follows:

[0065] ;

[0066] in: To query the embedding representation of node 𝑞; An embedding representation of a positive sample node 𝑖 that has a positive interaction relationship with the query node 𝑞; The nth negative sample node obtained by sampling according to the negative sampling distribution. Embedding indicates; This indicates that under the negative sampling distribution (q), for negative sample nodes... The Embedding takes the expectation; 𝑞 represents the query node in the current training sample, where the query node is a user node or a target center node; This represents the negative sample sampling distribution constructed for the query node 𝑞; Δ represents the preset interval boundary parameter, used to constrain the minimum difference between positive sample similarity and negative sample similarity. This represents the loss function value during the graph embedding learning phase, used to measure the degree to which the ranking constraints between the query node and the positive and negative sample nodes are satisfied.

[0067] Furthermore, the effects of the present invention are evaluated using the following indicators.

[0068] (1) Hit rate:

[0069] ;

[0070] Where: HitRate@N represents the hit rate on the first n recommendation results, which is used to measure whether the model can recall the target material to the first n recommendation results; n represents the truncation length of the recommendation result list, that is, only the first n candidate materials are counted. This represents a test dataset, which consists of test sample pairs composed of users and target materials. composition; This represents a test sample in the test dataset, where 𝑢 represents the user to be evaluated; 𝑖 represents the real target material of the user in the test sample, and the real target material is the material that the user actually interacts with.

[0071] I(target item occurs in top N) represents the indicator function, which takes the value of 1 when the target item 𝑖 appears in the top N recommended results generated for user 𝑢; otherwise, it takes the value of 0. This represents the number of samples that were successfully matched out of all test samples. This represents the total number of samples in the test dataset.

[0072] (2) Top-K recall rate:

[0073] ;

[0074] in: This indicates the truncation length of the recommended results list, meaning only the results before ranking are counted. One candidate material; This represents the recommended material at the nth position in the recommendation list; 𝑇 represents the set of real relevant materials corresponding to this user, that is, the set of target materials that the user actually interacted with during the testing phase; This indicates the indicator function, when the nth recommended material... If it belongs to the real relevant material set 𝑇, the value is 1; otherwise, the value is 0. This represents the number of truly relevant materials hit in the first K recommendation results; N represents the total number of relevant materials contained in the set of truly relevant materials, usually N=|T|; the value of Recall@K ranges from 0 to 1, and the larger the value, the more truly relevant materials the model can recall in the first K recommendation results.

[0075] (3) Difference rate:

[0076] The difference rate measures the difference between a particular channel and other channels. This is especially important when adding a new recall channel, which typically needs to bring in some new items that weren't previously recalled, thereby improving overall coverage. The difference rate is expressed as the ratio of the number of new items to the total number of recalls for that channel.

[0077] (4) Coverage:

[0078] Measure the recall link's ability to discover long-tail items.

[0079] To verify the role of recall in the overall recommendation chain, this implementation conducted relevant evaluations on the test dataset. Detailed metrics and performance evaluation results are shown in Table 1 below.

[0080] Table 1:

[0081]

[0082] In summary, the purpose of this invention, which enhances the embedding method of Pinsage based on Prompt, is to provide a basic capability that can accurately characterize materials, thereby enabling more accurate and convenient use of relevant i2i or u2i capabilities to supplement the recall capability of the system in the field of personalized recommendation.

[0083] First, a GCN-based neural network structure is designed to comprehensively utilize the higher-order neighborhoods of nodes and the side information of the nodes themselves to derive the material embedding information. Particularly in the data construction stage, on the one hand, the model can easily learn the expressive power of new nodes by adopting and learning the local neighbor features of aggregated nodes; on the other hand, model training mainly adopts a minibatch approach to ensure maximum GPU utilization during training.

[0084] Then, for a new node, the Pinsage algorithm achieves efficient local convolution by sampling its surrounding neighbors and dynamically constructing a computation graph from those sampled neighbors. These dynamically constructed computation graphs specify how to perform local convolutions around the node and reduce the need to operate on the entire graph during training.

[0085] Finally, the model training adopts a minibatch approach, which ensures that the model makes the most of the GPU during training. A large memory, CPU-bound producer effectively samples the neighborhood of the node network and obtains the features required to define local convolutions, while a GPU-bound TensorFlow model consumes these predefined computation graphs, thereby achieving efficient SGD computation.

[0086] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An embedding method based on Prompt to enhance Pinsage, characterized in that, include: Constructing a GCN model: Iteratively aggregate neighborhood information and integrate the aggregated information with the representation of the current node, capturing dependencies on the graph through message propagation between nodes; Construct a Pinsage network structure based on Prompt enhancement: Based on the GCN model, construct a User-Item interaction bipartite graph, with users and materials as nodes in the graph and the interaction relationship between users and materials as edges; combine random walk and graph convolution algorithms to perform embedding learning on the nodes in the User-Item interaction bipartite graph, and the Pinsage network generates the embedding information of each node by sampling neighbor nodes based on importance and combining the edge weight information, and introduces Prompt to enhance the representation of materials; Hard negative sampling: A hierarchical negative sampling strategy is used to process training samples. A portion of samples are randomly sampled outside the range of nodes contained in each minibatch as common negative samples. The PageRank method is used during the iteration process. By adding hard negative samples, the model learns to distinguish samples with different similarities. Loss function: The model is trained using a loss function based on the maximum boundary, which maximizes the inner product of positive samples while satisfying the constraints. After the model training is completed, it outputs accurate material embeddings and user interest representations.

2. The Embedding method according to claim 1, characterized in that, Based on whether or not the spectral convolution operator is used, the GCN model is divided into spectral methods and non-spectral methods; The spectral method defines a graph convolution operator in the Fourier domain and represents the relationships between nodes through the original graph structure; the non-spectral method aggregates messages from neighbors through an aggregator and merges neighbor nodes and the center node through an updater.

3. The Embedding method according to claim 1, characterized in that, In the User-Item interaction bipartite diagram, node information includes node ID information. User nodes introduce features that are important in the ranking model as auxiliary information, and material nodes improve their semantic representation capabilities by using the Prompt enhancement method.

4. The Embedding method according to claim 3, characterized in that, The method is applied to a recommendation system in the financial field, and the material nodes include video content materials and individual stock materials.

5. The Embedding method according to claim 4, characterized in that, The method for enhancing the Prompt of the material node is as follows: construct the Prompt input according to the preset format of the material information, and send it into the fine-tuned BGE model to obtain the title representation of the material, which serves as the semantic enhancement information of the material node.

6. The Embedding method according to claim 5, characterized in that, The preset format includes: For video content materials, the Prompt input format is: Title is: {Title}__Keywords are: {Word1}&{Word2}&{Word3}&...&{Wordn}; where {Title} is the video title, and {Word1} to {Wordn} are the keywords corresponding to the video; For individual stock materials, the Prompt input format is: Stock Name: {Secuname}__Industry: {Industry}, where {Secuname} is the stock name and {Industry} is the industry to which the stock belongs.

7. The Embedding method according to claim 1, characterized in that, The Hard negative sampling method is as follows: Extract samples that are ranked low during model training and use them as Hard negative samples. The first round of training uses only simple negative sampling, and Hard negative samples are gradually added in subsequent training. During the nth round of training, n-1 Hard negative samples are added to the negative sample set of each sample, enabling the model to learn to distinguish samples with different levels of similarity.

8. The Embedding method according to claim 1, characterized in that, The loss function for the node embedding representation is defined as follows: ； in, To query the embedding representation of node 𝑞; An embedding representation of a positive sample node 𝑖 that has a positive interaction relationship with the query node 𝑞; The nth negative sample node obtained by sampling according to the negative sampling distribution. Embedding indicates; This indicates that under the negative sampling distribution (q), for negative sample nodes... The Embedding takes the expectation; 𝑞 represents the query node in the current training sample, where the query node is a user node or a target center node; This represents the negative sample sampling distribution constructed for the query node 𝑞; Δ represents the preset interval boundary parameter, used to constrain the minimum difference between positive sample similarity and negative sample similarity. This represents the loss function value during the graph embedding learning phase, used to measure the degree to which the ranking constraints between the query node and the positive and negative sample nodes are satisfied.

9. The Embedding method according to claim 8, characterized in that, The constraint condition is that the inner product of negative samples is smaller than the inner product of positive samples, and the difference exceeds a preset boundary.

10. The embedding method based on Prompt-enhanced Pinsage according to claim 1, characterized in that, The material embedding is applied to similar recommendation and / or "you might like" scenarios in recommendation systems in the financial field, as well as direct recommendation scenarios from users to materials.

Images