Method and apparatus for generating a recommendation model for sequence recommendation

Abstract

The application provides a generation method of a recommendation model for sequence recommendation and a device thereof. The generation method comprises the following steps: acquiring a user set U, an item set V', an interaction sequence of a user and an item, and a recommendation item corresponding to each interaction sequence; generating a recommendation model, wherein the recommendation model comprises a first preprocessing module, a second preprocessing module, a Transformer neural network, a hypergraph neural network and a collaborative network; and training the recommendation model. Thus, the recommendation model is generated.

Description

Technical Field

[0001] This invention relates to the field of sequence recommendation technology, and in particular to a method and apparatus for generating a sequence recommendation model. Background Technology

[0002] Due to the rapid development of the internet and information technology, people are bombarded with a massive amount of information. As a system that helps users find what they want, recommender systems have been widely deployed to alleviate information overload in various applications. In many everyday applications, such as watching movies, playing games, and shopping, users typically perform a series of actions over a period of time to form an interaction sequence. Sequence recommendation is one of the classic tasks of recommender systems, aiming to infer and recommend the next item that a target user might be interested in based on their historical interaction sequences. Traditional sequence recommendation methods mainly include sequence pattern mining and Markov chains (MC). Subsequently, the rapid development of deep learning has significantly improved the performance of recommender systems.

[0003] Therefore, designing a recommendation model has become an urgent problem to be solved. Summary of the Invention

[0004] The purpose of this invention is to provide a method and apparatus for generating a recommendation model for sequence recommendation.

[0005] To achieve one of the aforementioned objectives, one embodiment of the present invention provides a method for generating a recommendation model for sequence recommendation, comprising the following steps: obtaining a user set. Project Collection User and project interaction sequence , ... ,as well as , ... Each of the I recommended items corresponds to one-to-one. , , With users There are interactions between them; where I, T, and J are all natural numbers and are all greater than or equal to 2, 1≤i≤I, 1≤j≤J; a recommendation model is generated, which includes: a first preprocessing module, a second preprocessing module, a Transformer neural network, a hypergraph neural network, and a collaboration network; based on the user set Project Collection Interaction sequence , ... ,as well as , ... The recommendation model is trained using I corresponding recommendation items.

[0006] As a further improvement of one embodiment of the present invention, the first preprocessing module processes the received interaction sequence All are processed as follows: the interaction sequence is... Input sequence embedding layer, extracting based on embedding function Potential characteristics Embedded representation of position ,project Corresponding initial embedding Where d is a natural number, d≥2; then, all the initial embeddings are... , ... All inputs are fed into the Transformer neural network.

[0007] As a further improvement to one embodiment of the present invention, the Transformer neural network is used to: receive initial embeddings , ... Iterative calculation of each The hidden representation of at any pos layer ,Will , ... They are stacked together to form a matrix The Transformer neural network has P layers, where pos and P are natural numbers, and 1 ≤ pos ≤ P. The multi-head self-attention layer in the Transformer neural network is used to: use different learnable linear projections to... Linear projection to In each subspace, parallel use Each attention function generates representations, which are then concatenated and projected again: , , ,in, and These are learnable projection parameters, where Q represents query, K represents key, and V represents key. The values ​​are represented by n and h, where n ≥ 2 and 1 ≤ 1. ≤n; In the Transformer neural network, each item at each location passes through the same feedforward neural network individually. , ,in, These are learnable parameters shared by all positions.

[0008] As a further improvement of one embodiment of the present invention, the second preprocessing module is used to: generate an interaction sequence. Corresponding hypergraph ,in, Let E be the set of all nodes, and E be the set of all hyperedges; for the interaction sequence Construct J hyperedges The semantic dependency score and the correlation value in the final connection matrix are... , , Among them, the multi-channel weighting function The cosine similarity estimation method is used for calculation. Indicates a specific item The first K semantically relevant items are used as the basis for determining the dependency weights calculated across multiple channels using an average pooling operation, ultimately yielding the item set. and Semantic dependency scores between them; final hypergraph hyperedge in Represented as , For semantic hyperedges; transform the hypergraph Input the hypergraph neural network.

[0009] As a further improvement to one embodiment of the present invention, the hypergraph neural network is used for point-hyperedge embedding propagation and hyperedge-node embedding propagation. , ,in, Represents the hypergraph connectivity matrix. Represents the item embedding encoded from the hypergraph convolutional layer. and These are normalized diagonal matrices based on vertex and edge degrees, respectively; the cooperative network uses InfoNCE to calculate the standard binary cross-entropy loss for real and corrupted samples. , ,in, yes The negative sampling process disrupts the original order through row-column shuffling. The score is a rating function. The scores are then calculated and sorted to obtain the final recommendation result.

[0010] This invention also provides a device for generating a recommendation model for sequence recommendation, comprising the following modules: an information acquisition module, used to acquire a user set. Project Collection User and project interaction sequence , ... ,as well as , ... Each of the I recommended items corresponds to one-to-one. , , With users There are interactions between them; where I, T, and J are all natural numbers and all greater than or equal to 2, 1≤i≤I, 1≤j≤J; a model generation model is used to generate a recommendation model, which includes: a first preprocessing module, a second preprocessing module, a Transformer neural network, a hypergraph neural network, and a collaboration network; a training module is used to train a user set... Project Collection Interaction sequence , ... ,as well as , ... The recommendation model is trained using I corresponding recommendation items.

[0011] As a further improvement of one embodiment of the present invention, the first preprocessing module processes the received interaction sequence All are processed as follows: the interaction sequence is... Input sequence embedding layer, extracting based on embedding function Potential characteristics Embedded representation of position ,project Corresponding initial embedding Where d is a natural number, d≥2; then, all the initial embeddings are... , ... All inputs are fed into the Transformer neural network.

[0012] As a further improvement to one embodiment of the present invention, the Transformer neural network is used to: receive initial embeddings , ... Iterative calculation of each The hidden representation of at any pos layer ,Will , ... They are stacked together to form a matrix The Transformer neural network has P layers, where pos and P are both natural numbers, and 1 ≤ pos ≤ P.

[0013] The multi-head self-attention layer in the Transformer neural network is used to: use different learnable linear projections to... Linear projection to In each subspace, parallel use Each attention function generates representations, which are then concatenated and projected again: , , ,in, and These are learnable projection parameters, where Q represents query, K represents key, and V represents key. The values ​​are represented by n and h, where n ≥ 2 and 1 ≤ 1. ≤n; In the Transformer neural network, each item at each location passes through the same feedforward neural network individually. , ,in, These are learnable parameters shared by all positions.

[0014] As a further improvement of one embodiment of the present invention, the second preprocessing module is used to: generate an interaction sequence. Corresponding hypergraph ,in, Let E be the set of all nodes, and E be the set of all hyperedges; for the interaction sequence Construct J hyperedges The semantic dependency score and the correlation value in the final connection matrix are... , , Among them, the multi-channel weighting function The cosine similarity estimation method is used for calculation. Indicates a specific item The first K semantically relevant items are used as the basis for determining the dependency weights calculated across multiple channels using an average pooling operation, ultimately yielding the item set. and Semantic dependency scores between them; final hypergraph hyperedge in Represented as , For semantic hyperedges; transform the hypergraph Input the hypergraph neural network.

[0015] As a further improvement to one embodiment of the present invention, the hypergraph neural network is used for point-hyperedge embedding propagation and hyperedge-node embedding propagation. , ,in, Represents the hypergraph connectivity matrix. Represents the item embedding encoded from the hypergraph convolutional layer. and These are normalized diagonal matrices based on vertex and edge degrees, respectively; the cooperative network uses InfoNCE to calculate the standard binary cross-entropy loss for real and corrupted samples. , ,in, yes The negative sampling process disrupts the original order through row-column shuffling. The score is a rating function. The scores are then calculated and sorted to obtain the final recommendation result.

[0016] Compared to existing technologies, the technical advantages of this invention are as follows: This invention provides a method and apparatus for generating a sequence recommendation model, the method comprising: obtaining a user set U and an item set U. The process involves generating a recommendation model by considering the user-item interaction sequence and the corresponding recommended items for each interaction sequence; the recommendation model includes a first preprocessing module, a second preprocessing module, a Transformer neural network, a hypergraph neural network, and a collaboration network; and training the recommendation model. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the generation method in an embodiment of the present invention; Figure 2 This is a structural diagram of the recommendation model in an embodiment of the present invention; Figure 3 and Figure 4 This is a graph showing the experimental results of the recommended model in this embodiment of the invention. Detailed Implementation

[0018] The present invention will now be described in detail with reference to the embodiments shown in the accompanying drawings. However, these embodiments are not limited to the present invention, and any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the scope of protection of the present invention.

[0019] The terms used herein, such as “above,” “over,” “below,” and “under,” indicating spatial relative position, are for illustrative purposes to describe the relationship of one unit or feature relative to another unit or feature as shown in the accompanying drawings. These terms may be intended to include different orientations of the device in use or operation other than those shown in the figures. For example, if the device in the figures is flipped, a unit described as being “below” or “under” another unit or feature would be “above” that unit or feature. Therefore, the exemplary term “below” can encompass both above and below orientations. The device may be oriented in other ways (rotated 90 degrees or otherwise), and the spatially related descriptive terms used herein will be interpreted accordingly.

[0020] Embodiment 1 of the present invention provides a method for generating a recommendation model for sequence recommendation, such as... Figure 1 and Figure 2 As shown, it includes the following steps: Step 101: Obtain the user set Project Collection User and project interaction sequence , ... ,as well as , ... Each of the I recommended items corresponds to one-to-one. , , With users There are interactions between them; where I, T, and J are all natural numbers and all greater than or equal to 2, 1≤i≤I, 1≤j≤J; Step 102: Generate a recommendation model, which includes: a first preprocessing module, a second preprocessing module, a Transformer neural network, a hypergraph neural network, and a collaborative network; here, the purpose of sequence recommendation is to base it on the previous interactions in the sequence. To predict the next possible item In other words, it is hoped that according to for All projects Preference scores were generated for all. Then, the top-ranked... The item is considered a recommended candidate.

[0021] Step 103: Based on user set Project Collection Interaction sequence , ... ,as well as , ... The recommendation model is trained using I corresponding recommendation items.

[0022] In this embodiment, the first preprocessing module processes the received interaction sequence. All are processed as follows: the interaction sequence is... Input sequence embedding layer, extracting based on embedding function Potential characteristics Embedded representation of position ,project Corresponding initial embedding Where d is a natural number, d≥2; then, all the initial embeddings are... , ... All inputs are fed into the Transformer neural network.

[0023] In this embodiment, the Transformer neural network is used to: receive initial embeddings , ... Iterative calculation of each The hidden representation of at any pos layer ,Will , ... They are stacked together to form a matrix The Transformer neural network has P layers, where pos and P are natural numbers, and 1 ≤ pos ≤ P. Here, in the Transformer neural network, each layer consists of two sub-layers: a multi-head self-attention sub-layer and a position feedforward network layer.

[0024] The multi-head self-attention layer in the Transformer neural network is used to: use different learnable linear projections to... Linear projection to In each subspace, parallel use Each attention function generates representations, which are then concatenated and projected again: , , ,in, and These are learnable projection parameters, where Q represents query, K represents key, and V represents key. The values ​​are represented by n and h, where n ≥ 2 and 1 ≤ 1. ≤n; In the Transformer neural network, each item at each location passes through the same feedforward neural network individually. , ,in, These are learnable parameters shared by all positions.

[0025] Understandably, the implementation process of this Transformer neural network is as follows: after obtaining a preliminary embedded representation of the user's interaction sequence through the sequence embedding layer, it is fed into the Transformer layer to model the sequence-level item representation.

[0026] In this embodiment, the second preprocessing module is used to: generate an interaction sequence. Corresponding hypergraph ,in, Let E be the set of all nodes, and E be the set of all hyperedges; for the interaction sequence Construct J hyperedges The semantic dependency score and the correlation value in the final connection matrix are... , , Among them, the multi-channel weighting function The cosine similarity estimation method is used for calculation. Indicates a specific item The first K semantically relevant items are used as the basis for determining the dependency weights calculated across multiple channels using an average pooling operation, ultimately yielding the item set. and Semantic dependency scores between them; final hypergraph hyperedge in Represented as , For semantic hyperedges; transform the hypergraph Input the hypergraph neural network.

[0027] Here, a custom supergraph was designed. ,in, Let E be the set of all nodes, and let E be the set of all hyperedges. Two methods for defining hyperedges were chosen to capture higher-order relations from two different perspectives: sequential hyperedges and semantic hyperedges.

[0028] use Let represent a set of sequence hyperedges, where each hyperedge is derived from the interaction sequence of each user. Specifically, for a sequence Each item in Construct the hyperedge as Formally, a hyperedge can be viewed as a set of nodes. A subset of that set, any two items in the same user interaction sequence will be concatenated. Additionally, [the following will be included:] Defined as a set of semantic hyperedges, each item will be connected to a hyperedge based on the semantic dependency score calculated by the metric learning component. The top K items with the highest scores will be connected to this hyperedge. Assume... Indicates a specific item The semantic dependency scores and relevance values ​​in the final connectivity matrix of the first K semantically relevant item sets can be defined as follows: , , Among them, the multi-channel weighting function Cosine similarity estimation is used to calculate dependency weights across multiple channels under average pooling operations, ultimately yielding the project results. and Semantic dependency scores between them; final hypergraph The hyperedge E in the equation is denoted as E .

[0029] In this embodiment, the hypergraph neural network is used for point-hyperedge embedding propagation and hyperedge-node embedding propagation. , ,in, Represents the hypergraph connectivity matrix. Represents the item embedding encoded from the hypergraph convolutional layer. and These are normalized diagonal matrices based on vertex and edge degree, respectively. The implementation process of the hypergraph neural network is as follows: a hypergraph that integrates high-order information from different angles is obtained by using two hyperedge construction methods: sequential hyperedge and semantic hyperedge. The hypergraph is then fed into a hypergraph convolutional network to obtain a global graph-level item representation.

[0030] The collaborative network uses InfoNCE to calculate the standard binary cross-entropy loss for real and corrupted samples. , ,in, yes The negative sampling process disrupts the original order through row-column shuffling. The score is a rating function. The scores are then calculated and sorted to obtain the final recommendation result.

[0031] Here, node representations learned from the two networks communicate with each other through a contrastive learning task, maximizing the mutual information between sequence embeddings learned in different networks, enabling the two networks to have a certain degree of synergy to obtain better representations. InfoNCE is used to compute the standard binary cross-entropy loss for real samples (positive) and corrupted samples (negative). , .

[0032] in, yes Negative sampling disrupts the original order through row-column shuffling. The score is a rating function that takes two vectors as input, simply takes the dot product of the two vectors, and uses the result as a consistency score between them. Finally, the recommendation task and the contrastive learning task are unified into a single framework. It is a hyperparameter that controls the impact of the comparison task.

[0033] The embeddings of the two networks are aggregated by an attention layer, and the inner product of the representation of the predicted item and the embedding representations of different candidate items in the candidate set is used to calculate the score. The calculated scores are then sorted to obtain the final recommendation result. , .

[0034] Implementation process: The representations from the two networks are aggregated through an attention layer, and the score is calculated by combining the inner product of the representation of the predicted item with the embedding representations of the different candidate items in the candidate set.

[0035] To evaluate the performance of the aforementioned recommendation model, the inventors conducted experiments on three real-world recommendation datasets: Taobao, Retailrocket, and IJCAI-15. Detailed dataset information is available as follows: Figure 3 As shown.

[0036] The inventors selected two commonly used evaluation metrics in sequence recommendation, hit rate and NDCG (Normalized Discounted Cumulative Gain), to evaluate the recommendation performance of the aforementioned recommendation model. Hit rate calculates the percentage of items the user's next actual interaction will appear in the top N items in the predicted ranking, and is used to evaluate the accuracy of the prediction. On the other hand, NDCG is a location-aware metric that focuses on whether the discovered items are located in the user's salient position to evaluate the predicted ranking. Results for HR@{5,10} and NDCG@{5,10} are reported. Furthermore, comparisons with common sequence recommendation models are made, with specific recommendation performance as follows: Figure 4 As shown.

[0037] Embodiment 2 of the present invention provides a device for generating a recommendation model for sequence recommendation, comprising the following modules: The information acquisition module is used to acquire user sets. Project Collection User and project interaction sequence , ... ,as well as , ... Each of the I recommended items corresponds to one-to-one. , , With users There are interactions between them; where I, T, and J are all natural numbers and all greater than or equal to 2, 1≤i≤I, 1≤j≤J; A model generation model is used to generate a recommendation model, the recommendation model comprising: a first preprocessing module, a second preprocessing module, a Transformer neural network, a hypergraph neural network, and a collaborative network; Training module, used for training based on user set Project Collection Interaction sequence , ... ,as well as , ... The recommendation model is trained using I corresponding recommendation items.

[0038] In this embodiment, the first preprocessing module processes the received interaction sequence. All are processed as follows: the interaction sequence is... Input sequence embedding layer, extracting based on embedding function Potential characteristics Embedded representation of position ,project Corresponding initial embedding Where d is a natural number, d≥2; then, all the initial embeddings are... , ... All inputs are fed into the Transformer neural network.

[0039] In this embodiment, the Transformer neural network is used to: receive initial embeddings , ... Iterative calculation of each The hidden representation of at any pos layer ,Will , ... They are stacked together to form a matrix The Transformer neural network has P layers, where pos and P are both natural numbers, and 1 ≤ pos ≤ P. The multi-head self-attention layer in the Transformer neural network is used to: use different learnable linear projections to... Linear projection to In each subspace, parallel use Each attention function generates representations, which are then concatenated and projected again: , , ,in, and These are learnable projection parameters, where Q represents query, K represents key, and V represents key. The values ​​are represented by n and h, where n ≥ 2 and 1 ≤ 1. ≤n; In the Transformer neural network, each item at each location passes through the same feedforward neural network individually. , ,in, These are learnable parameters shared by all positions.

[0040] In this embodiment, the second preprocessing module is used to: generate an interaction sequence. Corresponding hypergraph ,in, Let E be the set of all nodes, and E be the set of all hyperedges; for the interaction sequence Construct J hyperedges The semantic dependency score and the correlation value in the final connection matrix are... , , Among them, the multi-channel weighting function The cosine similarity estimation method is used for calculation. Indicates a specific item The first K semantically relevant items are used as the basis for determining the dependency weights calculated across multiple channels using an average pooling operation, ultimately yielding the item set. and Semantic dependency scores between them; final hypergraph hyperedge in Represented as , For semantic hyperedges; transform the hypergraph Input the hypergraph neural network.

[0041] In this embodiment, the hypergraph neural network is used for point-hyperedge embedding propagation and hyperedge-node embedding propagation. , ,in, Represents the hypergraph connectivity matrix. Represents the item embedding encoded from the hypergraph convolutional layer. and These are normalized diagonal matrices based on vertex and edge degree, respectively. The collaborative network uses InfoNCE to calculate the standard binary cross-entropy loss for real and corrupted samples. , ,in, yes The negative sampling process disrupts the original order through row-column shuffling. The score is a rating function. The scores are then calculated and sorted to obtain the final recommendation result.

[0042] It should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0043] The detailed descriptions listed above are merely specific descriptions of feasible embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. All equivalent embodiments or modifications made without departing from the spirit of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating a recommendation model for sequence recommendation, characterized in that, Includes the following steps: Get user set Project Collection User and project interaction sequence , ... ,as well as , ... Each of the I recommended items corresponds to one-to-one. , , With users There are interactions between them; where I, T, and J are all natural numbers and all greater than or equal to 2, 1≤i≤I, 1≤j≤J; A recommendation model is generated, comprising: a first preprocessing module, a second preprocessing module, a Transformer neural network, a hypergraph neural network, and a collaboration network; the first preprocessing module processes the received interaction sequences. All are processed as follows: the interaction sequence is... Input sequence embedding layer, extracting based on embedding function Potential characteristics Embedded representation of position ,project Corresponding initial embedding Where d is a natural number, d≥2; then, all the initial embeddings are... , ... All inputs are fed into the Transformer neural network; the second preprocessing module is used to generate interaction sequences. Corresponding hypergraph ,in, Let E be the set of all nodes, and E be the set of all hyperedges; for the interaction sequence Construct J hyperedges The semantic dependency score and the correlation value in the final connection matrix are... , , Among them, the multi-channel weighting function The cosine similarity estimation method is used for calculation. Indicates a specific item The first K semantically relevant items are used as the basis for calculating dependency weights across multiple channels using an average pooling operation, ultimately yielding the item set. and Semantic dependency scores between them; final hypergraph hyperedge in Represented as , For semantic hyperedges; transform the hypergraph Input the hypergraph neural network; the hypergraph neural network is used for point-hyperedge embedding propagation and hyperedge-node embedding propagation. , ,in, Represents the hypergraph connectivity matrix. Represents the item embedding encoded from the hypergraph convolutional layer. and These are normalized diagonal matrices based on vertex and edge degrees, respectively; the cooperative network uses InfoNCE to calculate the standard binary cross-entropy loss for real and corrupted samples. , ,in, yes Negative sampling is performed, and the original order is disrupted through row-column shuffling. The score is a scoring function. The scores are then calculated and sorted to obtain the final recommendation result. Based on user set Project Collection Interaction sequence , ... ,as well as , ... The recommendation model is trained using I corresponding recommendation items.

2. The generation method according to claim 1, characterized in that, The Transformer neural network is used to: receive initial embeddings , ... Iterative calculation of each Hidden representation of at any pos layer ,Will , ... They are stacked together to form a matrix The Transformer neural network has P layers, where pos and P are both natural numbers, and 1 ≤ pos ≤ P. The multi-head self-attention layer in the Transformer neural network is used to: use different learnable linear projections to... Linear projection to In each subspace, parallel use Each attention function generates representations, which are then concatenated and projected again: , , ,in, and These are learnable projection parameters, where Q represents query, K represents key, and V represents key. The values ​​are represented by n and h, where n ≥ 2 and 1 ≤ 1. ≤n; In the Transformer neural network, each item at each location passes through the same feedforward neural network individually. , ,in, These are learnable parameters shared by all positions.

3. A device for generating a recommendation model for sequence recommendation, characterized in that, Includes the following modules: The information acquisition module is used to acquire user sets. Project Collection User and project interaction sequence , ... ,as well as , ... Each of the I recommended items corresponds to one-to-one. , , With users There are interactions between them; where I, T, and J are all natural numbers and all greater than or equal to 2, 1≤i≤I, 1≤j≤J; A model generation model is used to generate a recommendation model, which includes: a first preprocessing module, a second preprocessing module, a Transformer neural network, a hypergraph neural network, and a collaboration network; the first preprocessing module processes the received interaction sequences. All are processed as follows: the interaction sequence is... Input sequence embedding layer, extracting based on embedding function Potential characteristics Embedded representation of position ,project Corresponding initial embedding Where d is a natural number, d≥2; then, all the initial embeddings are... , ... All inputs are fed into the Transformer neural network; the second preprocessing module is used to generate interaction sequences. Corresponding hypergraph ,in, Let E be the set of all nodes, and E be the set of all hyperedges; for the interaction sequence Construct J hyperedges The semantic dependency score and the correlation value in the final connection matrix are... , , Among them, the multi-channel weighting function The cosine similarity estimation method is used for calculation. Indicates a specific item The first K semantically relevant items are used as the basis for calculating dependency weights across multiple channels using an average pooling operation, ultimately yielding the item set. and Semantic dependency scores between them; final hypergraph hyperedge in Represented as , For semantic hyperedges; transform the hypergraph Input the hypergraph neural network; the hypergraph neural network is used for point-hyperedge embedding propagation and hyperedge-node embedding propagation. , ,in, Represents the hypergraph connectivity matrix. Represents the item embedding encoded from the hypergraph convolutional layer. and These are normalized diagonal matrices based on vertex and edge degrees, respectively; the cooperative network uses InfoNCE to calculate the standard binary cross-entropy loss for real and corrupted samples. , ,in, yes Negative sampling is performed, and the original order is disrupted through row-column shuffling. The score is a scoring function. The scores are then calculated and sorted to obtain the final recommendation result. Training module, used for training based on user set Project Collection Interaction sequence , ... ,as well as , ... The recommendation model is trained using I corresponding recommendation items.

4. The generating apparatus according to claim 3, characterized in that, The Transformer neural network is used to: receive initial embeddings , ... Iterative calculation of each Hidden representation of at any pos layer ,Will , ... They are stacked together to form a matrix The Transformer neural network has P layers, where pos and P are both natural numbers, and 1 ≤ pos ≤ P. The multi-head self-attention layer in the Transformer neural network is used to: use different learnable linear projections to... Linear projection to In each subspace, parallel use Each attention function generates representations, which are then concatenated and projected again: , , ,in, and These are learnable projection parameters, where Q represents query, K represents key, and V represents key. The values ​​are represented by n and h, where n ≥ 2 and 1 ≤ 1. ≤n; In the Transformer neural network, each item at each location passes through the same feedforward neural network individually. , ,in, These are learnable parameters shared by all positions.

Images