A text representation alignment sequence recommendation method based on decoupled contrast learning

Abstract

The application discloses a text representation alignment sequence recommendation method based on decoupling contrast learning, and belongs to the technical field of sequence recommendation and deep learning. The specific steps comprise: constructing a text-based description data set of sequence recommendation; creating a text semantic encoder of an item and a sequence to respectively generate corresponding text semantic vector representations; learning decoupling representations of the sequence and a target item through unsupervised and supervised contrast learning tasks, and realizing bidirectional semantic alignment between them; the application effectively separates the overall sequence semantics and single item features by decoupling the representations of the sequence and the target item, realizes high-quality semantic representations of the interactive sequence and the target item, and thus significantly improves the accuracy and effect of sequence recommendation. The method innovatively applies decoupling contrast learning to sequence recommendation, and can be widely applied to e-commerce recommendation, content recommendation and other scenes, and has important practical application value.

Description

Technical Field

[0001] This invention belongs to the field of sequence recommendation technology, specifically relating to a text representation alignment sequence recommendation method based on decoupled contrastive learning. Background Technology

[0002] The goal of sequential recommendation is to accurately predict the next item a user might be interested in by modeling their historical interaction sequences, such as browsing and clicking, on online platforms. With the development of pre-trained language models, especially the successful application of BERT (Bidirectional Encoder Representation from Transformers) in machine translation and sequential text-based natural language processing tasks and recommendations, feasible solutions for processing sequential text data have emerged. These models have demonstrated an impressive ability to understand textual semantics and extract information. Based on this, the basic idea of ​​this study is to use pre-trained models such as BERT to encode rich textual information of items to extract user interest representations of sequences and improve recommendation performance. These models generate textual representations of items (e.g., titles) and learn item and sequence representations, directly optimizing user interest representations by aligning the sequence representations with the textual semantic representations of the target items. In this process, the textual representation of the target item in the recommendation data serves as a supervisory signal. However, this approach has several limitations: (1) the unidirectional emphasis on semantic alignment with the next item leads to overfitting of the model to patterns in the training data related to positive items, resulting in biased interest representations. (2) This approach ignores the benefits of bidirectional supervision in achieving comprehensive semantic representation, resulting in incomplete sequence representation and semantic matching bias. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention aims to provide a text representation alignment sequence recommendation method based on decoupled contrastive learning. This invention proposes decoupling sequence and item representations and learning them independently to avoid information mixing. Furthermore, effectively aligning sequence representations with target item representations helps to understand more accurate user interest representations, ultimately improving recommendation performance. Inspired by self-supervised learning techniques that construct self-supervised signals using raw data, and the effectiveness of bidirectional learning in sequence representations, we propose a hybrid contrastive learning strategy. This strategy combines unsupervised and supervised contrastive learning paradigms to decouple and enhance the ability to decouple and augment sequence representations of user interest. Our unsupervised contrastive learning facilitates robust exploration of data features in recommendation tasks by decoupling sequence and item representations into two distinct subspaces. We employ a bidirectional supervised contrastive approach to achieve deep alignment between sequence representations and the target item space.

[0004] To achieve the above objectives, this invention proposes a text representation alignment sequence recommendation method based on decoupled contrastive learning, comprising the following steps:

[0005] Step 1: Concatenate the multiple attribute descriptions of the items in sequence to construct a sequence-item recommendation data based on text descriptions;

[0006] Step 2: Create a semantic encoder for the description text of the item based on the pre-trained language model. Take the text description of the item as input and output the corresponding text semantic vector representation of the description text as the text representation of the item.

[0007] Step 3: Create a semantic encoder for the descriptive text of the sequence based on the pre-trained language model. The text description of the sequence is taken as input, and the corresponding text semantic vector representation of the descriptive text is output as the text representation of the sequence.

[0008] Step 4: Construct an unsupervised contrastive learning task based on the item text representation and sequence text representation obtained in Step 2 and Step 3 respectively, and learn the decoupled representation of the sequence and the target item;

[0009] Step 5: Based on the sequence text representation and item text representation obtained in Step 2 and Step 3, construct a supervised bidirectional contrastive learning task to achieve bidirectional semantic alignment between the sequence and its target item;

[0010] Step 6: Utilize a multi-task training strategy to jointly optimize the loss function of the unsupervised and supervised contrastive learning training tasks in Steps 4 and 5, thereby promoting the deep alignment between the sequence and the semantic vector representation of the target item text.

[0011] The beneficial effects of this invention are as follows:

[0012] The proposed method for recommending text representations based on decoupled contrastive learning addresses the problem of overfitting of models to patterns in training data related to positive items during model training, leading to semantic matching bias and biased interest representations. This is achieved by decoupling the sequence and item representation spaces and independently learning their text semantic vector representations. Furthermore, a bidirectional supervised alignment method is employed between the sequence text semantic vector representation and the target item text semantic vector representation to achieve high-quality optimization of sequence representations, ultimately improving recommendation performance and thus solving the problems mentioned in the background art. Attached Figure Description

[0013] Figure 1 This is a flowchart of the text representation alignment sequence recommendation method based on decoupled contrastive learning according to the present invention. Detailed Implementation

[0014] The technical solutions of the present invention will be further clearly and completely described below with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] This invention provides the following technical solution: a text representation alignment sequence recommendation method based on decoupled contrastive learning. The specific steps are illustrated using a shopping website recommendation scenario as an example. The described text representation alignment sequence recommendation method based on decoupled contrastive learning is applied to the shopping recommendation scenario, as follows: Figure 1 As shown, the specific steps are as follows:

[0016] Step 1: Concatenate the multiple attribute descriptions of the item in sequence to construct recommendation data based on the text descriptions;

[0017] In practice, the text-based recommendation data is constructed based on the user-generated interactive item sequences (represented by numerical identifiers) in the existing dataset, the target items in the sequence (represented by numerical identifiers), and the corresponding attribute description text for each item. The specific processing method is as follows:

[0018] Step 1-1. Use s = {v1, v2, ..., v...} n} represents the original sequence sample represented using a numeric identifier, where v i This refers to an item that has been interacted with by the user for the i-th time in chronological order, where n represents the total number of interactions generated by the user in this sequence of samples. Here, "interaction" refers to a consistent description across different scenarios, such as clicks and purchases. Each item v... i Each item is associated with a unique identifier (represented by a number) and a descriptive text (e.g., the item's product description, title, or brand). Assuming the items in the sequence recommendation dataset contain k attribute types, this invention focuses only on the textual descriptions of these items to learn the vector representation of the sequence. Specifically, for each item v in the sequence that interacts with the user... i In this invention, the descriptive text is processed into "key-value pair" form: {(a1, d1), (a2, d2), ..., (a...} k d k)}, where key 'a' represents a textual attribute (called attribute text, such as 'brand'), and value 'd' represents the text describing the attribute, i.e., the attribute description text. For multiple types of attribute text, [SEP] is used as a separator to concatenate various attribute texts and their attribute description texts. In this way, the item order expressed by the original numerical identifier is processed and mapped to the corresponding text description {a1d1[SEP]a2d2[SEP]...[SEP]a k d k}

[0019] Steps 1-2. For the assembled item v i The attribute description text {a1d1[SEP]a2d2[SEP]...[SEP]a k d k} is truncated. Specifically, item v i The description text is a sequence of words, here using T i This can be formalized as T i ={w i 1, w i 2, ..., w i j , ..., w i c}, where any one word w i j From a shared vocabulary V, c is the length of the truncated text after concatenating various types of descriptive text. Here, "item" refers to the clicked item and the target item in the interaction sequence.

[0020] Steps 1-3. For the sequence s = {v1, v2, ..., v...} n The attribute text and attribute description text of each item in the sequence are concatenated according to the order of interaction in the sequence to obtain the corresponding text-based sequence data. This constitutes a text-based sequence-target item data sample.

[0021] Step 2: Create a semantic encoder for the descriptive text of items (clicked items and target items in the interaction sequence) based on a pre-trained language model. The encoder takes the text description of the item as input and outputs its corresponding semantic vector representation as the item's text representation. Specifically, Step 2 involves:

[0022] Step 2-1. Construct an item text semantic encoder based on the BERT model, in item v i Add a special marker [CLS] before the text sequence corresponding to the clicked item in the sequence and the corresponding target item in the sequence.

[0023] Step 2-2. Combine the sequentially concatenated [CLS] with the multiple item attribute texts in the sequence to obtain {[CLS]; T1, T2, ..., T n}. Where Ti={w i 1, w i 2, ..., w i c} represents item v i All words in the attribute text are input into the BERT model, and the average of the output vectors is taken to generate the vector for that item. Represents the vector space in the real number field The median dimension is d W The vector. Where the parameters of BERT are θ. b .

[0024] Steps 2-3. For the target item v+ that the user actually clicked corresponding to the sequence, its attribute text is T. i+ ={w i 1+ w i 2+ , ..., w i c+}, using the same parameter θ as in step 2-2 b The BERT model is used for encoding to obtain the text semantic vector of the target item.

[0025] Step 3: Create a semantic encoder for the sequence description text based on a pre-trained language model. The encoder takes the text description of the sequence as input and outputs its corresponding semantic vector representation as the sequence text representation. Specifically, Step 3 involves:

[0026] Step 3-1. Based on the semantic vector of the item text generated in Step 2, the process corresponds to the original representation of the sequence sample s = {v1, v2, ..., v...} n The BERT model is used to generate text semantic vectors corresponding to each item in s, represented as follows:

[0027] Step 3-2. Construct a Transformer-based sequence text semantic encoder to encode {x1, x2, ..., x...} n As input, the output of the Transformer at the last time step represents an initial sequential text semantic vector f = Transformer(θ). e [x1, x2, ..., x n ]) Where θ e Represents the parameters of a Transformer-based sequence text semantic encoder;

[0028] Step 3-3. Building upon Step 3-2, a sequence-level residual connective block is introduced, which is the vector of each item corresponding to the sequence before the original representation is input into the Transformer. Performing average pooling on it yields f′ = MeanPooling([x1, x2, ..., x...). n ]), f′ represents the expression obtained by analyzing the vectors of all items in the sequence [x1, x2, ..., x]. n The vector obtained after performing mean pooling is a vector f′ with dimension d. W The vector is a composite vector. Specifically, f′ is a vector that averages the representations of all items in the sequence, generating a comprehensive representation of the overall sequence information. This vector is used to maintain the consistency of the original input information and preserve the overall features when introducing sequence-level residual join blocks. Then, f′ is added to the initial sequence text semantic vector f from the Transformer's output to obtain h. s =f+f′, This serves as the semantic vector representation of the sequence text for use as input in subsequent steps 4 and 5. This step primarily addresses the potential problems of gradient vanishing and loss of original information caused by multiple Transformer layers.

[0029] Step 4: Construct an unsupervised contrastive learning task based on the target item text representation and sequence text representation obtained in Steps 2 and 3, respectively, to learn the decoupled representation of the sequence and the target item. Specifically, Step 4 involves:

[0030] Step 4-1. Semantic vector representation of the sequence text generated in Step 3-3 A sequence-level contrastive learning training task is introduced. Positive and negative sample pairs are constructed, and a sequence-level contrastive learning loss function is designed for model training. The process of constructing sequence-level positive sample pairs is as follows: Given a sequence sample, by randomly deleting items from the sequence, variants of the original sequence are generated as positive samples. The text semantic vector representation of the original sequence uses... This indicates that the text semantic vector representation of the positive sample sequence uses... The process of constructing sequence-level negative sample pairs is as follows: For a given sequence sample, other sequence samples within the batch during training are used as negative samples of that sequence sample, and their text semantic vector representation uses... Let j represent the sample number in the batch. In the "Sequence-Sequence (SS)" contrastive learning training task, each batch contains B training instances. The objective of contrastive learning is to maximize the similarity of positive sample pairs while minimizing the similarity of negative sample pairs. The loss function is defined as:

[0031]

[0032] Here, ex() represents an exponential function with base e; τ is a human-defined hyperparameter used to adjust the smoothness of the similarity distribution between positive and negative sequence samples.

[0033] Step 4-2. Based on the target item text semantic vector representation generated in Step 2-3, an item-level "item-item" contrastive learning training task is introduced. This is achieved by constructing positive and negative sample pairs and designing an item-level contrastive learning loss function for model training. The process of constructing item-level positive sample pairs is as follows: Given a target item sample, by randomly deleting words from the target item sample, variants of the original target item are generated as positive samples of that target item sample. The text semantic vector representation of the original target item uses... This indicates that the text semantic vector representation of the positive sample target item uses... The process of constructing item-level negative sample pairs is as follows: For a given target item sample, other target item samples within the batch during training are used as negative samples of that target item sample, and their text semantic vector representation uses... Let j represent the sample number in the batch. In the "item-item (II)" contrastive learning training task, the goal of contrastive learning is to maximize the similarity of positive sample pairs while minimizing the similarity of negative sample pairs. The loss function is defined as:

[0034]

[0035] Here, exp() represents an exponential function with base e; τ is a user-defined hyperparameter used to adjust the smoothness of the similarity distribution between positive and negative target item samples. The τ here is the same as the τ in step 4-1.

[0036] The goal of this unsupervised contrastive learning method is to teach the model to distinguish between similar items (target item and its positive samples) and different items (target item and negative samples). Constructing positive samples through random deletion operations enhances the model's robustness to item representations. Using other items within the batch as negative samples effectively utilizes computational resources and provides diverse contrast samples.

[0037] Step 5: Based on the sequence text representations and target item text representations obtained in Steps 2 and 3, a supervised bidirectional contrastive learning task is constructed to achieve bidirectional semantic alignment between the sequence and its target item; Step 5 specifically involves:

[0038] Step 5-1. When introducing the contrastive learning training task of "Sequence-Target Item (SI)" and "Target Item-Sequence (IS)," set a batch containing B training instances during the contrastive learning training process, where each training instance contains a pair of samples: a sequence sample and its corresponding target item sample.

[0039] Step 5-2. Based on the sequence text semantic vector representation and target item text semantic vector representation generated in Steps 3-3 and 2-3, generate the text semantic vector representation of the training instance: in, It is the text semantic vector representation of the sequence obtained in step 3-3. It is the text semantic vector representation of the target item obtained from steps 2-3.

[0040] Step 5-3. Construct a contrastive learning training task of "sequence-target item" by building positive and negative sample pairs. The process of constructing positive sample pairs is as follows: Given sequence samples s j Compare it with the corresponding target item sample v in the training instances. j+ These are combined to form positive sample pairs, whose text semantic vectors are represented as follows: The process of constructing negative sample pairs is as follows: For a given sequence of samples, take B-1 target item samples v from the training batch, excluding the positive item samples. j′ When combined with it to form a negative sample pair, its text semantic vector is represented as: In the "Sequence-Target Item (SI)" contrastive learning training task, the goal of contrastive learning is to maximize the similarity of positive sample pairs while minimizing the similarity of negative sample pairs. The loss function is defined as:

[0041]

[0042] Here, exp() represents an exponential function with base e; τ is a user-defined hyperparameter used to adjust the smoothness of the similarity distribution between positive and negative samples. The τ here is the same as the τ in steps 4-1 and 4-2.

[0043] Step 5-4. Construct a contrastive learning training task for "target item-sequence (IS)" by building positive and negative sample pairs. The process of constructing positive sample pairs is as follows: Given a target item sample v j+ Compare it with the corresponding sequence samples s in the training instances j+These are combined to form positive sample pairs, whose text semantic vectors are represented as follows: The process of constructing negative sample pairs is as follows: For a given target item sample, take the B-1 other sequence samples s from the training batch, excluding the positive sequence samples. j′ When combined with it to form a negative sample pair, its text semantic vector is represented as: In the "Target Item-Sequence (IS)" contrastive learning training task, the goal of contrastive learning is to maximize the similarity of positive sample pairs while minimizing the similarity of negative sample pairs. The loss function is defined as:

[0044]

[0045] Here, exp() represents an exponential function with base e; τ is a user-defined hyperparameter used to adjust the smoothness of the similarity distribution between positive and negative samples. The τ here is the same as the τ in steps 4-1, 4-2, and 5-3.

[0046] Its core objective is to learn highly aligned representations of sequences and their target item positive samples in the semantic space, thereby ensuring that the target item ranks higher in the recommendation probability order.

[0047] Step 6: Utilize a multi-task training strategy to jointly optimize the loss functions of the unsupervised and supervised contrastive learning training tasks in Steps 4 and 5, promoting deep alignment between the sequence and the semantic vector representation of the target item text; finally, apply the trained text representation alignment sequence recommendation model based on decoupled contrastive learning to specific recommendation scenarios. Specifically, Step 6 involves:

[0048] Based on the loss functions of the unsupervised and supervised contrastive learning training tasks in steps 4 and 5, a weight is assigned to each loss function to balance their contributions. Their weighted sum is then calculated as the total loss function for the entire text representation alignment sequence recommendation model based on decoupled contrastive learning, i.e.:

[0049]

[0050] Where, λ i These are artificially defined hyperparameters that represent the weights of each loss function, λ1+λ2+λ3+λ4=1.

[0051] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A text representation alignment sequence recommendation method based on decoupled contrastive learning, characterized in that, Includes the following steps: Step 1: Concatenate the multiple attribute descriptions of the items in sequence to construct a sequence-item recommendation data based on text descriptions; Step 2: Create a semantic encoder for the description text of the item based on the pre-trained language model. Take the text description of the item as input and output the corresponding text semantic vector representation of the description text as the text representation of the item. Step 3: Create a semantic encoder for the descriptive text of the sequence based on the pre-trained language model. The text description of the sequence is taken as input, and the corresponding text semantic vector representation of the descriptive text is output as the text representation of the sequence. Step 4: Construct an unsupervised contrastive learning task based on the item text representation and sequence text representation obtained in Step 2 and Step 3 respectively, and learn the decoupled representation of the sequence and the target item; Step 5: Based on the sequence text representation and item text representation obtained in Step 2 and Step 3, construct a supervised bidirectional contrastive learning task to achieve bidirectional semantic alignment between the sequence and its target item; Step 6: Utilize a multi-task training strategy to jointly optimize the loss function of the unsupervised and supervised contrastive learning training tasks in Steps 4 and 5, thereby promoting the deep alignment between the sequence and the semantic vector representation of the target item text. Step 5 specifically involves: Step 5-1: Introduce the contrastive learning training tasks of "sequence-target item" and "target item-sequence". Set up a batch containing B training instances during the contrastive learning training process, where each training instance contains a pair of samples: a sequence sample and its corresponding target item sample. Step 5-2: Based on the sequence text semantic vector representation and target item text semantic vector representation generated in Steps 3-3 and 2-3, generate the text semantic vector representation of the training instance: ; Step 5-3: Construct positive and negative sample pairs to build a contrastive learning training task of "sequence-target item"; The process of constructing positive sample pairs is as follows: Given a sequence of samples Compare it with the corresponding target item samples in the training instances. These are combined to form positive sample pairs, whose text semantic vectors are represented as follows: The process of constructing negative sample pairs is as follows: For a given sequence of samples, select all samples from the training batch except for the positive items. Target item sample When combined with it to form a negative sample pair, its text semantic vector is represented as: In the "sequence-target item" contrastive learning training task, the loss function is defined as: ； in, This represents an exponential function with base e; It is a hyperparameter, where B represents the number of training instances in a batch; Step 5-4: Construct positive and negative sample pairs to build a contrastive learning training task of "target item-sequence"; The process of constructing positive sample pairs is as follows: Given a target item sample Compare it with the corresponding sequence samples in the training instances. These are combined to form positive sample pairs, whose text semantic vectors are represented as follows: The process of constructing negative sample pairs is as follows: For a given target item sample, all samples in the training batch other than the positive sequence samples are selected. Sequence samples When combined with it to form a negative sample pair, its text semantic vector is represented as: In the "target item-sequence" contrastive learning training task, the loss function is defined as: 。 2. The text representation alignment sequence recommendation method based on decoupled contrastive learning according to claim 1, characterized in that, Step 1 specifically includes the following sub-steps: Step 1-1: Use This represents the original sequence sample represented using numeric identifiers, where It is generated in chronological order with the user. Each item represents an interaction with another item, where n represents the total number of interactions generated by the user in this sequence sample; each item Each item in the sequence is associated with a unique identifier and a descriptive text; this applies to every item that the user interacts with. The description text is processed into key-value pairs: Here, key 'a' represents a textual attribute, and value 'd' represents the text describing the attribute; that is, the attribute description text. As a separator, it concatenates multiple attribute texts and their attribute description texts, processing and mapping the item order expressed by the original numerical identifiers to the corresponding text descriptions. ; Steps 1-2: Assemble the items Attribute description text Cut off; items Description text It is a sequence of words , of which any one word From a shared vocabulary , Describe the length of the truncated text after concatenation for various types of text; Steps 1-3: For the sequence The attribute text and attribute description text of each item in the sequence are concatenated according to the interaction order in the sequence to obtain the corresponding text-based sequence data, which constitutes the text description sequence-item recommendation data.

3. The text representation alignment sequence recommendation method based on decoupled contrastive learning according to claim 2, characterized in that, Step 2 is described in detail below: Step 2-1: Construct an item text semantic encoder based on the BERT model, in the item Add a special marker before the corresponding text sequence ; Step 2-2: Separate the sequentially connected parts The result of multiple item attribute texts in the sequence The input is fed into a pre-trained language model, and the average of the output vectors is taken to generate the text semantic vector of the item. , representing the vector space of the real number field The dimension is The vector; where the parameters of the pre-trained language model are ; Steps 2-3: For the target item actually clicked by the user corresponding to the sequence. Its attribute text is Use the same parameters as in step 2-2 Encode the target item using a pre-trained language model to obtain the text semantic vector. .

4. The text representation alignment sequence recommendation method based on decoupled contrastive learning according to claim 3, characterized in that, Step 3 specifically involves: Step 3-1: Construct a Transformer-based sequence text semantic encoder to process the text semantic vector x. i sequence As input, the output of the sequence text semantic encoder at the last time step represents an initial sequence text semantic vector. ,in Parameters representing the sequence text semantic encoder; Step 3-2: Vectors of each item corresponding to the sequence before the original representation is input into the Transformer. Perform average pooling on it to obtain Then The initial sequence text semantic vector of the Transformer's output Adding them together gives , as a semantic vector of the sequence text.

5. The text representation alignment sequence recommendation method based on decoupled contrastive learning according to claim 4, characterized in that, Step 4 specifically involves: Step 4-1: Based on the semantic vector representation of sequential text, introduce a sequence-level contrastive learning training task. This involves constructing positive and negative sample pairs and designing a sequence-level contrastive learning loss function for model training. The process of constructing sequence-level positive sample pairs is as follows: Given a sequence sample, by randomly deleting items from the sequence, a variant of the original sequence is generated as a positive sample of that sequence sample; the text semantic vector representation of the original sequence is used... This indicates that the text semantic vector representation of the positive sample sequence uses... Indicates; among which, Indicates the sample number in the batch; The process of constructing sequence-level negative sample pairs is as follows: For a given sequence sample, other sequence samples within the batch during training are used as negative samples of that sequence sample, and their text semantic vector representation uses... In sequence-level contrastive learning training tasks, the loss function is defined as: ; Step 4-2: Based on the text semantic vector representation of the target item, introduce an item-level contrastive learning training task. This involves constructing positive and negative sample pairs and designing an item-level contrastive learning loss function for model training. The process of constructing positive sample pairs at the item level is as follows: Given a target item sample, by randomly deleting words from the target item sample, variants of the original target item are generated as positive samples of that target item sample; the text semantic vector representation of the original target item uses... This indicates that the text semantic vector representation of the positive sample target item uses... The process of constructing item-level negative sample pairs is as follows: For a given target item sample, other target item samples within the batch during training are used as negative samples of that target item sample, and their text semantic vector representation uses... express; In item-level contrastive learning training tasks, the loss function is defined as: 。 6. The text representation alignment sequence recommendation method based on decoupled contrastive learning according to claim 5, characterized in that, The total loss function during training is as follows: ; in, These are manually defined hyperparameters that represent the weights of each loss function. .

Images