A text embedding joint training method based on policy gradient optimization

Abstract

The application discloses a text embedding joint training method based on policy gradient optimization, comprising: (1) an asymmetric encoder-decoder architecture; (2) a dynamic reward alignment mechanism; (3) an optimization mechanism based on policy gradient; the training process of the method is as follows: 1. initializing a policy model pi θ and a reference model pi ref ; 2. in a batch, for each text pair, a set of outputs {o1, o2,..., o G} is sampled from the current policy model, wherein G is the size of the set, and the size G of the set is 8; 3. the outputs sampled from the decoder are parsed and the reward is calculated; (4) the rewards in the set are normalized, and the advantage function is calculated using the normalized rewards; (5) the policy model is updated by maximizing the objective function; (6) the encoder and the decoder are cross-iterated in the training process.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet text analysis, and particularly relates to a text embedding joint training method based on policy gradient optimization. BACKGROUND

[0002] Text embedding models play a crucial role in natural language processing and machine learning. Text embedding encodes text into a digital representation, mapping words, sentences or documents into a low-dimensional dense continuous vector space, allowing similar texts to have closer vector representations. This representation not only enhances the usability of text data, but also significantly improves the performance of various downstream tasks such as search, question answering, semantic text similarity and item recommendation. In particular, in the retrieval-augmented generation technology, the ability of embedding models directly affects the quality of large model generation results.

[0003] Generating text vectors based on pre-trained language models combined with pooling operations has been the main method for a long time. Then, the selected weakly supervised text pairs are further pre-trained through contrastive learning or directly fine-tuned for downstream tasks using supervised text pairs.

[0004] Although the existing method performs well in benchmark tasks, it still has the following defects:

[0005] 1. The current model performance is highly dependent on human annotation or carefully designed negative example mining strategies. The supervised fine-tuning stage relies on MS MARCO and other annotated datasets, which have high annotation costs and are difficult to cover long-tail domains. Contrastive learning requires the construction of positive and negative sample pairs, and the acquisition of high-quality difficult examples relies on heuristic rules or complex preprocessing procedures, resulting in low training efficiency and high construction cost.

[0006] 2. The text embedding vectors generated under the current contrastive learning framework have the risk of semantic representation distortion, which is manifested as follows: the sentence vectors generated by traditional methods lack bidirectional mapping ability or explicit modeling of semantic equivalence, i.e., they cannot reverse the original text semantics by vector (such as reconstructing the original sentence by a decoder). This indicates that the existing feature vectors may only capture local statistical features of the text, rather than complete semantic structures; since contrastive learning only optimizes the relative distance between samples, it does not explicitly constrain the semantic coverage of the vector space, resulting in the loss of low-frequency semantic patterns, i.e., the model tends to strengthen high-frequency semantic relationships and is difficult to preserve long-tail or fine-grained semantic features. SUMMARY

[0007] To solve the above technical problems, the application adopts the following technical solutions: a text embedding joint training method based on policy gradient optimization, comprising: (1) an asymmetric encoder-decoder architecture; the asymmetric encoder-decoder architecture comprises an encoder and a decoder, wherein the operation steps of the encoder are that input texts generate digitized sequences through word embedding matrices and position encoding; then, global context modeling is performed by a multilayer encoder, and the calculation is formally expressed as: H = Encoder (E tok +E pos ) ;

[0008] Finally, the output of the last layer is averaged and pooled to obtain the vector representation of the text:

[0009] vector text = Pool mean (H) ;

[0010] The operation steps of the decoder are that input instructions generate digitized sequences through word embedding matrices and position encoding; then, global context modeling is performed by a multilayer decoder, and the calculation is formally expressed as: H = Decoder (E tok +E pos ,vector text ) ; the output of the last layer is sent to a linear transformation layer to obtain the probability distribution of each word, and then the text output is decoded;

[0011] (2) a dynamic reward alignment mechanism, the dynamic reward alignment mechanism comprises generating two subtasks, and the decoder completes the two subtasks according to the instructions; the instructions: reconstruct two texts and compare the similarity or correlation between the two texts, and must be output in the format of <build>Reconstructed text< / build> <score>Fraction< / score> ; the score value range is

[0012] [0, 1], with four decimal places;

[0013] (3) an optimization mechanism based on policy gradient, the implementation of the optimization task relies on the dynamic reward mechanism in the reinforcement learning framework, and the feedback reward signal is used to guide the model parameter update direction; group relative policy optimization estimates the baseline through the relative reward in the group to avoid using an additional value function model, thereby significantly reducing the training resource consumption and improving the performance of the model in complex tasks such as inference;

[0014] The training process of the method is as follows:

[0015] (1) initialize the policy model π θ ; and the reference model π ref ;

[0016] (2) in a batch, for each text pair, from the current policy model πref A set of outputs {o1,o2,...,o3} is sampled. G}, where G is the size of the group, and in this invention, the group size G is 8;

[0017] (3) Parse and calculate the reward [r1,r2,...,r] based on the output sampled from the decoder. G Specifically, the following includes:

[0018] Instruction compliance reward: The output contains <build> 、< / build> , <score> 、< / score> For each hit, you are awarded 0.25 points;

[0019] Text Reconstruction Reward: Parsing using regular expressions <build>With< / build> The overlap between the generated text and the target text is calculated using the n-gram method, specifically the BLEU evaluation method.

[0020] Where BP is the length penalty factor, and the formula is as follows:

[0021]

[0022] Where c is the length of the generated text, r is the length of the target text; P n This represents the proportion of n-grams appearing in the generated text compared to the target text, indicating the weight. By default, the weights are equal. N = 4;

[0023] Similarity reward: Parsed from regular expressions <score>With< / score> The similarity between text pairs is calculated based on the content of the tags and the pooled output vector of the encoder. The similarity score generated by the decoder is used as the target, and the deviation between the two is calculated as the reward. The calculation formula is as follows:

[0024] reward similarity =1.0 - abs(score) encoder -score decoder );

[0025] Therefore, the total reward is recorded as:

[0026] reward = reward prompt +reward rebuild +reward similarity ;

[0027] (4) Normalize the rewards within the group, and use the normalized rewards to calculate the advantage function, which is as follows:

[0028]

[0029] (5) Update the policy model by maximizing the objective function, which is as follows:

[0030]

[0031]

[0032] where ε and β are hyperparameters, π θ is the current policy model, π old is the old policy model, π ref is the reference model.

[0033] (6) Cross-iterate the encoder and the decoder during the training process. In the initial stage, freeze the encoder parameters, train the decoder first, freeze the decoder parameters when iterating a certain number of times, and start training the encoder. Switch every 10k steps, and save the encoder as the final text embedding model after the model training is completed.

[0034] Further, each layer of the Encoder includes self-attention, residual + normalization, and feedforward neural network three parts; the calculation is as follows: let E emb = E tok + E pos , the self-attention calculation is as follows:

[0035]

[0036] where W Q , W K and W V represent the query matrix, the key matrix and the value matrix respectively;

[0037] The first residual + normalization calculation is as follows:

[0038]

[0039] The nonlinear transformation is made through the feedforward neural network, and the calculation is as follows:

[0040]

[0041] where W1 and b1 represent the weight and bias of the first full connection layer;

[0042] The second residual + normalization calculation is as follows:

[0043]

[0044] Further, each layer of the Decoder includes masked self-attention, cross-attention, residual

[0045] + Normalization, feedforward neural network four parts; calculation as follows:

[0046] count E emb = E tok + E pos , mask self-attention calculation is:

[0047]

[0048] Where W Q , W K and W V represent the query matrix, key matrix and value matrix Mask causal represent the causal mask matrix.

[0049] The first residual + normalization calculation as follows:

[0050]

[0051] Through the feedforward neural network to do nonlinear transformation, calculation as follows:

[0052]

[0053] Where, W1 and b1 represent the first full connection layer weight and bias;

[0054] The third residual + normalization calculation as follows:

[0055]

[0056] The decoder used for Qwen, without the ability to handle encoding layer features, so the cross attention is added to realize the bridge between the decoder and the encoder.

[0057] Further, according to the task design of three reward mechanism: (1) instruction following reward: the output result contains <build> 、< / build> 、 <score>< / score> , each hit, reward 0.25 points, instruction following reward value range is [0, 1.0];

[0058] (2) text reconstruction reward: input text pair as the decoder text reconstruction task target, through n-gram method to calculate the overlap between the generated text and the target text, specific can adopt BLEU evaluation method:

[0059]

[0060] Where, c is the length of the generated text, r is the length of the target text; P n represent the proportion of n-gram in the generated text appears in the target text, represents the weight, the default weight is the same, namely N=4; the reward value of text reconstruction is in the range of [0, 1.0];

[0061] (3) Similarity reward: according to the pooling output vector of the encoder, the similarity of the text pair is calculated, and the similarity score generated by the decoder is taken as the target, and the deviation between the two is taken as the reward. The calculation formula is as follows:

[0062]

[0063] The value range is [0, 1.0]; after the three rewards are summarized, the value range is [0, 3.0].

[0064] Further, the specific process of the policy gradient-based optimization mechanism includes:

[0065] (1) A group of outputs is sampled for an input sample, and the reward of each output is calculated. The calculation of the reward is according to the reward mechanism [r1, r2, …, r G ] in section two;

[0066] (2) Normalize the rewards in the group, and use the normalized rewards to calculate the advantage function, which is as follows:

[0067]

[0068] (3) Update the policy model by maximizing the objective function, which is as follows:

[0069]

[0070] Where ε and β are hyperparameters, π θ is the current policy model, π old is the old policy model, and π ref is the reference model.

[0071] (4) Iterative training, gradually optimizing the training model.

[0072] Further, in the entire iterative training, the encoder inputs text 1 and text 2. The input of the encoder refers to the instruction encoder using the BERT model, and the twin architecture, that is, two BERT models share the weights, respectively process the text. The specific processing process is as follows: text 1 is converted into a digital Token-Ids by a tokenizer, and then word embedding and position encoding are performed to generate the input sequence of the text Where n is the sequence length, the maximum length is 512, d is the feature dimension, and the value is 1024,

[0073] E emb1 = E tok1 + E pos1 ;

[0074] Subsequently, the self-attention mechanism models the input sequence to capture the dependencies between elements within the sequence; specifically as follows:

[0075] The input vector generates a query vector (Q), a key vector (K) and a value vector (V) through linear transformation:

[0076] Q = E emb1 W Q ,K = E emb1 W K ,V = E emb1 W V ;

[0077] Wherein, The dot product of Q and K is calculated, and after scaling, the attention is obtained by Softmax normalization:

[0078]

[0079] Then the output of the self-attention is processed by residual connection and normalization to alleviate gradient disappearance, accelerate training convergence and stabilize the model learning process; the residual connection is to add the input E emb1 of the self-attention layer directly to its output:

[0080] H res-attention = E emb1 + Attention(Q, K, V);

[0081] The output of the residual connection is layer normalized:

[0082] H norm = LayerNorm(H res-attention );

[0083] Subsequently, the output of the feedforward neural network is nonlinearly transformed to enhance the model expression ability, and the feedforward neural network includes two linear layers and an activation function, and the calculation is as follows:

[0084] H FFN = mac(0, H norm W1 + b1) W2 + b2;

[0085] Wherein, The output of the feedforward neural network is connected to the input by residual connection and normalized:

[0086] H res-ffn = H norm + H FFN ;

[0087] H encoder = LayerNorm(H res-ffn );

[0088] The above is the calculation process of the single-layer Encoder, and the calculation methods of the remaining layers are the same, the difference is that the input is the output of the previous layer Encoder, the BERT 24-layer Encoder adopted by the application is N=24; after N-layer Encoder processing, the output features of the last layer are obtained, that is:

[0089] H1=Encoder(E tok1 +E pos1 );

[0090] Finally, the output of the last layer is averaged and pooled, and a dimension expansion layer is added to match the feature dimension of the decoder, obtaining the vector representation of text 1:

[0091] vector text1 =Pool mean (H1)W expand +b;

[0092] where, d extend =1536;

[0093] The vector representation of text 2 is:

[0094] H2=Encoder(E tok2 +E pos2 );

[0095] vector text2 =Pool mean (H2)W expand +b.

[0096] Further, the vectors of text 1 and text 2 are fused to obtain the feature representation of the text pair, using the concatenation operation, as follows:

[0097] H text =Concat(vector text1 ,vector texrt2 );

[0098] The decoder uses the Qwen model, which uses the pre-training ability and instruction following ability of the Qwen model as a teacher model and encoder for joint training, the specific processing process is as follows: the instructions of the decoder consist of two parts: task instructions + reference data; The format is as follows:

[0099] (1) Reconstruct two texts and compare the similarity or correlation between the two texts, which must be output in the format of <build>Reconstructed text< / build> <score>Fraction< / score>

[0100] ​(2) The score value range is [0, 1], and four decimal places are reserved;

[0101] (3) {text pair};

[0102] Where {text pair} is the text pair data input by the encoder, and in the encoder-only training stage, it needs to be randomly masked with a mask rate of 50%; instructions are converted into digital Token-Ids by the tokenizer of the Qwen model, and then word embedding and position encoding are performed to generate the input sequence of the text Where n is the sequence length, the maximum length is 32768, d is the feature dimension, and the value is 1536,

[0103] E emb-prompt =E tok-prompt +E pos-prompt ;

[0104] The input vector generates the query vector (Q), the key vector (K), and the value vector (V) through linear transformation:

[0105] Q=E emb-prompt W Q ,K=E emb-prompt W K ,V=E emb-prompt W V ;

[0106] Where, d group = 256, then the masked self-attention mechanism models the input sequence to capture the dependency between elements in the sequence; Specifically as follows: the input instruction needs to be masked to prevent the model from "cheating" during training, i.e. using future information, to ensure that the current position prediction only depends on the generated sequence, an upper triangular matrix mask is generated, the element value is negative infinity, so that the current position cannot pay attention to the subsequent position:

[0107]

[0108] Add the mask to the attention mechanism calculation, and then normalize by Softmax to get attention: Since attention uses grouped query attention, after calculating attention, a multi-group feature integration needs to be done, which specifically adds a fully connected layer;

[0109] Attention self =Attention(Q,K,V)W;

[0110] Where, Then, residual processing and normalization are performed on the output of the self-attention layer to alleviate gradient vanishing, accelerate training convergence, and stabilize the model learning process; the residual connection is to process the input E of the self-attention layer. emb-prompt Add it directly to its output:

[0111] H res-attention =E emb-prompt +Attentio self ;

[0112] Perform layer normalization on the output of the residual connection:

[0113] H norm =LayerNorm(H res-attention );

[0114] The cross-attention mechanism models the text-pair features of the encoder's output and the instruction features of the decoder. The calculation process is as follows: the input vector is transformed linearly to generate a query vector (Q), a key vector (K), and a value vector (V):

[0115] Q = H text W Q K = H norm W K V=H norm W V ;

[0116] in, d group =256;

[0117] Cross-attention also employs a multi-query attention mechanism and integrates multiple attention sets.

[0118]

[0119] Attention cross =Attention(Q, K, V)W;

[0120] in, Then, residual processing and normalization are performed on the output of the cross-attention layer, and the input and output of the cross-attention layer are directly added together:

[0121] H res-attention =H norm +Attention cross ;

[0122] Perform layer normalization on the output of the residual connection:

[0123] H norm =LayerNorm(H res-attention );

[0124] Subsequently, the expression ability of the model is enhanced by a nonlinear transformation through a feedforward neural network, and the feedforward neural network comprises three linear layers and a SiLU activation function, and the calculation is as follows:

[0125]

[0126] wherein, d ffn = 8960.

[0127] The output of the feedforward neural network is connected in residual with the input and normalized:

[0128] H res-ffn = H norm + Attention cross ;

[0129] H decoder = Layer Norm(H res-ffn );

[0130] The above is the calculation process of a single layer Decoder, and the calculation methods of the remaining layers are the same, the difference is that the input is the output of the previous layer Decoder, and the 28-layer Decoder of Qwen is adopted in the application, that is, M = 28; after M-layer Decoder processing, the output feature of the last layer is obtained, that is: H = Decoder(E tok-prompt +E pos-prompt ); finally, the output feature is linearly mapped,

[0131] mapped to the vocabulary dimension, and then the output text is decoded.

[0132] Compared with the prior art, the application has the following beneficial effects:

[0133] The application proposes a text embedding joint training method based on policy gradient optimization, aiming to systematically solve the following core technical problems: 1. Reduce the dependence on artificial labeled data and high-quality negative example samples. The traditional contrast learning framework requires the construction of large-scale positive and negative sample pairs, and its training efficiency is limited by the cost of artificial labeling and the complexity of difficult example mining. The application needs to solve how to convert the semantic similarity modeling into a sequence generation task of the decoder through the policy optimization paradigm of reinforcement learning, and only rely on the input of the sample pair to guide the feature space optimization through the reward signal, so as to fundamentally eliminate the collection and labeling demand of explicit negative example samples.

[0134] 2. Improve the bidirectional completeness of semantic representation.

[0135] The text vector generated by the existing single-encoder architecture has one-way semantic information compression loss. The asymmetric encoder-decoder architecture is constructed, the bidirectional mapping constraint of the semantic space is established through the text reconstruction ability of the decoder, and it is ensured that the embedding vector can not only reflect the relative distance between texts, but also completely retain the deep semantic structure features of the original text. BRIEF DESCRIPTION OF DRAWINGS

[0136] Figure 1 A joint training flowchart for text embedding based on policy gradient optimization;

[0137] Figure 2 An asymmetric encoder-decoder architecture;

[0138] Figure 3 A joint training flowchart for an asymmetric encoder-decoder architecture. DETAILED DESCRIPTION

[0139] The present application will be described in detail below in conjunction with the drawings and specific embodiments.

[0140] As shown in the drawings, Figures 1-3 A joint training method for text embedding based on policy gradient optimization, comprising: (1) an asymmetric encoder-decoder architecture; the asymmetric encoder-decoder architecture comprises an encoder and a decoder, wherein the operation steps of the encoder are as follows: input text generates a digitized sequence through word embedding matrix and position encoding; then, global context modeling is performed by a multi-layer encoder, and the calculation is formally expressed as: H = Encoder (E tok +E pos );

[0141] Finally, the output of the last layer is averaged and pooled to obtain the vector representation of the text:

[0142] νector text =Pool mean (H);

[0143] The operation steps of the decoder are as follows: input instructions generate a digitized sequence through a word embedding matrix and position encoding; then, global context modeling is performed by a multi-layer decoder, and the calculation is formally expressed as: H = Decoder (E tok +E pos ,vector text ); the output of the last layer is sent to a linear transformation layer to obtain the probability distribution of each word, and then the text output is decoded;

[0144] (2) a dynamic reward alignment mechanism, the dynamic reward alignment mechanism comprises generating two sub-tasks, and the decoder completes the two sub-tasks according to the instructions; the instructions: reconstruct two texts, and compare the similarity or correlation between the two texts, which must be<build>Reconstructed text< / build> <score>Fraction< / score> The format output is outputted; the score value range is [0, 1], and four decimal places are reserved;

[0145] (3) The optimization mechanism based on the policy gradient, the implementation of the optimization task relies on the dynamic reward mechanism in the reinforcement learning framework, and the feedback reward signal is used to guide the model parameter update direction; the group relative strategy optimization is estimated by using the baseline to avoid using an additional value function model, thereby significantly reducing the training resource consumption and improving the performance of the model in complex tasks such as inference;

[0146] The training process of the method is as follows:

[0147] (1) Initialize the policy model π θ ; and the reference model π ref ;

[0148] (2) In a batch, for each text pair, a group of outputs {o1, o2,..., o G} is sampled from the current policy model π ref , wherein G is the size of the group, and the group size G is 8 in the present application;

[0149] (3) According to the output sampled from the decoder, the output is parsed and the reward [r1, r2,..., r G ] is calculated, specifically as follows, including:

[0150] Instruction compliance reward: the output result contains <build> 、< / build> 、 <score> 、< / score> , and 0.25 points are rewarded for each hit;

[0151] Text reconstruction reward: the content in the <build>With< / build> tag is parsed by using a regular expression, and the overlap degree between the generated text and the target text is calculated by using an n-gram method, and the BLEU evaluation method can be used specifically:

[0152] Wherein, BP is a length penalty factor, and the formula is as follows:

[0153]

[0154] Wherein, c is the length of the generated text, and r is the length of the target text; P n represents the proportion of n-grams in the generated text appearing in the target text, and represents the weight, and the default weight is the same, that is N=4;

[0155] Similarity reward: the content in the <score>With< / score>The content in the label is calculated according to the pooling output vector of the encoder, the similarity of the text pair is calculated, the similarity score generated by the decoder is taken as the target, and the deviation between the two is taken as the reward; the calculation formula is as follows:

[0156] reward similarity =1.0-abs(score encoder -score decoder );

[0157] Therefore, the total reward is recorded as:

[0158] reward=reward prompt +reward rebuild +reward similarity ;

[0159] (4) The intra-group reward is normalized, and the advantage function is calculated using the normalized reward, and the advantage function is as follows:

[0160]

[0161] (5) The policy model is updated by maximizing the objective function, and the objective function is as follows:

[0162]

[0163] Wherein, epsilon and beta are hyperparameters, pi is the current policy model, pi is the old policy model, and pi is the reference model. θ old ref

[0164] (6) Cross iteration of the encoder and the decoder in the training process, in the initial stage, the encoder parameters are frozen, the decoder is trained first, the decoder parameters are frozen when the iteration is a certain number, the encoder is trained, and the switching is done every 10k step. After the model training is completed, the encoder is saved as the final text embedding model.

[0165] In this embodiment, each layer of the encoder includes self-attention, residual+normalization, and feedforward neural network three parts; the calculation is as follows: E emb = E tok +E pos , and the self-attention calculation is as follows:

[0166]

[0167] Wherein, W Q , W K and W V represent query matrix, key matrix and value matrix respectively.

[0168] ​​​The first residual + normalization calculation is as follows:

[0169]

[0170] The nonlinear transformation is calculated by the feedforward neural network as follows:

[0171]

[0172] where W1 and b1 represent the weights and bias of the first fully connected layer;

[0173] The second residual + normalization calculation is as follows:

[0174]

[0175] In this embodiment, each layer of the Decoder includes a masked self-attention, a cross-attention,

[0176] a residual + normalization, and a feedforward neural network four parts; the calculation is as follows:

[0177] count E emb = E tok + E pos , the calculation of the masked self-attention is:

[0178]

[0179] where W Q , W K and W V represent the query matrix, the key matrix and the value matrix Mask causal represents the causal mask matrix;

[0180] The first residual + normalization calculation is as follows:

[0181]

[0182] The nonlinear transformation is calculated by the feedforward neural network as follows:

[0183]

[0184] where W1 and b1 represent the weights and bias of the first fully connected layer;

[0185] The third residual + normalization calculation is as follows:

[0186]

[0187] The decoder used is Qwen, which does not have the ability to process the features of the encoding layer, so the cross-attention is added to realize the bridging of the decoder and the encoder.

[0188] In this embodiment, three reward mechanisms are designed according to the task: (1) Instruction following reward: the output result contains <build> 、< / build> 、 <score>< / score> , each hit, reward 0.25 points, the reward value range of instruction following is [0, 1.0];

[0189] (2) Text reconstruction reward: the input text pair is the target of the decoder text reconstruction task, and the overlap between the generated text and the target text is calculated by the n-gram method, which can use the BLEU evaluation method:

[0190]

[0191] Wherein, c is the length of the generated text, r is the length of the target text; P n represents the proportion of n-gram in the generated text appearing in the target text, and represents the weight, and the default weight is the same, that is The reward value range of text reconstruction is [0, 1.0];

[0192] (3) Similarity reward: according to the pooling output vector of the encoder, the similarity of the text pair is calculated, and the similarity score generated by the decoder is taken as the target, and the deviation between the two is taken as the reward. The calculation formula is as follows:

[0193] reward=1.0-abs(score encoder -score decoder ); The value range is [0, 1.0]; After the three rewards are integrated, the value range is [0, 3.0].

[0194] In this embodiment, the specific process of the optimization mechanism based on policy gradient includes:

[0195] (1) A set of outputs is sampled for an input sample, and the reward of each output is calculated, and the calculation of the reward is according to the reward mechanism [r1, r2, …, r G ] in the second section;

[0196] (2) The rewards in the group are normalized, and the advantage function is calculated using the normalized reward, and the advantage function is as follows:

[0197]

[0198] (3) Update the policy model by maximizing the objective function, and the objective function is as follows:

[0199]

[0200] Wherein, ε and β are hyperparameters, π θ is the current policy model, and πold is the old policy model, π ref is the reference model;

[0201] (4) Iterative training, step by step optimization of the training model.

[0202] In this embodiment, in the whole iterative training, the encoder input text 1, text 2, the input of the encoder refers to instructing the encoder to adopt the BERT model, the twin architecture, that is, two BERT model weights are shared, and the text is processed respectively, and the specific processing process is as follows: text 1 is converted into digital Token-Ids by a tokenizer, and then word embedding and position coding are performed to generate the input sequence of the text Where n is the sequence length, the maximum length is 512, d is the feature dimension, and the value is 1024,

[0203] E emb1 = E tok1 + E pos1 ;

[0204] Subsequently, the self-attention mechanism models the input sequence to capture the dependency between the elements in the sequence; the specific process is as follows:

[0205] The input vector generates a query vector (Q), a key vector (K) and a value vector (V) through linear transformation:

[0206] Q = E emb1 W Q , K = E emb1 W K , V = E emb1 W V ;

[0207] Where, The dot product of Q and K is calculated, and after scaling, the attention is obtained by Softmax normalization:

[0208]

[0209] Then, the output of the self-attention is processed by residual and normalization to alleviate gradient disappearance, accelerate training convergence and stabilize the model learning process; the residual connection is to add the input Eemb1 of the self-attention layer to its output directly:

[0210] H res-attention = E emb1 + Attention(Q, K, V);

[0211] The output of the residual connection is layer normalized:

[0212] H norm = LayerNorm(H res-attention);

[0213] Subsequently, the expression ability of the model is enhanced by nonlinear transformation through a feedforward neural network, which includes two linear layers and an activation function, and is calculated as follows:

[0214] H FFN = max(0, H norm W1+b1)W2+b2;

[0215] wherein, The output of the feedforward neural network is connected in residual connection with the input and normalized:

[0216] H res-ffn = H norm + H FFN ;

[0217] H encoder = LayerNorm(H res-ffn );

[0218] The above is the calculation process of a single layer of the Encoder, and the calculation methods of the remaining layers are the same, the difference being that the input is the output of the previous layer of the Encoder, and the 24-layer Encoder of BERT is used in the present application, i.e., N = 24; after N layers of Encoder processing, the output feature of the last layer is obtained, i.e.:

[0219] H1 = Encoder(E tok1 + E pos1 );

[0220] Finally, the output of the last layer is subjected to mean pooling, and a dimension expansion layer is added to match the feature dimension of the decoder, to obtain the vector representation of text 1:

[0221] vector text1 = Pool mean (H1)W expand + b;

[0222] wherein, d extend = 1536;

[0223] The vector representation of text 2 is as follows:

[0224] H2 = Encoder(E tok2 + E pos2 );

[0225] vector text2 = Pool mean (H2)W expand + b.

[0226] In this embodiment, the vectors of text 1 and text 2 are fused to obtain the feature representation of the text pair. A concatenation operation is used, as follows:

[0227] H text =Concat(vector< ... text1 vector text2 );

[0228] The decoder uses the Qwen model, leveraging its pre-training and instruction-following capabilities as a teacher model for joint training with the encoder. The specific processing flow is as follows: The decoder's instructions consist of two parts: task instructions + reference data; the format is as follows:

[0229] (1) To reconstruct two texts and compare their similarity or correlation, it is necessary to follow the steps outlined above. <build>Reconstructed text< / build> <score>Fraction< / score> Output in the specified format;

[0230] (2) The range of values ​​for the fractions is [0,1], and they are rounded to four decimal places.

[0231] (3) {text pairs};

[0232] Here, {text pairs} represent the text pairs input to the encoder. During the encoder-only training phase, these pairs need to be randomly masked with a masking rate of 50%. The instructions use the Qwen model's tokenizer to convert the text into digitized tokens, and then perform word embedding and positional encoding to generate the input text sequence. Where n is the sequence length, with a maximum length of 32768, and d is the feature dimension, with a value of 1536.

[0233] E emb-prompt =E rok-prompt +E pos-prompt ;

[0234] The input vector is transformed linearly to generate a query vector (Q), a key vector (K), and a value vector (V):

[0235] Q = E emb-prompt W Q K = E emb-prompt W K V = E emb-prompt W V ;

[0236] in, d group= 256, then the mask self-attention mechanism models the input sequence to capture the dependency between elements within the sequence; as follows: the input instruction needs to be masked to prevent the model from "cheating" during training, i.e. using future information, to ensure that the prediction of the current position only depends on the generated sequence, generate an upper triangular matrix mask, the element value is negative infinity, so that the current position cannot pay attention to the subsequent position:

[0237]

[0238] Add the mask to the attention mechanism calculation, and then normalize by Softmax to get attention: Since attention uses grouped query attention, a multi-group feature integration needs to be done after calculating attention; the specific operation is to add a fully connected layer;

[0239] Attention self = Attention (Q, K, V) W;

[0240] Where, Then do residual processing and normalization processing on the output of self-attention to alleviate gradient disappearance, speed up training convergence, and stabilize the model learning process; the residual connection is to add the input E emb-prompt of the self-attention layer directly to the output:

[0241] H norm = E res-attention + Attention text ;

[0242] Layer normalization is performed on the output of the residual connection:

[0243] H Q = LayerNorm (H norm ) ;

[0244] The cross-attention mechanism models the text pair features of the encoder output and the instruction features of the decoder, and the calculation process is as follows: the input vector is transformed by linear transformation to generate the query vector (Q), the key vector (K) and the value vector (V):

[0245] Q = H K W norm , K = H V W group , V = H cross W res-attention ;

[0246] Where, d norm = 256;

[0247] The cross attention also adopts a plurality of query attention mechanisms, and integrates a plurality of attentions,

[0248]

[0249] Attention cross = Attention (Q, K, V) W;

[0250] wherein, Then, the output of the cross attention is subjected to residual processing and normalization processing, the input of the cross attention layer is directly added to the output of the cross attention layer:

[0251] H res-attention = H norm + Attention cross ;

[0252] The output of the residual connection is subjected to layer normalization:

[0253] H norm = LayerNorm (H res-attention ) ;

[0254] Subsequently, a feedforward neural network is used for nonlinear transformation to enhance the expression ability of the model, and the feedforward neural network comprises three linear layers and a SiLU activation function, and the calculation is as follows:

[0255] H FFN = SiLU (H res-norm W gate ) (H res-norm W up ) W down ;

[0256] wherein, d ffn = 8960.

[0257] The output of the feedforward neural network is subjected to residual connection and normalization with the input:

[0258] H res-ffn = H norm + Attention cross ;

[0259] H decoder = LayerNorm (H res-ffn ) ;

[0260] The above is the calculation process of a single layer of the Decoder, and the calculation methods of the remaining layers are the same, the difference is that the input is the output of the previous layer of the Decoder, and the 28-layer Decoder of Qwen is adopted in the application, that is, M = 28; after M layers of Decoder processing, the output features of the last layer are obtained, that is:

[0261] H = Decoder(E tok-prompt + E pos-prompt ); finally, the output features are mapped through a linear mapping,

[0262] to the vocabulary dimension, and the output text is decoded.

[0263] It should also be noted that the terms "comprising," "including," and any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.

[0264] The various embodiments in this application are described in a progressive manner, and the same or similar parts among the various embodiments can be mutually referred to. Each embodiment focuses on the differences from other embodiments. In particular, the system embodiments are described in a relatively simple manner because they are substantially similar to the method embodiments, and the relevant parts can be referred to the description of the method embodiments.

Claims

1. A method for joint training of text embeddings based on policy gradient optimization, characterized in that, Comprise: (1) Asymmetric encoder-decoder architecture; The asymmetric encoder-decoder architecture comprises an encoder and a decoder, wherein the operation steps of the encoder are to generate a digitized sequence for the input text through a word embedding matrix and position encoding; The operation steps of the decoder are to generate a digitized sequence for the input instruction through a word embedding matrix and position encoding; (2) A dynamic reward alignment mechanism, which comprises generating two sub-tasks, and the decoder completes the two sub-tasks according to the instruction; (3) The optimization training process of the policy gradient is as follows:

1. Initialize policy model p θ ; and reference model p ref ; 2. In one batch, for each text pair, sample a set of outputs {o1, o2,..., oG} from the current policy model, where G is the size of the set. G}, where G is the size of the set.

3. Analyze and calculate the reward according to the output sampled from the decoder [r1, r2,..., r G ], in particular as follows, comprising: Instructions follow reward: the output result contains <build> 、< / build> , <score> 、< / score> , 0.25 points for each hit; Text reconstruction reward: parse <build>And< / build> The content in the label, specifically can adopt BLEU evaluation method: Similarity reward: Analysis <score>And< / score> The content in the label, according to the pooling output vector of the encoder, calculates the deviation of the two as a reward; Therefore, the total reward is recorded as: reward = reward prompt + reward rebuild + reward similarity ; (4) Normalize the intra-group reward, and calculate the advantage function using the normalized reward, the advantage function is as follows: (5) Update the policy model by maximizing the objective function, the objective function is as follows: (6) Cross-iterate the encoder and the decoder during the training process, and save the encoder as the final text embedding model after the model training is completed.

2. The method of claim 1, wherein, Each layer of Encoder includes self-attention, residual + normalization, and feed-forward neural network three parts; the calculation is as follows: let E emb = E tok + E pos , the self-attention calculation is: Q = W Q E emb K = W K E emb V = W V E emb ; where W Q , W K , and W V represent the query matrix, key matrix, and value matrix, respectively; The first residual + normalization calculation is as follows: Nonlinear transformation is performed through a feedforward neural network, and the calculation is as follows: Wherein, W1 and b1 represent the weight and bias of the first full connection layer; The second residual + normalization calculation is as follows:

3. The method of claim 1, wherein, Each layer of the Decoder includes a mask self-attention, a cross-attention, a residual + normalization, and a feedforward neural network; the calculation is as follows: Recall E emb = E tok + E pos , the mask self-attention calculation is: Q = W Q E emb K = W K E emb V = W V E emb ; Among them W Q W K and W V Mask represents the query matrix, key matrix, and value matrix, respectively. causal Represents the causal mask matrix; The first residual + normalization calculation is as follows: Nonlinear transformation is performed through a feedforward neural network, and the calculation is as follows: Wherein, W1 and b1 represent the weight and bias of the first full connection layer; The third residual + normalization calculation is as follows: The decoder used is Qwen, which does not have the ability to process the features of the encoding layer, so a cross-attention is added to realize the bridging of the decoder and the encoder.

4. The text embedding joint training method based on policy gradient optimization according to claim 1, characterized in that, According to the task, three reward mechanisms are designed: (1) instruction following reward: the output result contains <build> 、< / build> , <score>< / score> , and each hit is rewarded 0.25 points. The reward value of instruction following is in the range of [0, 1.0]; (2) Text reconstruction reward: The input text pair is the target of the text reconstruction task of the decoder, and the overlap between the generated text and the target text is calculated through the n-gram method, and the BLEU evaluation method can be used: Wherein, BP is a length penalty factor, and the formula is as follows: where c is the length of the generated text, r is the length of the target text; P n represents the proportion of n-gram in the generated text appearing in the target text, and represents the weight, and the default weight is the same, that is The reward value of text reconstruction is in the range of [0, 1.0]; (3) Similarity reward: According to the pooling output vector of the encoder, the similarity of the text pair is calculated, and the similarity score generated by the decoder is taken as the target, and the deviation between the two is taken as the reward; The calculation formula is as follows: reward = 1.0 - abs(score encoder - score decoder ); The value range is [0, 1.0]; after the three rewards are summarized, the value range is [0, 3.0].

5. The method of claim 1, wherein, The specific process of the optimization mechanism based on the policy gradient includes: (1) sample a set of outputs from an input sample and compute a reward for each output, the reward computation following the reward mechanism of Section 2 [r1, r2,..., r G ]; (2) Normalize the intra-group reward, and calculate the advantage function using the normalized reward, the advantage function is as follows: (3) Update the policy model by maximizing the objective function, the objective function is as follows: where ε and β are hyperparameters, π θ is the current policy model, π old is the old policy model, π ref is the reference model; (4) Iterative training, gradually optimize the training model.

6. The text embedding joint training method based on policy gradient optimization according to claim 5, characterized in that, In the whole iterative training, the encoder input text 1, text 2, the input of the encoder refers to instructing the encoder to adopt the BERT model, a twin architecture, that is, two BERT model weights are shared, and the text is processed respectively, and the specific processing process is as follows: text 1 is converted into digital Token-Ids by a tokenizer, and then word embedding and position coding are performed to generate the input sequence of the text Wherein n is the sequence length, the maximum length is 512, d is the feature dimension, and the value is 1024, E emb1 = E tok1 + E pos1 ; Subsequently, the self-attention mechanism models the input sequence to capture the dependency between the elements in the sequence; the specific is as follows: The input vector generates a query vector (Q), a key vector (K), and a value vector (V) through linear transformation: Q = E emb1 W Q K = E emb1 W K V = E emb1 W V ; where, The dot product of Q and K is computed, scaled and normalized by Softmax to get the attention: Then the output of self-attention is processed with residual connection and normalization to alleviate gradient vanishing, accelerate training convergence and stabilize the model learning process. The residual connection is to add the output of the self-attention layer directly to the input E emb1 to the input E H res-attention = E emb1 + Attention(Q, K, V); Layer normalization is performed on the output of the residual connection: H norm = Layer Norm(H res-attention ); Subsequently, nonlinear transformation is performed through a feedforward neural network to enhance the expression ability of the model, and the feedforward neural network includes two linear layers and an activation function, and the calculation is as follows: H FFN = max(0, H norm W1+b1)W2+b2; wherein, The output of the feedforward neural network is connected in residual connection with the input and normalized: H res-ffn = H norm + H FFN ; H encoder = LayerNorm(H res-ffn ); The above is the calculation process of the single-layer Encoder, and the calculation methods of the remaining layers are the same, the difference is that the input is the output of the previous layer Encoder, the 24-layer Encoder of BERT is used in the application, that is, N = 24; after N-layer Encoder processing, the output features of the last layer are obtained, that is: H1 = Encoder(E tok1 + E pos1 ); Finally, the output of the last layer is averaged and pooled, and a dimension expansion layer is added to match the feature dimension of the decoder, obtaining the vector representation of text 1: vector text1 = Pool mean (H1)W expand +b; wherein, d extend = 1536; The vector representation of text 2 is shown in the following table: H2 = Encoder(E tok2 + E pos2 ); vector text2 = Pool mean (H2)W expand +b.

7. The method of claim 6, wherein the method further comprises: The vector of text 1 and the vector of text 2 are fused to obtain the feature representation of the text pair, and the application adopts splicing operation, and the specific process is as follows: H text = Concat(vector text1 ,vector text2 ) The decoder uses the Qwen model, which uses the pre-training ability and instruction following ability of the Qwen model as a teacher model and encoder for joint training, and the specific processing process is as follows: the instructions of the decoder consist of two parts: task instructions + reference data; The format is as follows: (1) Reconstruct two texts and compare the degree of similarity or correlation between the two texts, which must be output in the format of <build>Reconstructed text< / build> <score>Score< / score> ; (2) The score value range is [0, 1], and the decimal part is retained to four decimal places; (3) {Text pair}; Where {text pair} is the text pair data input of the encoder, in the encoder-only training stage, it needs to be randomly masked, and the masking rate is 50%; the instruction is converted into digital Token-Ids by the tokenizer of the Qwen model, and then word embedding and position encoding are performed to generate the input sequence of the text Where n is the sequence length, the maximum length is 32768, and d is the feature dimension, which is 1536. E emb-prompt = E tok-prompt + E pos-prompt ; The input vector generates a query vector (Q), a key vector (K), and a value vector (V) through linear transformation: Q = E emb-prompt W Q , K = E emb-prompt W K , V = E emb-prompt W V ; wherein, d group = 256, then, the mask self-attention mechanism models the input sequence to capture the dependency between the elements inside the sequence; as follows: it is necessary to mask the input instruction to prevent the model from "cheating" during training, that is, using future information, to ensure that the prediction of the current position only depends on the generated sequence, and generate an upper triangular matrix mask, the element value is negative infinity, so that the current position cannot pay attention to the subsequent position: Add the mask to the calculation of the attention mechanism, and then normalize it by Softmax to get attention: Since the attention uses grouped query attention, a multi-group feature integration needs to be done after the attention is calculated. The specific operation is to add a fully connected layer. Attention self = Attention(Q, K, V) W; wherein, Then, the output of the self-attention is subjected to residual processing and normalization processing, so as to alleviate gradient disappearance, accelerate training convergence, and stabilize the model learning process. The residual connection is to directly add the input E of the self-attention layer and the output thereof. emb-prompt . H res-attention = E emb-prompt + Attention self ; The output of the residual connection is normalized by layer normalization: H norm = LayerNorm(H res-attention ); The cross-attention mechanism models the text pair features of the encoder output and the instruction features of the decoder, and the calculation process is as follows: the input vector generates a query vector (Q), a key vector (K), and a value vector (V) through linear transformation: Q = H text W Q , K = H norm W K , V = H norm W V ; wherein, d group = 256; Cross-attention also uses multiple sets of query attention mechanisms, and integrates multiple attention mechanisms, Attention cross = Attention(Q, K, V) W; wherein, The output of the cross-attention is then residual and normalized, and the input to the cross-attention layer is added directly to its output: H res-attention = H norm + Attention cross ; The output of the residual connection is normalized by layer normalization: H norm = LayerNorm(H res-attention ); Then, a feedforward neural network is used for nonlinear transformation to enhance the model expression ability, and the feedforward neural network includes three linear layers and SiLU activation functions, and the calculation is as follows: H FFN = SiLU(H res-norm W gate )(H res-norm W up )W down ; wherein, d ffn = 8960; The output of the feedforward neural network is connected to the input through a residual connection and normalized: H res-ffn = H norm + Attention cross ; H decoder = LayerNorm(H res-ffn ); The above is the calculation process of the single-layer Decoder, and the calculation methods of the remaining layers are the same, the difference is that the input is the output of the previous layer Decoder, the 28-layer Decoder of Qwen adopted by the application is M=28; after M-layer Decoder processing, the output feature of the last layer is obtained, that is: H=Decoder(E tok-prompt +E pos-prompt ); finally, the output feature is mapped to the vocabulary dimension through linear mapping, and then the output text is decoded.

Images