Text embedding joint training method based on strategy gradient optimization

Abstract

The invention discloses a text embedding joint training method based on strategy gradient optimization. The text embedding joint training method comprises the following steps: (1) constructing an asymmetric encoder-decoder; (2) a dynamic reward alignment mechanism; (3) an optimization mechanism based on strategy gradient; the training process of the method comprises the following steps: 1, initializing a strategy model pi theta; and a reference model [pi] ref; 2, in a batch, for each text pair, sampling a group of outputs {o1, o2,..., oG} from the current policy model, where G is the size of the group, and the size G of the group is 8; 3, analyzing according to the output sampled from the decoder and calculating the reward; (4) carrying out normalization processing on the rewards in the group, and calculating a dominant function by using the normalized rewards; (5) updating the strategy model by maximizing the objective function; and (6) performing cross iteration on the encoder and the decoder in the training process.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet text analysis, and particularly relates to a text embedding joint training method based on policy gradient optimization. BACKGROUND

[0002] Text embedding models play a crucial role in natural language processing and machine learning. Text embedding encodes text into a digital representation, mapping words, sentences or documents into a low-dimensional dense continuous vector space, allowing similar texts to have closer vector representations. This representation not only enhances the usability of text data, but also significantly improves the performance of various downstream tasks such as search, question answering, semantic text similarity and item recommendation. In particular, in the retrieval-augmented generation technology, the ability of embedding models directly affects the quality of large model generation results.

[0003] Generating text vectors based on pre-trained language models combined with pooling operations has been the main method for a long time. Then, the selected weakly supervised text pairs are further pre-trained through contrastive learning or directly fine-tuned for downstream tasks using supervised text pairs.

[0004] Although the existing method performs well in benchmark tasks, it still has the following defects:

[0005] 1. The current model performance is highly dependent on human annotation or carefully designed negative example mining strategies. The supervised fine-tuning stage relies on MS MARCO and other annotated datasets, which have high annotation costs and are difficult to cover long-tail domains. Contrastive learning requires the construction of positive and negative sample pairs, and the acquisition of high-quality difficult examples relies on heuristic rules or complex preprocessing procedures, resulting in low training efficiency and high construction cost.

[0006] 2. The text embedding vectors generated under the current contrastive learning framework have the risk of semantic representation distortion, which is manifested as follows: the sentence vectors generated by traditional methods lack bidirectional mapping ability or explicit modeling of semantic equivalence, i.e., they cannot reverse the original text semantics by vector (such as reconstructing the original sentence by a decoder). This indicates that the existing feature vectors may only capture local statistical features of the text, rather than complete semantic structures; since contrastive learning only optimizes the relative distance between samples, it does not explicitly constrain the semantic coverage of the vector space, resulting in the loss of low-frequency semantic patterns, i.e., the model tends to strengthen high-frequency semantic relationships and is difficult to preserve long-tail or fine-grained semantic features. SUMMARY

[0007] To solve the above technical problems, the application adopts the following technical solutions: a text embedding joint training method based on policy gradient optimization, comprising: (1) an asymmetric encoder-decoder architecture; the asymmetric encoder-decoder architecture comprises an encoder and a decoder, wherein the operation steps of the encoder are that input texts generate digitized sequences through word embedding matrices and position encoding; then, global context modeling is performed by a multilayer encoder, and the calculation is formally expressed as: H = Encoder (E tok +E pos ) ;

[0008] Finally, the output of the last layer is averaged and pooled to obtain the vector representation of the text:

[0009] vector text = Pool mean (H) ;

[0010] The operation steps of the decoder are that input instructions generate digitized sequences through word embedding matrices and position encoding; then, global context modeling is performed by a multilayer decoder, and the calculation is formally expressed as: H = Decoder (E tok +E pos ,vector text ) ; the output of the last layer is sent to a linear transformation layer to obtain the probability distribution of each word, and then the text output is decoded;

[0011] (2) a dynamic reward alignment mechanism, the dynamic reward alignment mechanism comprises generating two subtasks, and the decoder completes the two subtasks according to the instructions; the instructions: reconstruct two texts and compare the similarity or correlation between the two texts, and must be output in the format of <build>Reconstructed text< / build> <score>Fraction< / score> ; the score value range is

[0012] [0, 1], with four decimal places;

[0013] (3) an optimization mechanism based on policy gradient, the implementation of the optimization task relies on the dynamic reward mechanism in the reinforcement learning framework, and the feedback reward signal is used to guide the model parameter update direction; group relative policy optimization estimates the baseline through the relative reward in the group to avoid using an additional value function model, thereby significantly reducing the training resource consumption and improving the performance of the model in complex tasks such as inference;

[0014] The training process of the method is as follows:

[0015] (1) initialize the policy model π θ ; and the reference model π ref ;

[0016] (2) in a batch, for each text pair, from the current policy model πref A set of outputs {o1,o2,...,o G}, where G is the size of the group, and in the present invention, the group size G is 8;

[0017] (3) Parse and calculate the reward [r1, r2, ..., r G ], specifically as follows, including:

[0018] Instruction following reward: The output contains <build> 、< / build> 、 <score> 、< / score> , each hit will be rewarded with 0.25 points;

[0019] Text reconstruction bonus: parse out the text using regular expressions <build>With< / build> The content in the tag is calculated by the n-gram method to calculate the overlap between the generated text and the target text. Specifically, the BLEU evaluation method can be used:

[0020] Among them, BP is the length penalty factor, the formula is as follows:

[0021]

[0022] Among them, c is the length of the generated text, r is the length of the target text; P n Indicates the proportion of n-grams in the generated text that appear in the target text, which represents the weight. The default weight is the same, that is, N = 4;

[0023] Similarity reward: parse through regular expression <score>With< / score> The content in the label is calculated based on the pooled output vector of the encoder to calculate the similarity of the text pair. The similarity score generated by the decoder is used as the target, and the deviation between the two is calculated as the reward. The calculation formula is as follows:

[0024] reward similarity =1.0-abs(score encoder -score decoder );

[0025] So the total reward is:

[0026] reward=reward prompt +reward rebuild +reward similarity ;

[0027] (4) Normalize the rewards within the group and use the normalized rewards to calculate the advantage function. The advantage function is as follows:

[0028]

[0029] (5) Update the policy model by maximizing the objective function, which is as follows:

[0030]

[0031]

[0032] where ε and β are hyperparameters, π θ is the current policy model, π old is the old policy model, π ref is the reference model.

[0033] (6) Cross-iterate the encoder and the decoder during the training process. In the initial stage, freeze the encoder parameters, train the decoder first, freeze the decoder parameters when iterating a certain number of times, and start training the encoder. Switch every 10k steps, and save the encoder as the final text embedding model after the model training is completed.

[0034] Further, each layer of the Encoder includes self-attention, residual + normalization, and feedforward neural network three parts; the calculation is as follows: let E emb = E tok + E pos , the self-attention calculation is as follows:

[0035]

[0036] where W Q , W K and W V represent the query matrix, the key matrix and the value matrix respectively;

[0037] The first residual + normalization calculation is as follows:

[0038]

[0039] The nonlinear transformation is made through the feedforward neural network, and the calculation is as follows:

[0040]

[0041] where W1 and b1 represent the weight and bias of the first full connection layer;

[0042] The second residual + normalization calculation is as follows:

[0043]

[0044] Further, each layer of the Decoder includes masked self-attention, cross-attention, residual

[0045] + Normalization, feedforward neural network four parts; calculation as follows:

[0046] count E emb = E tok + E pos , mask self-attention calculation is:

[0047]

[0048] Where W Q , W K and W V represent the query matrix, key matrix and value matrix Mask causal represent the causal mask matrix.

[0049] The first residual + normalization calculation as follows:

[0050]

[0051] Through the feedforward neural network to do nonlinear transformation, calculation as follows:

[0052]

[0053] Where, W1 and b1 represent the first full connection layer weight and bias;

[0054] The third residual + normalization calculation as follows:

[0055]

[0056] The decoder used for Qwen, without the ability to handle encoding layer features, so the cross attention is added to realize the bridge between the decoder and the encoder.

[0057] Further, according to the task design of three reward mechanism: (1) instruction following reward: the output result contains <build> 、< / build> 、 <score>< / score> , each hit, reward 0.25 points, instruction following reward value range is [0, 1.0];

[0058] (2) text reconstruction reward: input text pair as the decoder text reconstruction task target, through n-gram method to calculate the overlap between the generated text and the target text, specific can adopt BLEU evaluation method:

[0059]

[0060] Where, c is the length of the generated text, r is the length of the target text; P n represent the proportion of n-gram in the generated text in the target text, represent the weight, the default weight is the same, namely N=4; the reward value of text reconstruction is in the range of [0, 1.0];

[0061] (3) Similarity reward: according to the pooling output vector of the encoder, the similarity of the text pair is calculated, and the similarity score generated by the decoder is taken as the target, and the deviation between the two is taken as the reward. The calculation formula is as follows:

[0062]

[0063] The value range is [0, 1.0]; after the three rewards are summarized, the value range is [0, 3.0].

[0064] Further, the specific process of the policy gradient-based optimization mechanism includes:

[0065] (1) A group of outputs is sampled for an input sample, and the reward of each output is calculated. The calculation of the reward is according to the reward mechanism [r1, r2, …, r G ] in section two;

[0066] (2) Normalize the rewards in the group, and use the normalized rewards to calculate the advantage function, which is as follows:

[0067]

[0068] (3) Update the policy model by maximizing the objective function, which is as follows:

[0069]

[0070] Where ε and β are hyperparameters, π θ is the current policy model, π old is the old policy model, and π ref is the reference model.

[0071] (4) Iterative training, gradually optimizing the training model.

[0072] Further, in the entire iterative training, the encoder inputs text 1 and text 2. The input of the encoder refers to the instruction encoder using the BERT model, and the twin architecture, that is, two BERT models share the weights, respectively process the text. The specific processing process is as follows: text 1 is converted into a digital Token-Ids by a tokenizer, and then word embedding and position encoding are performed to generate the input sequence of the text Where n is the sequence length, the maximum length is 512, d is the feature dimension, and the value is 1024,

[0073] E emb1 = E tok1 + E pos1 ;

[0074] The self-attention mechanism then models the input sequence to capture the dependencies between elements within the sequence; specifically:

[0075] The input vector is linearly transformed to generate the query vector (Q), key vector (K), and value vector (V):

[0076] Q=E emb1 W Q ,K=E emb1 W K ,V=E emb1 W V ;

[0077] in, Calculate the dot product of Q and K, scale it, and normalize it through Softmax to get the attention:

[0078]

[0079] Then the output of the self-attention layer is processed with residual and normalized to alleviate the gradient disappearance, accelerate the training convergence, and stabilize the model learning process; the residual connection is to connect the input E emb1 Add directly to its output:

[0080] H res-attention =E emb1 +Attention(Q,K,V);

[0081] Perform layer normalization on the output of the residual connection:

[0082] H norm =LayerNorm(H res-attention );

[0083] Then, a nonlinear transformation is performed through a feedforward neural network to enhance the model's expressiveness. The feedforward neural network consists of two linear layers and an activation function, which is calculated as follows:

[0084] H FFN =mac(0,H norm W1+b1)W2+b2;

[0085] in, The output of the feedforward neural network is connected to the input with residuals and normalized:

[0086] H res-ffn =H norm +H FFN ;

[0087] H encoder =LayerNorm(H res-ffn );

[0088] The above is the calculation process of the single-layer Encoder, and the calculation methods of the remaining layers are the same, the difference is that the input is the output of the previous layer Encoder, the BERT 24-layer Encoder adopted by the application is N=24; after N-layer Encoder processing, the output features of the last layer are obtained, that is:

[0089] H1=Encoder(E tok1 +E pos1 );

[0090] Finally, the output of the last layer is averaged and pooled, and a dimension expansion layer is added to match the feature dimension of the decoder, obtaining the vector representation of text 1:

[0091] vector text1 =Pool mean (H1)W expand +b;

[0092] where, d extend =1536;

[0093] The vector representation of text 2 is:

[0094] H2=Encoder(E tok2 +E pos2 );

[0095] vector text2 =Pool mean (H2)W expand +b.

[0096] Further, the vectors of text 1 and text 2 are fused to obtain the feature representation of the text pair, using the concatenation operation, as follows:

[0097] H text =Concat(vector text1 ,vector texrt2 );

[0098] The decoder uses the Qwen model, which uses the pre-training ability and instruction following ability of the Qwen model as a teacher model and encoder for joint training, the specific processing process is as follows: the instructions of the decoder consist of two parts: task instructions + reference data; The format is as follows:

[0099] (1) Reconstruct two texts and compare the similarity or correlation between the two texts, which must be output in the format of <build>Reconstructed text< / build> <score>Fraction< / score>

[0100] ​(2) The score value range is [0, 1], and four decimal places are reserved;

[0101] (3) {text pair};

[0102] Where {text pair} is the text pair data input by the encoder, and in the encoder-only training stage, it needs to be randomly masked with a mask rate of 50%; instructions are converted into digital Token-Ids by the tokenizer of the Qwen model, and then word embedding and position encoding are performed to generate the input sequence of the text Where n is the sequence length, the maximum length is 32768, d is the feature dimension, and the value is 1536,

[0103] E emb-prompt =E tok-prompt +E pos-prompt ;

[0104] The input vector generates the query vector (Q), the key vector (K), and the value vector (V) through linear transformation:

[0105] Q=E emb-prompt W Q ,K=E emb-prompt W K ,V=E emb-prompt W V ;

[0106] Where, d group = 256, then the masked self-attention mechanism models the input sequence to capture the dependency between elements in the sequence; Specifically as follows: the input instruction needs to be masked to prevent the model from "cheating" during training, i.e. using future information, to ensure that the current position prediction only depends on the generated sequence, an upper triangular matrix mask is generated, the element value is negative infinity, so that the current position cannot pay attention to the subsequent position:

[0107]

[0108] Add the mask to the attention mechanism calculation, and then normalize by Softmax to get attention: Since attention uses grouped query attention, after calculating attention, a multi-group feature integration needs to be done, which specifically adds a fully connected layer;

[0109] Attention self =Attention(Q,K,V)W;

[0110] Where, Then the output of the self-attention layer is processed with residual and normalized to alleviate the gradient disappearance, accelerate the training convergence, and stabilize the model learning process; the residual connection is to connect the input E emb-prompt , and add directly to its output:

[0111] H res-attention =E emb-prompt +Attentio self ;

[0112] Perform layer normalization on the output of the residual connection:

[0113] H norm =LayerNorm(H res-attention );

[0114] The cross-attention mechanism models the text pair features of the encoder's output and the instruction features of the decoder. The calculation process is as follows: the input vector is linearly transformed to generate the query vector (Q), key vector (K) and value vector (V):

[0115] Q=H text W Q ,K=H norm W K ,V=H norm W V ;

[0116] in, d group =256;

[0117] Cross attention also uses multiple query attention mechanisms and integrates multiple attention groups.

[0118]

[0119] Attention cross =Attention(Q,K,V)W;

[0120] in, Then the output of the cross attention is processed with residuals and normalized, and the input of the cross attention layer is directly added to its output:

[0121] H res-attention =H norm +Attention cross ;

[0122] Perform layer normalization on the output of the residual connection:

[0123] H norm =LayerNorm(H res-attention );

[0124] Subsequently, the expression ability of the model is enhanced by a nonlinear transformation through a feedforward neural network, and the feedforward neural network comprises three linear layers and a SiLU activation function, and the calculation is as follows:

[0125]

[0126] wherein, d ffn = 8960.

[0127] The output of the feedforward neural network is connected in residual with the input and normalized:

[0128] H res-ffn = H norm + Attention cross ;

[0129] H decoder = Layer Norm(H res-ffn );

[0130] The above is the calculation process of a single layer Decoder, and the calculation methods of the remaining layers are the same, the difference is that the input is the output of the previous layer Decoder, and the 28-layer Decoder of Qwen is adopted in the application, that is, M = 28; after M-layer Decoder processing, the output feature of the last layer is obtained, that is: H = Decoder(E tok-prompt +E pos-prompt ); finally, the output feature is linearly mapped,

[0131] mapped to the vocabulary dimension, and then the output text is decoded.

[0132] Compared with the prior art, the application has the following beneficial effects:

[0133] The application proposes a text embedding joint training method based on policy gradient optimization, aiming to systematically solve the following core technical problems: 1. Reduce the dependence on artificial labeled data and high-quality negative example samples. The traditional contrast learning framework requires the construction of large-scale positive and negative sample pairs, and its training efficiency is limited by the cost of artificial labeling and the complexity of difficult example mining. The application needs to solve how to convert the semantic similarity modeling into a sequence generation task of the decoder through the policy optimization paradigm of reinforcement learning, and only rely on the input of the sample pair to guide the feature space optimization through the reward signal, so as to fundamentally eliminate the collection and labeling demand of explicit negative example samples.

[0134] 2. Improve the bidirectional completeness of semantic representation.

[0135] The text vector generated by the existing single-encoder architecture has one-way semantic information compression loss. The asymmetric encoder-decoder architecture is constructed, the bidirectional mapping constraint of the semantic space is established through the text reconstruction ability of the decoder, and it is ensured that the embedding vector can not only reflect the relative distance between texts, but also completely retain the deep semantic structure features of the original text. BRIEF DESCRIPTION OF DRAWINGS

[0136] Figure 1 A joint training flowchart for text embedding based on policy gradient optimization;

[0137] Figure 2 An asymmetric encoder-decoder architecture;

[0138] Figure 3 A joint training flowchart for an asymmetric encoder-decoder architecture. DETAILED DESCRIPTION

[0139] The present application will be described in detail below in conjunction with the drawings and specific embodiments.

[0140] As shown in the drawings, Figures 1-3 A joint training method for text embedding based on policy gradient optimization, comprising: (1) an asymmetric encoder-decoder architecture; the asymmetric encoder-decoder architecture comprises an encoder and a decoder, wherein the operation steps of the encoder are as follows: input text generates a digitized sequence through word embedding matrix and position encoding; then, global context modeling is performed by a multi-layer encoder, and the calculation is formally expressed as: H = Encoder (E tok +E pos );

[0141] Finally, the output of the last layer is averaged and pooled to obtain the vector representation of the text:

[0142] νector text =Pool mean (H);

[0143] The operation steps of the decoder are as follows: input instructions generate a digitized sequence through a word embedding matrix and position encoding; then, global context modeling is performed by a multi-layer decoder, and the calculation is formally expressed as: H = Decoder (E tok +E pos ,vector text ); the output of the last layer is sent to a linear transformation layer to obtain the probability distribution of each word, and then the text output is decoded;

[0144] (2) a dynamic reward alignment mechanism, the dynamic reward alignment mechanism comprises generating two sub-tasks, and the decoder completes the two sub-tasks according to the instructions; the instructions: reconstruct two texts, and compare the similarity or correlation between the two texts, which must be<build>Reconstructed text< / build> <score>Fraction< / score> The format output is outputted; the score value range is [0, 1], and four decimal places are reserved;

[0145] (3) The optimization mechanism based on the policy gradient, the implementation of the optimization task relies on the dynamic reward mechanism in the reinforcement learning framework, and the feedback reward signal is used to guide the model parameter update direction; the group relative strategy optimization is estimated by using the baseline to avoid using an additional value function model, thereby significantly reducing the training resource consumption and improving the performance of the model in complex tasks such as inference;

[0146] The training process of the method is as follows:

[0147] (1) Initialize the policy model π θ ; and the reference model π ref ;

[0148] (2) In a batch, for each text pair, a group of outputs {o1, o2,..., o G} is sampled from the current policy model π ref , wherein G is the size of the group, and the group size G is 8 in the present application;

[0149] (3) The output sampled from the decoder is parsed and the reward [r1, r2,..., r G ] is calculated, specifically as follows, including:

[0150] Instruction compliance reward: the output result contains <build> 、< / build> 、 <score> 、< / score> , and 0.25 points are rewarded for each hit;

[0151] Text reconstruction reward: the content in the <build>With< / build> tag is parsed by using a regular expression, and the overlap degree between the generated text and the target text is calculated by using an n-gram method, and the BLEU evaluation method can be used:

[0152] Wherein, BP is a length penalty factor, and the formula is as follows:

[0153]

[0154] Wherein, c is the length of the generated text, r is the length of the target text; P n represents the proportion of n-gram in the generated text appearing in the target text, and represents the weight, and the default weight is the same, that is N=4;

[0155] Similarity reward: the content in the <score>With< / score>The content in the label is calculated according to the pooling output vector of the encoder, the similarity of the text pair is calculated, the similarity score generated by the decoder is taken as the target, and the deviation between the two is taken as the reward; the calculation formula is as follows:

[0156] reward similarity =1.0-abs(score encoder -score decoder );

[0157] Therefore, the total reward is recorded as:

[0158] reward=reward prompt +reward rebuild +reward similarity ;

[0159] (4) The intra-group reward is normalized, and the advantage function is calculated using the normalized reward, and the advantage function is as follows:

[0160]

[0161] (5) The policy model is updated by maximizing the objective function, and the objective function is as follows:

[0162]

[0163] Wherein, epsilon and beta are hyperparameters, pi is the current policy model, pi is the old policy model, and pi is the reference model. θ old ref

[0164] (6) Cross iteration of the encoder and the decoder in the training process, in the initial stage, the encoder parameters are frozen, the decoder is trained first, the decoder parameters are frozen when the iteration is a certain number, the encoder is trained, and the switching is done every 10k step. After the model training is completed, the encoder is saved as the final text embedding model.

[0165] In this embodiment, each layer of the encoder includes self-attention, residual+normalization, and feedforward neural network three parts; the calculation is as follows: E emb = E tok +E pos , and the self-attention calculation is as follows:

[0166]

[0167] Wherein, W Q , W K and W V represent query matrix, key matrix and value matrix respectively.

[0168] ​​​The first residual + normalization calculation is as follows:

[0169]

[0170] The nonlinear transformation is calculated by the feedforward neural network as follows:

[0171]

[0172] where W1 and b1 represent the weights and bias of the first fully connected layer;

[0173] The second residual + normalization calculation is as follows:

[0174]

[0175] In this embodiment, each layer of the Decoder includes a masked self-attention, a cross-attention,

[0176] a residual + normalization, and a feedforward neural network four parts; the calculation is as follows:

[0177] count E emb = E tok + E pos , the calculation of the masked self-attention is:

[0178]

[0179] where W Q , W K and W V represent the query matrix, the key matrix and the value matrix Mask causal represents the causal mask matrix;

[0180] The first residual + normalization calculation is as follows:

[0181]

[0182] The nonlinear transformation is calculated by the feedforward neural network as follows:

[0183]

[0184] where W1 and b1 represent the weights and bias of the first fully connected layer;

[0185] The third residual + normalization calculation is as follows:

[0186]

[0187] The decoder used is Qwen, which does not have the ability to process the features of the encoding layer, so the cross-attention is added to realize the bridging of the decoder and the encoder.

[0188] In this embodiment, three reward mechanisms are designed according to the task: (1) Instruction following reward: the output result contains <build> 、< / build> 、 <score>< / score> , each hit, reward 0.25 points, the reward value range of instruction following is [0, 1.0];

[0189] (2) Text reconstruction reward: the input text pair is the target of the decoder text reconstruction task, and the overlap between the generated text and the target text is calculated by the n-gram method, which can use the BLEU evaluation method:

[0190]

[0191] Wherein, c is the length of the generated text, r is the length of the target text; P n represents the proportion of n-gram in the generated text appearing in the target text, and represents the weight, and the default weight is the same, that is The reward value range of text reconstruction is [0, 1.0];

[0192] (3) Similarity reward: according to the pooling output vector of the encoder, the similarity of the text pair is calculated, and the similarity score generated by the decoder is taken as the target, and the deviation between the two is taken as the reward. The calculation formula is as follows:

[0193] reward=1.0-abs(score encoder -score decoder ); The value range is [0, 1.0]; After the three rewards are integrated, the value range is [0, 3.0].

[0194] In this embodiment, the specific process of the optimization mechanism based on policy gradient includes:

[0195] (1) A set of outputs is sampled for an input sample, and the reward of each output is calculated, and the calculation of the reward is according to the reward mechanism [r1, r2, …, r G ] in the second section;

[0196] (2) The rewards in the group are normalized, and the advantage function is calculated using the normalized reward, and the advantage function is as follows:

[0197]

[0198] (3) Update the policy model by maximizing the objective function, and the objective function is as follows:

[0199]

[0200] Wherein, ε and β are hyperparameters, π θ is the current policy model, and πold is the old policy model, π ref is the reference model;

[0201] (4) Iterative training, step by step optimization of the training model.

[0202] In this embodiment, in the whole iterative training, the encoder input text 1, text 2, the input of the encoder refers to instructing the encoder to adopt the BERT model, the twin architecture, that is, two BERT model weights are shared, and the text is processed respectively, and the specific processing process is as follows: text 1 is converted into digital Token-Ids by a tokenizer, and then word embedding and position coding are performed to generate the input sequence of the text Where n is the sequence length, the maximum length is 512, d is the feature dimension, and the value is 1024,

[0203] E emb1 = E tok1 + E pos1 ;

[0204] Subsequently, the self-attention mechanism models the input sequence to capture the dependency between the elements in the sequence; the specific process is as follows:

[0205] The input vector generates a query vector (Q), a key vector (K) and a value vector (V) through linear transformation:

[0206] Q = E emb1 W Q , K = E emb1 W K , V = E emb1 W V ;

[0207] Where, The dot product of Q and K is calculated, and after scaling, the attention is obtained by Softmax normalization:

[0208]

[0209] Then, the output of the self-attention is processed by residual and normalization to alleviate gradient disappearance, accelerate training convergence and stabilize the model learning process; the residual connection is to add the input Eemb1 of the self-attention layer to its output directly:

[0210] H res-attention = E emb1 + Attention(Q, K, V);

[0211] The output of the residual connection is layer normalized:

[0212] H norm = LayerNorm(H res-attention);

[0213] Subsequently, the expression ability of the model is enhanced by nonlinear transformation through a feedforward neural network, which includes two linear layers and an activation function, and is calculated as follows:

[0214] H FFN = max(0, H norm W1+b1)W2+b2;

[0215] wherein, The output of the feedforward neural network is connected in residual connection with the input and normalized:

[0216] H res-ffn = H norm + H FFN ;

[0217] H encoder = LayerNorm(H res-ffn );

[0218] The above is the calculation process of a single layer of the Encoder, and the calculation methods of the remaining layers are the same, the difference being that the input is the output of the previous layer of the Encoder, and the 24-layer Encoder of BERT is used in the present application, i.e., N = 24; after N layers of Encoder processing, the output feature of the last layer is obtained, i.e.:

[0219] H1 = Encoder(E tok1 + E pos1 );

[0220] Finally, the output of the last layer is subjected to mean pooling, and a dimension expansion layer is added to match the feature dimension of the decoder, to obtain the vector representation of text 1:

[0221] vector text1 = Pool mean (H1)W expand + b;

[0222] wherein, d extend = 1536;

[0223] The vector representation of text 2 is as follows:

[0224] H2 = Encoder(E tok2 + E pos2 );

[0225] vector text2 = Pool mean (H2)W expand + b.

[0226] In this embodiment, the vectors of text 1 and text 2 are fused to obtain the feature representation of the text pair using a concatenation operation, as follows:

[0227] H text =Concat(vector text1 ,vector text2 );

[0228] The decoder uses the Qwen model, and uses its pre-training and instruction-following capabilities as a teacher model for joint training with the encoder. The specific processing flow is as follows: The decoder's instructions consist of two parts: task instructions + reference data; the format is as follows:

[0229] (1) Reconstructing two texts and comparing the similarity or relevance between the two texts must be done in accordance with <build>Reconstructed text< / build> <score>Fraction< / score> Output in the format of

[0230] (2) The score range is [0,1], with four decimal places;

[0231] (3){text pair};

[0232] The {text pair} is the text pair data input by the encoder. When only the encoder is trained, it needs to be randomly masked with a masking rate of 50%. The instruction converts the text into digital Token-Ids through the word segmenter of the Qwen model, and then performs word embedding and position encoding to generate the input sequence of the text. Where n is the sequence length, the maximum length is 32768, d is the feature dimension, the value is 1536,

[0233] E emb-prompt =E rok-prompt +E pos-prompt ;

[0234] The input vector is linearly transformed to generate the query vector (Q), key vector (K), and value vector (V):

[0235] Q=E emb-prompt W Q ,K=E emb-prompt W K ,V=E emb-prompt W V ;

[0236] in, d group= 256, then the mask self-attention mechanism models the input sequence to capture the dependency between elements in the sequence; as follows: the input instruction needs to be masked to prevent the model from "cheating" during training, i.e. using future information, to ensure that the prediction of the current position only depends on the generated sequence, generate an upper triangular matrix mask, the element value is negative infinity, so that the current position cannot pay attention to the subsequent position:

[0237]

[0238] Add the mask to the attention mechanism calculation, and then normalize by Softmax to get attention: Since attention uses grouped query attention, a multi-group feature integration needs to be done after calculating attention; the specific operation is to add a fully connected layer;

[0239] Attention self = Attention (Q, K, V) W;

[0240] Where, Then do residual processing and normalization processing on the output of self-attention to alleviate gradient disappearance, speed up training convergence, and stabilize the model learning process; the residual connection is to add the input E emb-prompt of the self-attention layer directly to the output:

[0241] H norm = E res-attention + Attention text ;

[0242] Layer normalization is performed on the output of the residual connection:

[0243] H Q = LayerNorm (H norm ) ;

[0244] The cross-attention mechanism models the text pair features of the encoder output and the instruction features of the decoder, and the calculation process is as follows: the input vector is transformed by linear transformation to generate the query vector (Q), the key vector (K) and the value vector (V):

[0245] Q = H K W norm , K = H V W group , V = H cross W res-attention ;

[0246] Where, d norm = 256;

[0247] The cross attention also adopts a plurality of query attention mechanisms, and integrates a plurality of attentions,

[0248]

[0249] Attention cross = Attention (Q, K, V) W;

[0250] wherein, Then, the output of the cross attention is subjected to residual processing and normalization processing, the input of the cross attention layer is directly added to the output of the cross attention layer:

[0251] H res-attention = H norm + Attention cross ;

[0252] The output of the residual connection is subjected to layer normalization:

[0253] H norm = LayerNorm (H res-attention ) ;

[0254] Subsequently, a feedforward neural network is used for nonlinear transformation to enhance the expression ability of the model, and the feedforward neural network comprises three linear layers and a SiLU activation function, and the calculation is as follows:

[0255] H FFN = SiLU (H res-norm W gate ) (H res-norm W up ) W down ;

[0256] wherein, d ffn = 8960.

[0257] The output of the feedforward neural network is subjected to residual connection and normalization with the input:

[0258] H res-ffn = H norm + Attention cross ;

[0259] H decoder = LayerNorm (H res-ffn ) ;

[0260] The above is the calculation process of a single layer of the Decoder, and the calculation methods of the remaining layers are the same, the difference is that the input is the output of the previous layer of the Decoder, and the 28-layer Decoder of Qwen is adopted in the application, that is, M = 28; after M layers of Decoder processing, the output features of the last layer are obtained, that is:

[0261] H = Decoder(E tok-prompt + E pos-prompt ); finally, the output features are mapped through a linear mapping,

[0262] to the vocabulary dimension, and the output text is decoded.

[0263] It should also be noted that the terms "comprising," "including," and any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.

[0264] The various embodiments in this application are described in a progressive manner, and the same or similar parts among the various embodiments can be mutually referred to. Each embodiment focuses on the differences from other embodiments. In particular, the system embodiments are described in a relatively simple manner because they are substantially similar to the method embodiments, and the relevant parts can be referred to the description of the method embodiments.

Claims

1. A text embedding joint training method based on policy gradient optimization, characterized by: include: (1) Asymmetric encoder-decoder architecture; The asymmetric encoder-decoder architecture consists of an encoder and a decoder, where the encoder operates by generating a digitized sequence from the input text through word embedding moments and positional encoding; The decoder operates by inputting instructions and generating a digital sequence through word embedding matrix and position encoding; (2) a dynamic reward alignment mechanism, wherein the dynamic reward alignment mechanism includes generating two subtasks, and the decoder completes the two subtasks according to the instructions; (3) The training process of policy gradient optimization is as follows:

1. Initialize the policy model π θ ; and the reference model π ref ; 2. In a batch, for each text pair, sample a set of outputs {o1, o2, ..., o G }, where G is the size of the group; 3. Parse and calculate rewards based on the output sampled from the decoder [r1, r2, ..., r G ], specifically as follows, including: Instruction following reward: The output contains <build> 、< / build> 、 <score> 、< / score> , each hit will be rewarded with 0.25 points; Text Reconstruction Bonus: Parsing <build> and< / build> The content in the label can be evaluated using the BLEU method: Similarity Reward: Explanation <score> and< / score> The content in the label is calculated based on the encoder's pooled output vector, and the deviation between the two is used as a reward; So the total reward is: reward=reward prompt +reward rebuild +reward similarity ; (4) Normalize the rewards within the group and use the normalized rewards to calculate the advantage function. The advantage function is as follows: (5) Update the policy model by maximizing the objective function, which is as follows: (6) The encoder and decoder are cross-iterated during the training process. After the model training is completed, the encoder is saved as the final text embedding model.

2. A text embedding joint training method based on policy gradient optimization according to claim 1, characterized in that: Each layer of Encoder includes three parts: self-attention, residual + normalization, and feedforward neural network; the calculation is as follows: emb =E tok +E pos , the self-attention is calculated as: Q=W Q E emb ,K=W K E emb ,V=W V E emb ; Where W Q 、W K and W V represent the query matrix, key matrix, and value matrix respectively; The first residual + normalization is calculated as follows: Through the feedforward neural network to do nonlinear transformation, the calculation is as follows: Among them, W1 and b1 represent the weight and bias of the first fully connected layer; The second residual + normalization is calculated as follows:

3. The text embedding joint training method based on policy gradient optimization according to claim 1 is characterized in that Each layer of Decoder includes four parts: mask self-attention, cross attention, residual + normalization, and feedforward neural network; the calculation is as follows: Remember E emb =E tok +E pos , the masked self-attention is calculated as: Q=W Q E emb ,K=W K E emb ,V=W V E emb ; Where W Q 、W K and W V Represents the query matrix, key matrix and value matrix Mask respectively causal represents the causal mask matrix; The first residual + normalization is calculated as follows: Through the feedforward neural network to do nonlinear transformation, the calculation is as follows: Where W1 and b1 represent the weight and bias of the first fully connected layer; The third residual + normalization is calculated as follows: The decoder used is Qwen, which does not have the ability to process the features of the encoding layer, so cross attention is added to achieve the bridge between the decoder and the encoder.

4. The text embedding joint training method based on policy gradient optimization according to claim 1 is characterized in that Three reward mechanisms are designed based on the task: (1) Instruction follow-up reward: the output result contains <build> 、< / build> 、 <score>< / score> , each hit is rewarded with 0.25 points, and the reward value range for instruction compliance is [0,1.0]; (2) Text reconstruction reward: The input text pair is used as the target of the decoder text reconstruction task. The overlap between the generated text and the target text is calculated using the n-gram method. Specifically, the BLEU evaluation method can be used: Among them, BP is the length penalty factor, the formula is as follows: Among them, c is the length of the generated text, r is the length of the target text; P n Indicates the proportion of n-grams in the generated text that appear in the target text, which represents the weight. The default weight is the same, that is, The reward value range for text reconstruction is [0, 1.0]; (3) Similarity reward: Based on the pooled output vector of the encoder, the similarity of the text pair is calculated. The similarity score generated by the decoder is used as the target, and the deviation between the two is calculated as the reward. The calculation formula is as follows: reward=1.0-abs(score encoder -score decoder ); The value range is [0, 1.0]. After the three rewards are combined, the value range is [0, 3.0].

5. The text embedding joint training method based on policy gradient optimization according to claim 1 is characterized in that The specific process of the policy gradient-based optimization mechanism includes: (1) Sample a set of outputs for an input sample and calculate the reward for each output. The reward is calculated according to the reward mechanism [r1, r2, …, r G ]; (2) Normalize the rewards within the group and use the normalized rewards to calculate the advantage function. The advantage function is as follows: (3) Update the policy model by maximizing the objective function, which is as follows: Among them, ε and β are hyperparameters, π θ is the current policy model, π old is the old strategy model, π ref It is a reference model; (4) Iterate training and gradually optimize the training model.

6. A text embedding joint training method based on policy gradient optimization according to claim 5, characterized in that: During the entire iterative training, the encoder inputs text 1 and text 2. The encoder input is the instruction encoder using the BERT model and twin architecture, that is, the two BERT models share weights and process the text separately. The specific processing flow is as follows: Text 1 is converted into digital Token-Ids through the word segmenter, and then word embedding and position encoding are performed to generate the input sequence of the text. Where n is the sequence length, the maximum length is 512, d is the feature dimension, the value is 1024, AND emb1 =And tok1 +E pos1 ; The self-attention mechanism then models the input sequence to capture the dependencies between elements within the sequence; specifically: The input vector is linearly transformed to generate the query vector (Q), key vector (K), and value vector (V): Q=E emb1 W Q ,K=E emb1 W K ,V=E emb1 W V ; in, Calculate the dot product of Q and K, scale it, and normalize it through Softmax to get the attention: Then the output of the self-attention layer is processed with residual and normalized to alleviate the gradient disappearance, accelerate the training convergence, and stabilize the model learning process; the residual connection is to connect the input E emb1 Add directly to its output: H res-attention =E emb1 +Attention(Q,K,V); Perform layer normalization on the output of the residual connection: H norm =Layer Norm(H res-attention ); Then, a nonlinear transformation is performed through a feedforward neural network to enhance the model's expressiveness. The feedforward neural network consists of two linear layers and an activation function, which is calculated as follows: H FFN =max(0,H norm W1+b1)W2+b2; in, The output of the feedforward neural network is connected to the input with residuals and normalized: H res-ffn =H norm +H FFN ; H encoder =LayerNorm(H res-ffn ); The above is the calculation process of a single-layer encoder. The calculation methods of the remaining layers are the same. The difference is that the input is the output of the previous layer encoder. The present invention uses BERT's 24-layer encoder, that is, N = 24; after N layers of encoder processing, the output features of the last layer are obtained, namely: H1=Encoder(E tok1 +E pos1 ); Finally, the output of the last layer is mean pooled. In order to match the feature dimension of the decoder, a dimension expansion layer is added to obtain the vector representation of text 1: vector text1 =Pool mean (H1)W expand +b; in, d extend =1536; The vector representation table of text 2 is: H2=Encoder(E tok2 +E pos2 ); vector text2 =Pool mean (H2)W expand +b。 7. A text embedding joint training method based on policy gradient optimization according to claim 6, characterized in that: The vector of text 1 and the vector of text 2 are fused to obtain the feature representation of the text pair. The present invention adopts the splicing operation, which is as follows: text =Concat(vector text1 ,vector text2 ); The decoder uses the Qwen model, and uses its pre-training and instruction-following capabilities as a teacher model for joint training with the encoder. The specific processing flow is as follows: The decoder's instructions consist of two parts: task instructions + reference data; the format is as follows: (1) Reconstructing two texts and comparing the similarity or relevance between the two texts must be done in accordance with <build> Reconstructing text< / build> <score> Fraction< / score> Output in the format of (2) The score range is [0,1], with four decimal places; (3){text pair}; The {text pair} is the text pair data input by the encoder. When only the encoder is trained, it needs to be randomly masked with a masking rate of 50%. The instruction converts the text into digital Token-Ids through the word segmenter of the Qwen model, and then performs word embedding and position encoding to generate the input sequence of the text. Where n is the sequence length, the maximum length is 32768, d is the feature dimension, the value is 1536, AND emb-prompt =And tok-prompt +E pos-prompt ; The input vector is linearly transformed to generate the query vector (Q), key vector (K), and value vector (V): Q=E emb-prompt W Q ,K=E emb-prompt W K ,V=E emb-prompt W V ; in, d group = 256. Subsequently, the masked self-attention mechanism models the input sequence to capture the dependencies between elements within the sequence. Specifically, the input instructions need to be masked to prevent the model from "cheating" during training. This means using future information to ensure that the prediction of the current position depends only on the generated sequence. This generates an upper triangular matrix mask with element values ​​of negative infinity, preventing the current position from paying attention to subsequent positions: Add the mask to the calculation of the attention mechanism and normalize it by Softmax to get the attention: Since attention uses group query attention, it is necessary to perform a multi-group feature integration after calculating the attention. The specific operation is to add a fully connected layer; Attention self =Attention(Q,K,V)W; in, Then the output of the self-attention layer is processed with residual and normalized to alleviate the gradient disappearance, accelerate the training convergence, and stabilize the model learning process; the residual connection is to connect the input E emb-prompt , and add directly to its output: H res-attention =E emb-prompt +Attention self ; Perform layer normalization on the output of the residual connection: H norm =LayerNorm(H res-attention ); The cross-attention mechanism models the text pair features of the encoder's output and the instruction features of the decoder. The calculation process is as follows: the input vector is linearly transformed to generate the query vector (Q), key vector (K) and value vector (V): Q=H text W Q ,K=H norm W K ,V=H norm W V ; in, d group =256; Cross attention also uses multiple query attention mechanisms and integrates multiple attention groups. Attention cross =Attention(Q,K,V)W; in, Then the output of the cross attention is processed with residuals and normalized, and the input of the cross attention layer is directly added to its output: H res-attention =H norm +Attention cross ; Perform layer normalization on the output of the residual connection: H norm =LayerNorm(H res-attention ); Then, a nonlinear transformation is performed through a feedforward neural network to enhance the model's expressiveness. The feedforward neural network contains three linear layers and a SiLU activation function. The calculation is as follows: A FFN =SiLU(H res-norm W gate )(H res-norm W up )W down ; in, d ffn =8960; The output of the feedforward neural network is connected to the input with residuals and normalized: H res-ffn =H norm +Attention cross ; H decoder =LayerNorm(H res-ffn ); The above is the calculation process of a single-layer decoder. The calculation methods of the other layers are the same. The difference is that the input is the output of the previous layer decoder. The present invention adopts Qwen's 28-layer decoder, that is, M = 28; after M layers of decoder processing, the output feature of the last layer is obtained, that is: H = Decoder (E tok-prompt +E pos-prompt ); Finally, the output features are linearly mapped to the vocabulary dimension and then decoded to obtain the output text.

Images