Text reinforcement learning method and device, electronic equipment and computer storage medium

Abstract

The invention provides a text reinforcement learning method and device, electronic equipment and a computer storage medium, relates to the technical field of text reinforcement learning, and is applied to a text generation model. The method comprises the following steps: determining one or more target vocabularies in a statement to be analyzed, and generating a replacement word set according to the target vocabularies; replacing the target vocabulary in the to-be-analyzed statement with a current candidate replacement word to obtain a replacement statement; calculating one or more difference index values between the replacement statement and the to-be-analyzed statement; wherein the difference index value is used for quantifying the influence of the replacement of the target vocabulary on the original sentence from different dimensions; based on the difference index value, determining an importance weight of the target vocabulary in the to-be-analyzed statement; wherein the importance weight is used for redistributing the reward of the single character in the sentence. Vocabulary importance is quantified through disturbance analysis and multi-dimensional evaluation, and then fine-grained distribution of rewards is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of text reinforcement learning, in particular to a text reinforcement learning method and device, an electronic device and a computer storage medium. BACKGROUND

[0002] With the rise of ChatGPT, large language models have gradually become a research hotspot in the field of natural language processing. In order to continuously improve the ability of the model, the training method has gradually expanded from the initial supervised fine-tuning (SFT) to include reinforcement learning (Reinforcement Learning), reward function learning (Reward Modeling), and chain-of-thought reasoning, etc. In reinforcement learning, especially human feedback-based reinforcement learning (RLHF), the value feedback mechanism of the reward function plays a crucial role, which directly affects the model optimization direction and generation quality.

[0003] Currently, in the text generation task, the widely used reward feedback mechanism is usually based on the whole generated text, and a total reward or advantage value (Advantage) is calculated as the global value estimate of the current sample. Specifically, whether using the REINFORCE algorithm, proximal policy optimization (PPO), or value function-based methods, the model often relies on the overall evaluation of the complete sentence or dialogue round, such as the paragraph-level score given by artificial annotation or pre-trained reward model, and then averages or aggregates a scalar reward representing the quality of the entire generated sequence. However, this reward allocation mechanism based on the whole text has obvious limitations. In the text generation task, the traditional reinforcement learning reward is usually only for the whole sentence (Sentence-level Reward), and the model cannot know which word or words in the sentence contribute most or least to the reward. This leads to low learning efficiency of the model and unclear optimization direction. The most prominent problem is the lack of fine-grained modeling of the internal local structure of the text, which cannot accurately depict the specific contribution of each token (word or character) to the overall reward, nor can it distinguish the value of the generated content at different positions. SUMMARY

[0004] Therefore, the purpose of the embodiments of the present application is to provide a text reinforcement learning method, device, electronic device, and computer storage medium to improve the above-mentioned problems existing in the prior art.

[0005] In a first aspect, embodiments of this application provide a text reinforcement learning method applied to a text generation model. The method includes: determining one or more target words in a sentence to be analyzed; generating a set of replacement words based on the target words; replacing the target words in the sentence to be analyzed with current candidate replacement words to obtain a replaced sentence; calculating one or more difference index values ​​between the replaced sentence and the sentence to be analyzed; wherein the difference index values ​​are used to quantify the impact of the replacement of the target words on the original sentence from different dimensions; and determining the importance weight of the target words in the sentence to be analyzed based on the difference index values; wherein the importance weight is used to redistribute the reward for a single character in the sentence.

[0006] In the above implementation process, a difference index is introduced to quantify the impact of the replaced target word on the original sentence across multiple dimensions, thereby more precisely evaluating the semantic, structural, or stylistic role of the word in the sentence. Compared to traditional methods that rely solely on single similarity or gradient information, this multi-dimensional difference metric can more accurately identify keywords and avoid misclassifying high-frequency but irrelevant words as important. Finally, the calculated importance weights are used to redistribute character-level rewards, making the reinforcement learning signal more focused on the parts that contribute significantly to semantics, thus improving the training efficiency and effectiveness of the text generation model in terms of controllability, coherence, and goal orientation.

[0007] Optionally, calculating one or more difference index values ​​between the replacement statement and the statement to be analyzed includes: calculating the text fluency difference value of the replacement statement relative to the statement to be analyzed; wherein the text fluency difference value is used to characterize the change in text fluency before and after the replacement; calculating the word vector difference value between the word vector of the current candidate replacement word and the word vector of the target word; wherein the word vector difference value is used to characterize the semantic uniqueness of the word; calculating the overall semantic difference value between the semantic vector of the replacement statement and the semantic vector of the statement to be analyzed; wherein the overall semantic difference value is used to characterize the overall semantic change before and after the replacement; and calculating the contextual coherence difference value based on the semantic vector preceding and following the target word; wherein the contextual coherence difference value is used to characterize the contextual coherence of the position of the target word.

[0008] In the above implementation process, four complementary difference index values ​​are introduced from four dimensions: local fluency, lexical semantic uniqueness, global semantic fidelity, and contextual coherence. This prevents the model from misjudging words that are awkward but semantically similar as replaceable, avoids giving too high weight to synonyms or general words, and prevents the model from ignoring structural keywords.

[0009] Optionally, calculating the text fluency difference value between the replacement statement and the statement to be analyzed includes: calculating the original sentence perplexity of the statement to be analyzed, calculating the average perplexity of all replacement statements; and calculating the text fluency difference value after obtaining the original sentence perplexity and the average perplexity; wherein the text fluency difference value is the difference between the average perplexity and the original sentence perplexity.

[0010] In the above implementation process, the original sentence is first scored by the same language model to obtain its perplexity. Then, all candidate replacement words corresponding to the target word are filled in to obtain several replacement sentences. The perplexity of each sentence is calculated and the average perplexity is taken. If replacing a word makes the entire sentence very awkward and the perplexity increases, it indicates that the original word is very important. If replacing a word decreases the perplexity, it suggests that the candidate word actually makes the sentence smoother and can be regarded as a potential optimization point.

[0011] Optionally, calculating the word vector difference value between the word vector of the current candidate replacement word and the word vector of the target word includes: obtaining the word vector of the target word and obtaining the word vector of the current candidate replacement word; calculating the word vector cosine similarity between the word vector of the target word and the word vector of the current candidate replacement word; and calculating the word vector difference value based on the word vector cosine similarity.

[0012] In the above implementation process, the same word vector encoder first outputs the vectors of the target word and all candidate replacement words respectively. Then, the cosine similarity between the target word vector and the candidate word vector is calculated for each pair, and the average similarity is taken. The word vector difference value is obtained by subtracting the average similarity from 1. If the difference value is close to 0, it indicates that the candidate word and the original word have almost the same semantics, the meaning of the sentence remains unchanged after replacement, and the original word has low irreplaceability. If the difference value is significantly greater than 0, it indicates that the candidate word deviates from the original word as a whole, the original word has high semantic uniqueness, and should be given greater importance weight. If the difference value is less than 0 for individual candidates, it suggests that there is a possibility of a replacement with closer semantics, and the original word can be used as an optimization candidate.

[0013] Optionally, calculating the overall semantic difference value between the semantic vector of the replacement statement and the semantic vector of the statement to be analyzed includes: obtaining the semantic vector of the statement to be analyzed and obtaining the semantic vector of the replacement statement; calculating the semantic vector cosine similarity between the semantic vector of the statement to be analyzed and the semantic vector of the replacement statement; and calculating the overall semantic difference value based on the semantic vector cosine similarity.

[0014] In the above implementation process, semantic vectors of the sentence to be analyzed and the replacement sentence are first generated using the same sentence-level encoder. Then, the cosine similarity between the two sentence vectors is calculated, and the overall semantic difference value is obtained by subtracting this similarity from 1. If the difference value is close to 0, it indicates that the replacement has minimal impact on the sentence-level semantics, and the target word has a limited role in the global semantics. If the difference value is significantly greater than 0, it indicates that the semantics of the sentence has shifted significantly after the replacement, and the target word is crucial to maintaining the original sentence semantics, so its importance weight should be increased accordingly. If the difference value decreases abnormally under a certain candidate replacement, it suggests that the candidate may make the sentence semantics closer to the underlying intent, and there is room for optimization of the original word.

[0015] Optionally, calculating the contextual coherence difference value based on the semantic vectors preceding and following the target word includes: dividing the sentence to be analyzed into preceding and following text segments with the target word as the boundary; obtaining the segment vectors of the preceding and following text segments; calculating the cosine similarity of the segment vectors between the segment vectors of the preceding and following text segments; and calculating the contextual coherence difference value based on the cosine similarity of the segment vectors.

[0016] In the above implementation process, the sentence to be analyzed is first divided into preceding and following segments based on the target word. Then, the same encoder is used to obtain the segment vectors of the two text segments respectively. Subsequently, the cosine similarity between the two segment vectors is calculated, and the similarity is subtracted from 1 to obtain the contextual coherence difference value. If the difference value is close to 0, it indicates that the target word is closely connected to the preceding and following text, and replacing the word has a limited impact on local coherence, and its structural role is weak. If the difference value is significantly greater than 0, it indicates that the semantics before and after the target word is removed, and the word plays a key role in maintaining contextual coherence and should be given a higher importance weight. If the difference value decreases under a certain candidate replacement, it suggests that the candidate may be better at connecting the preceding and following text, and there is room for optimization in reselecting the original word.

[0017] Optionally, the method further includes: weighting and summing the text fluency difference value, the word vector difference value, the overall semantic difference value, and the contextual coherence difference value to obtain a comprehensive score; and normalizing the comprehensive score to obtain the importance weight.

[0018] In the above implementation process, the text fluency difference value, word vector difference value, overall semantic difference value, and contextual coherence difference value are first weighted and summed according to preset weight coefficients to obtain a comprehensive score. Then, the score is normalized by min-max or softmax and mapped to the [0,1] interval to form the final importance weight. If the weight after normalization is close to 1, it indicates that the target word is determined to be irreplaceable in multiple dimensions and needs to obtain a larger proportion in the reinforcement learning reward allocation. If the weight is close to 0, it indicates that the replacement impact is weak in all dimensions, the word has limited contribution to the sentence, and the reward ratio is reduced accordingly. If the weight is systematically reduced under candidate replacement, it suggests that the original word may be replaced by a better word, which can be used as an optimization direction for subsequent strategy updates.

[0019] Optionally, generating a set of replacement words based on the target vocabulary includes: using a bidirectional masking language model to replace the target vocabulary in the statement to be analyzed with masking markers; predicting the K candidate words with the highest probability at the masking position through the masking language modeling header of the model to form the set of replacement words; wherein, the set of replacement words includes the target vocabulary itself.

[0020] In the above implementation process, the target words in the sentence to be analyzed are first replaced with mask markers to obtain a masked input sequence. Then, this sequence is fed into a bidirectional masked language model. The masked language modeling header outputs the probability distribution of the mask positions, and the K candidate words with the highest probabilities are selected to form a replacement word set. At the same time, the target words themselves are forcibly included in this set. If the target word appears in a high-probability segment, it indicates that it conforms to the expected language habits of the model in the original sentence, and the replacement risk is low. If the target word ranks low or is not recalled, it suggests that the word is relatively abrupt in the sentence or has a unique semantic meaning. Replacement may significantly change the meaning of the sentence, and it should be given a higher importance weight in the subsequent difference index calculation. By retaining the target word in the set, empty references are avoided in the weight calculation.

[0021] Secondly, embodiments of this application provide a text reinforcement learning device, the device comprising: an analysis module, a replacement module, a calculation module, and a determination module; the analysis module is used to determine one or more target words in a sentence to be analyzed, and generate a set of replacement words based on the target words; the replacement module is used to replace the target words in the sentence to be analyzed with current candidate replacement words to obtain a replaced sentence; the calculation module is used to calculate one or more difference index values ​​between the replaced sentence and the sentence to be analyzed; wherein, the difference index values ​​are used to quantify the impact of the replacement of the target words on the original sentence from different dimensions; the determination module is used to determine the importance weight of the target words in the sentence to be analyzed based on the difference index values; wherein, the importance weight is used to redistribute the reward for a single character in the sentence.

[0022] In the above implementation process, the analysis module outputs candidate words and their replacement sets; the replacement module generates corresponding rewritten sentences; the calculation module quantifies sentence-level differences; and the determination module converts the differences into the importance weight of the word for subsequent reward redistribution.

[0023] Optionally, the difference index values ​​include: text fluency difference value, word vector difference value, overall semantic difference value, and contextual coherence difference value; wherein, the text fluency difference value is used to characterize the change in text fluency before and after the replacement; the word vector difference value is used to characterize the semantic uniqueness of the words; the overall semantic difference value is used to characterize the overall semantic change before and after the replacement; and the contextual coherence difference value is used to characterize the contextual coherence of the target word's location.

[0024] In the above implementation process, the introduction of difference index values ​​is to quantify and evaluate the impact of replacing a word in the text on text quality, semantic fidelity, and language naturalness. Difference index values ​​include text fluency difference value, word vector difference value, overall semantic difference value, and contextual coherence difference value. The text fluency difference value represents the change in text fluency before and after the replacement; the word vector difference value represents the semantic uniqueness of the word; the overall semantic difference value represents the overall semantic change before and after the replacement; and the contextual coherence difference value represents the contextual coherence of the target word's location.

[0025] Thirdly, embodiments of this application also provide an electronic device, which includes a memory and a processor. The memory stores program instructions, and when the processor reads and runs the program instructions, it executes the steps in any of the above implementation methods.

[0026] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer program instructions, which, when read and executed by a processor, perform the steps in any of the above implementations. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a first schematic diagram of a text reinforcement learning method provided in an embodiment of this application; Figure 2 This is a second schematic diagram of the text reinforcement learning method provided in the embodiments of this application; Figure 3 This is a third schematic diagram of the text reinforcement learning method provided in the embodiments of this application; Figure 4 This is a fourth schematic diagram of the text reinforcement learning method provided in the embodiments of this application; Figure 5 This is a fifth schematic diagram of the text reinforcement learning method provided in the embodiments of this application; Figure 6 The sixth schematic diagram of the text reinforcement learning method provided in the embodiments of this application; Figure 7 The seventh schematic diagram of the text reinforcement learning method provided in the embodiments of this application; Figure 8 The eighth schematic diagram of the text reinforcement learning method provided in the embodiments of this application; Figure 9 A block diagram illustrating the text reinforcement learning device provided in the embodiments of this application; Figure 10 This is a block diagram of an electronic device provided in an embodiment of this application.

[0029] Icons: 01-Analysis module; 02-Replacement module; 03-Calculation module; 04-Determination module; 100-Electronic device; 111-Memory; 112-Memory controller; 113-Processor; 114-Peripheral interface; 115-Input / output unit; 116-Display unit. Detailed Implementation

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of the embodiments of this application.

[0031] In a first aspect, embodiments of this application provide a text reinforcement learning method applied to a text generation model. The text generation model can be used on electronic devices with logical computing functions, such as personal computers (PCs), tablets, smartphones, and personal digital assistants (PDAs).

[0032] Optionally, please refer to Figure 1 , Figure 1 A first schematic diagram of a text reinforcement learning method provided in an embodiment of this application. The method includes: Identify one or more target words in the sentence to be analyzed, and generate a set of replacement words based on the target words; The target words in the sentence to be analyzed are replaced with the current candidate replacement words to obtain the replaced sentence; Calculate one or more difference index values ​​between the replacement sentence and the sentence to be analyzed; wherein, the difference index values ​​are used to quantify the impact of the replacement of target words on the original sentence from different dimensions; Based on the difference index value, the importance weight of the target word in the sentence to be analyzed is determined; among them, the importance weight is used to redistribute the reward for a single character in the sentence.

[0033] In one embodiment of this application, a set of candidate replacements for the target word is first recalled, and then each candidate is sequentially filled into the original sentence to generate the corresponding replacement sentence. Subsequently, four types of difference indicators—fluency, word vectors, overall semantics, and contextual coherence—are calculated in parallel to quantify the multi-dimensional shift caused by the replacement. Finally, the four types of difference values ​​are weighted, summed, and normalized to obtain the importance weight of the target word. If the weight is high, it indicates that the sentence meaning, fluency, or coherence is significantly impaired after the replacement, and the reinforcement learning stage will concentrate more rewards on the characters corresponding to the word, inhibiting the model from easily rewriting it. If the weight is low, it indicates that the candidate replacements are generally mild, the rewards are diluted, and the model can confidently explore better expressions. This achieves a fine redistribution of character-level rewards, improving the training efficiency and generation quality of the text generation model in terms of semantic fidelity, style consistency, and logical coherence.

[0034] Optionally, a masked language model (MLM) can be used to recall candidates, such as BERT or RoBERTa. The target words in the original sentence are replaced with [MASK] tags, and the result is input into the MLM model. The model predicts the possible words at the mask positions and selects the Top-k (e.g., k=10~100) as the candidate set (which can be sorted by probability or score). For each candidate replacement word, it is filled into the mask position of the original sentence to generate a complete sentence.

[0035] Optionally, please refer to Figure 2 , Figure 2 A second schematic diagram of the text enhancement method provided in this application embodiment. Calculating one or more difference index values ​​between the replacement statement and the statement to be analyzed includes: Calculate the text fluency difference value between the replacement statement and the statement to be analyzed; whereby the text fluency difference value is used to characterize the change in text fluency before and after the replacement. Calculate the word vector difference value between the word vector of the current candidate replacement word and the word vector of the target word; whereby the word vector difference value is used to characterize the semantic uniqueness of the word. Calculate the overall semantic difference value between the semantic vector of the replacement statement and the semantic vector of the statement to be analyzed; whereby the overall semantic difference value is used to characterize the overall semantic change before and after the replacement. Based on the semantic vectors of the preceding and following texts of the target word, a contextual coherence difference value is calculated; whereby the contextual coherence difference value is used to characterize the contextual coherence of the target word's location.

[0036] In the above implementation process, the same encoder is first called to extract four types of signals from the original and replacement sentences: fluency, word vectors, overall semantics, and contextual coherence, and outputs four difference values ​​in parallel. Then, the difference values ​​are weighted, summed, and normalized to obtain the importance weight of the target word. If the weight is high, it indicates that the replaced sentence has significantly degraded in terms of fluency, semantics, or coherence. Reinforcement learning will amplify the reward of the character corresponding to the word and suppress rewriting in subsequent generation. If the weight is low, it indicates that the multi-dimensional impact is weak and the reward is diluted accordingly. The model can continue to explore better candidates. This achieves accurate redistribution of character-level rewards, driving the text generation model to accelerate convergence and improve output quality while maintaining the original meaning, punctuation, and logical coherence.

[0037] Optionally, please refer to Figure 3 , Figure 3 This is a third schematic diagram of the text enhancement method provided in this application embodiment. Calculating the text fluency difference value between the replaced statement and the statement to be analyzed includes: Calculate the perplexity of the original sentence in the statement to be analyzed, and calculate the average perplexity of all replacement sentences; Given the original sentence perplexity and the average perplexity, calculate the text fluency difference value; where the text fluency difference value is the difference between the average perplexity and the original sentence perplexity.

[0038] In the above implementation process, perplexity is a metric used in natural language processing to measure the quality of a language model's predictions. The more accurate a model's predictions, the lower its perplexity. First, the same language model scores the original sentence to obtain its perplexity. Then, all candidate replacement words corresponding to the target word are sequentially added, and their perplexities are calculated for each word. The average perplexity is then obtained. The difference between the average perplexity and the original sentence's perplexity is used as the text fluency difference value. If the difference is significantly greater than zero, it indicates that the sentence becomes awkward after replacement, and the original word is crucial to fluency. If the difference is close to zero or even negative, it suggests that the candidate word is actually smoother, and the original word has room for optimization. This quantifies the change in surface language quality and serves as a direct basis for subsequent importance weighting.

[0039] Optionally, the difference between the perplexity level (PPL) of the original sentence and the average PPL of the replacement sentence can be calculated using a formula:

[0040] Here, The original sentence is given, and S = [ ], For the first sentence One character, The length of the sentence; To replace the candidate word set, For set Each candidate word in the list is used for replacement. A list of all possible words, The number of all candidate words.

[0041] in, The perplexity difference value measures the difference between the fluency of the original sentence and the average fluency of the sentence after the replacement words. The larger the value, the less fluent the replaced sentence is, and the more important the original word is. Calculate the perplexity of the original sentence S. , This is the most common scenario, meaning the original sentence has a lower average perplexity than the replaced sentence. In other words, the original sentence flows better than the replaced sentence. The higher this value, the better the original sentence is perceived to be. The more important and irreplaceable something is in this context.

[0042] The above calculation process is as follows: in an original sentence In the middle, select a word to study. Prepare a set of candidate replacement words. , count as the original sentence level of confusion For sets Each candidate word Generate replacement sentences Calculate the perplexity of the replacement sentence. Sum the perplexity scores of all replacement sentences and divide by the number of candidate words, k, to obtain the average replacement perplexity. Subtract this average perplexity from the perplexity score of the original sentence to obtain the final perplexity score. .

[0043] In one embodiment of this application, the word "green" in the line "The spring breeze has turned the south bank of the Yangtze River green again, when will the bright moon shine on my return?" is used as an example. The input text is "The spring breeze has turned the south bank of the Yangtze River green again," and the system analyzes it to replace "green" with candidate words ["dyed," "passed," "arrived," "full"]. The calculated results are PPL (original sentence) = 1.2, PPL (average replacement) = 2.8, and the score is... (High score, indicating the word "green" is extremely important). System feedback: The word "green" is the finishing touch of this sentence; it is recommended to keep it.

[0044] Optionally, please refer to Figure 4 , Figure 4 This is a fourth schematic diagram of the text enhancement method provided in this application embodiment. Calculating the word vector difference value between the word vector of the current candidate replacement word and the word vector of the target word includes: Obtain the word vectors of the target words and the word vectors of the current candidate replacement words; Calculate the cosine similarity between the target word vector and the current candidate replacement word vector; then, based on the cosine similarity, calculate the word vector difference value.

[0045] In the above implementation process, the context vectors of the target word and the current candidate replacement word are first extracted using the same word vector model to obtain their vector representations. Then, the cosine similarity between the two vectors is calculated, and the difference between 1 and the cosine similarity is used as the word vector difference value. If the difference value is close to zero, it indicates that the candidate word and the original word almost overlap in semantic space, and the replacement risk is low. If the difference value is significantly greater than zero, it indicates that the candidate word deviates far from the original word, the target word has semantic uniqueness, and should receive a higher weight in the subsequent weight allocation. Thus, the word-level semantic shift is quantified as a key signal for importance assessment.

[0046] Optionally, calculate the cosine similarity between the target word embedding and the current candidate replacement word embedding, and take the inverse value:

[0047] Here, The original sentence is given, and S = [ ], For the first sentence One character, The length of the sentence. To replace the candidate word set, For set Each candidate word in the list is used for replacement. A list of all possible words, The number of all candidate words.

[0048] Among these methods, word uniqueness is used to assess the non-substitutability of a target word in the semantic space. and These are the vector representations of the target word and the candidate word, respectively. This is used to calculate the cosine similarity between them. The formula calculates the average semantic difference between the target word and all candidate words. If the uniqueness score is close to 1, it means the target word... A word that is semantically significantly different from its candidate word set is a semantically unique word that lacks synonyms; conversely, a score close to 0 indicates that the word has many semantically similar alternatives.

[0049] The less similar it is to the current candidate replacement word, the more unique the semantics of the target word, and the more irreplaceable its semantics are in the context.

[0050] In one embodiment of this application, the input text is "Research on the robustness of machine learning algorithms"; the system analysis is the semantic uniqueness of "robustness"; the calculation process is the cosine similarity between emb ("robustness") and candidate words ["stability", "reliability", "anti-interference"]; the result is Uniqueness ("robustness") = 1. 0.3 = 0.7 (high uniqueness); the system feedback of "robustness" is a professional term that reflects the academic value of the paper and is recommended to be retained.

[0051] Optionally, please refer to Figure 5 , Figure 5 This is a fifth schematic diagram of the text enhancement method provided in this application embodiment. Calculating the overall semantic difference value between the semantic vector of the replaced statement and the semantic vector of the statement to be analyzed includes: Obtain the semantic vector of the statement to be analyzed, and obtain the semantic vector of the statement to be replaced; Calculate the semantic vector cosine similarity between the semantic vector of the statement to be analyzed and the semantic vector of the statement to be replaced; Calculate the overall semantic difference value based on the cosine similarity of semantic vectors.

[0052] In the above implementation, the first step is to select a pre-trained language model to convert the text into vectors. Cosine similarity assesses the similarity in direction between two vectors by measuring the cosine of the angle between them, while ignoring their magnitude. This is very suitable for semantic vectors because we are more concerned with direction in the semantic space than with the intensity of expression.

[0053] Optionally, a sentence encoder (such as sentence-BERT) is used to calculate the similarity between the original sentence and the replacement sentence, where the original sentence is encoded as follows: The replacement sentence is coded as follows: ,in Then, the similarity between the original sentence and the replacement sentence is calculated:

[0054] Here, For the original sentence S=[ ], For the first sentence One character, The length of the sentence. To replace the candidate word set, For set Each candidate word in the list is used for replacement. A list of all possible words, The number of all candidate words.

[0055] in, It usually represents the amount of semantic change or semantic difference, and is used to measure the degree of semantic change before and after a certain operation (such as word replacement). The larger the value, the greater the impact of the target word on the semantics of the whole sentence.

[0056] In one embodiment of this application, in the news headline optimization scenario, the input text is "Tech giant stock price plunge triggers investor panic"; the system analyzes the semantic change impact of "plunge". The calculation process is as follows: (1) replacing "plunge" with "decline" results in a semantic similarity of 0.45; (2) replacing "plunge" with "rebound" results in a semantic similarity of 0.35. The average similarity is (0.45 + 0.35) / 2 = 0.4; the result is... =1-0.4-0.6 (high semantic impact); the system feedback indicates that "plummeting" contributes the most to the sense of urgency and impact of the news, so it is recommended to retain it.

[0057] Optionally, please refer to Figure 6 , Figure 6 The sixth schematic diagram illustrates the text enhancement method provided in this application embodiment. Based on the semantic vectors preceding and following the target word, a contextual coherence difference value is calculated, including: The sentence to be analyzed is divided into a preceding text segment and a following text segment, with the target words as the boundary. Get the fragment vector of the preceding text segment and the fragment vector of the following text segment; Calculate the cosine similarity between the fragment vectors of the preceding text segment and the fragment vectors of the following text segment; Calculate the context coherence difference value based on the cosine similarity of the fragment vectors.

[0058] In the above implementation process, semantic vectors of the sentence to be analyzed and the replacement sentence are first generated separately using the same sentence-level encoder, resulting in vector representations of the two sentences. Then, the cosine similarity between the two vectors is calculated, and the difference between 1 and the cosine similarity is used as the overall semantic difference value. If the difference value is close to zero, it indicates that the meaning of the sentence before and after the replacement is almost unchanged, and the target word has a limited impact on the overall semantics. If the difference value is significantly greater than zero, it indicates that the semantics of the sentence after the replacement has shifted significantly, and the target word plays a key role in maintaining the original meaning and should receive a higher share in the importance weight. Thus, the sentence-level semantic drift is quantified as an important basis for subsequent reward allocation.

[0059] Optionally, the context is first divided into the preceding text. The following text is ; the previous text Input the BERT model and extract the [CLS] vector from the last layer as... ; the following text Input the BERT model and extract the [CLS] vector from the last layer as... Calculate the cosine similarity between the two; the greater the difference, the higher the penalty.

[0060] In one embodiment of this application, the input text is "Deep learning algorithms perform excellently. This method achieves best results on multiple datasets." The system analyzes the contextual coherence of "this method," calculating E[previous context] = emb("Deep learning algorithms perform excellently"), E[following context] = emb("Achieved best results on multiple datasets"), cos(previous context, following context) = 0.9 (high coherence) - degree of breakage = 1 - 0.9 = 0.1 (good coherence). The result shows a small breakage penalty, and "this method" maintains a high weight. The system feedback indicates that "this method" effectively maintains thematic consistency between the preceding and following contexts and is a key referential term.

[0061] Optionally, please refer to Figure 7 , Figure 7 A seventh schematic diagram illustrating the text enhancement method provided in this application embodiment. The method further includes: The weighted sum of the differences in text fluency, word vectors, overall semantics, and contextual coherence is used to obtain a comprehensive score. The overall score is normalized to obtain the importance weights.

[0062] In one embodiment of this application, for characters in a sentence The overall score can be obtained as follows: Let the sentence be S=[ ], target words are Its weighting function can be defined as:

[0063] in, Indicates replacement word The resulting change in confusion reflects its contribution to sentence fluency. As mentioned above, it embodies its own semantic uniqueness. Indicating "will" The amount of semantic change in the entire sentence after replacement.

[0064] From the beginning of the sentence to The contextual representation of all previous words (e.g., the embedding vector of the first half of the sentence encoded using models such as BERT). From the sentence The contextual representation of all words from then until the end. Calculate the semantic coherence or relevance between the first and second halves of a sentence. The value is high, close to 1, even after removal. The two parts of the sentence are still highly related, indicating that... The connection is weak. The value is low, close to 0, indicating that [the element] has been removed. Afterwards, the two parts of the sentence lost their coherence, indicating... It is a key link connecting the preceding and following text.

[0065] In the weight function The weight parameters are normalized. Returning to the value function, This allows for the measurement of the dominant value of individual characters. During model learning, the focus shifts directly to the most dominant token. This learning process enables the model to generate more effective and meaningful responses. For example, given the input "Spring breeze has turned the south bank of the Yangtze River green again, when will the bright moon shine on my return?", the calculation process involves evaluating the character "green" as... =1.6, , The fracture analysis score is 0.1. The weight calculation is: Weight(“Green”) = 0.3 × 1.6 + 0.25 × 0.9 + 0.25 × 0.8 0.2 × 0.1 = 0.885; the system output is that the word "green" has a weight of 0.885 (out of 1.0), making it the most important word in the whole sentence and reflecting the poet's unique creativity.

[0066] By conducting multi-faceted analysis at the linguistic level (fluency, lexical semantics, sentence semantics, and contextual structure), sparse sentence-level rewards can be transformed into dense lexical-level rewards. This could enable reinforcement learning to more efficiently focus on learning those truly crucial "key words," further improving the quality and accuracy of text generation models.

[0067] Optionally, please refer to Figure 8 , Figure 8 This is an eighth schematic diagram of the text enhancement method provided in this application embodiment. It generates a set of replacement words based on the target vocabulary, including: Using a bidirectional masked language model, the target words in the sentence to be analyzed are replaced with masked tags; By modeling the head of the masked language model, the K candidate words with the highest probability at the masked position are predicted to form a set of replacement words; the set of replacement words includes the target word itself.

[0068] In the above implementation process, the statement to be analyzed, S=[ The target words in the sequence are replaced with the [MASK] markers to obtain a masked input sequence. This sequence is then fed into a bidirectional masked language model, where the masked language modeling header outputs the probability distribution of the mask positions, and the K candidate words with the highest probabilities are selected to form an initial set. Finally, the target words themselves are added back to the set. If the target word ranks high in the Top-K, it indicates that it conforms to the model's language prior, and the replacement risk is low. If it does not enter the Top-K or ranks low, it suggests that the word has unconventional semantics, and replacement is more likely to cause semantic drift, thus giving it a higher score in subsequent importance weights. This balances recall and benchmark, ensuring the diversity and comparability of the replacement word set.

[0069] Secondly, embodiments of this application provide a text reinforcement learning device; please refer to [link to relevant documentation]. Figure 9 , Figure 9 A block diagram illustrating the text reinforcement learning device provided in an embodiment of this application.

[0070] The device includes: an analysis module 01, a replacement module 02, a calculation module 03, and a determination module 04. The analysis module 01 is used to determine one or more target words in the sentence to be analyzed and generate a set of replacement words based on the target words. The replacement module 02 is used to replace the target words in the sentence to be analyzed with the current candidate replacement words to obtain the replaced sentence. The calculation module 03 is used to calculate one or more difference index values ​​between the replaced sentence and the sentence to be analyzed. The difference index values ​​are used to quantify the impact of the replacement of the target words on the original sentence from different dimensions. The determination module 04 is used to determine the importance weight of the target words in the sentence to be analyzed based on the difference index values. The importance weight is used to redistribute the reward of a single character in the sentence.

[0071] In the aforementioned implementation process, by integrating four modules—analysis, replacement, calculation, and determination—the device can systematically evaluate the semantic and structural contributions of target words in sentences. Specifically, the analysis module 01 first identifies the words to be examined and generates a candidate replacement set; the replacement module 02 constructs a new sentence based on this; the calculation module 03 quantifies the differences before and after replacement from multiple dimensions, forming difference indicators; and the final determination module 04 integrates these indicators to assign an "importance weight" to the target words. This weight can be directly used to redistribute the reward for individual characters in the sentence, enabling the reinforcement learning model to focus more rewards on key characters during the training phase, suppressing noise signals, thereby improving the accuracy and interpretability of downstream tasks such as text generation, text correction, or keyword extraction.

[0072] Optionally, the difference index values ​​include: text fluency difference value, word vector difference value, overall semantic difference value, and contextual coherence difference value; wherein, the text fluency difference value is used to characterize the change in text fluency before and after the replacement; the word vector difference value is used to characterize the semantic uniqueness of the words; the overall semantic difference value is used to characterize the overall semantic change before and after the replacement; and the contextual coherence difference value is used to characterize the contextual coherence of the target word's location.

[0073] In the above implementation process, the text fluency difference value, scored using language model perplexity or fluency, can immediately detect whether the replacement leads to grammatical errors or awkward pronunciation, avoiding low-quality replacement sentences from misleading subsequent weight calculations. The word vector difference value, by calculating the distance between the target word and candidate words in the distributed representation space, quantifies whether there is "irreplaceable proprietary semantics" at that position, thus highlighting high-information-entropy, low-frequency but crucial entities or technical terms. The overall semantic difference value uses sentence-level embedding or cross-encoder scoring to measure whether the replacement causes semantic drift, preventing "fluent but off-topic" replacement sentences from receiving artificially high weights. The contextual coherence difference value utilizes the co-occurrence probability of adjacent windows or the referential resolution chain to check the dependency relationship between the target word and the preceding and following text, ensuring that subtle coherence information such as pronouns, omissions, and collocations is not disrupted.

[0074] Optionally, please refer to Figure 10 , Figure 10 This is a block diagram illustrating an electronic device according to an embodiment of this application. The electronic device 100 may include a memory 111, a memory controller 112, a processor 113, a peripheral interface 114, an input / output unit 115, and a display unit 116. Those skilled in the art will understand that... Figure 10 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device 100. For example, the electronic device 100 may also include components that are more... Figure 10 The more or fewer components shown, or having the same Figure 10 The different configurations shown.

[0075] The aforementioned memory 111, memory controller 112, processor 113, peripheral interface 114, input / output unit 115, and display unit 116 are electrically connected directly or indirectly to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines. The aforementioned processor 113 is used to execute executable modules stored in the memory.

[0076] The memory 111 can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory 111 stores programs, and the processor 113 executes these programs upon receiving execution instructions. The methods executed by the electronic device 100 as defined in any embodiment of this application can be applied to the processor 113, or implemented by the processor 113.

[0077] The aforementioned processor 113 may be an integrated circuit chip with signal processing capabilities. The processor 113 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a digital signal processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor.

[0078] The peripheral interface 114 described above couples various input / output devices to the processor 113 and the memory 111. In some embodiments, the peripheral interface 114, the processor 113, and the memory controller 112 can be implemented on a single chip. In other instances, they can be implemented on separate chips.

[0079] The input / output unit 115 described above is used to provide user input data. The input / output unit 115 may be, but is not limited to, a mouse and keyboard, etc.

[0080] The aforementioned display unit 116 provides an interactive interface (e.g., a user interface) between the electronic device 100 and the user, or displays image data for the user's reference. In this embodiment, the display unit can be a liquid crystal display (LCD) or a touch display. If it is a touch display, it can be a capacitive touchscreen or a resistive touchscreen that supports single-point and multi-point touch operations. Supporting single-point and multi-point touch operations means that the touch display can sense touch operations generated simultaneously from one or more locations on the touch display and pass the sensed touch operations to the processor for calculation and processing.

[0081] This application also provides a computer-readable storage medium storing computer program instructions, which are read and executed by a processor to perform steps in a text reinforcement learning method.

[0082] In summary, this application provides a text reinforcement learning method, apparatus, electronic device, and computer storage medium, relating to the field of text reinforcement learning technology and applied to text generation models. The method includes: determining one or more target words in a sentence to be analyzed; generating a set of replacement words based on the target words; replacing the target words in the sentence to be analyzed with current candidate replacement words to obtain a replaced sentence; calculating one or more difference index values ​​between the replaced sentence and the sentence to be analyzed; wherein the difference index values ​​are used to quantify the impact of the replacement of the target words on the original sentence from different dimensions; and determining the importance weight of the target words in the sentence to be analyzed based on the difference index values; wherein the importance weight is used to redistribute the reward for a single character in the sentence. By introducing difference index values ​​to quantify the impact of the target words being replaced on the original sentence in multiple dimensions, the semantic, structural, or stylistic role of the word in the sentence can be evaluated more precisely. Compared to traditional methods that rely solely on single similarity or gradient information, this multi-dimensional difference metric can more accurately identify keywords and avoid misjudging high-frequency but irrelevant words as important. Finally, the character-level rewards are redistributed using the calculated importance weights, so that the reinforcement learning signals are more focused on the parts that contribute more to semantics, thereby improving the training efficiency and effectiveness of the text generation model in terms of controllability, coherence and goal orientation.

[0083] In the several embodiments provided in this application, it should be understood that the disclosed device can also be implemented in other ways. The device embodiments described above are merely illustrative; for example, the block diagrams in the accompanying drawings illustrate the possible architecture, functions, and operations of the device according to various embodiments of this application. In this regard, each block in the block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram, and combinations of block diagrams, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0084] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0085] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0086] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0087] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

[0088] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A text reinforcement learning method applied to a text generation model, characterized in that, The method includes: Identify one or more target words in the statement to be analyzed, and generate a set of replacement words based on the target words; The target word in the sentence to be analyzed is replaced with the current candidate replacement word to obtain the replaced sentence; Calculate one or more difference index values ​​between the replaced sentence and the sentence to be analyzed; wherein, the difference index values ​​are used to quantify the impact of the replacement of the target word on the original sentence from different dimensions; Based on the difference index value, the importance weight of the target word in the sentence to be analyzed is determined; wherein, the importance weight is used to redistribute the reward for a single character in the sentence.

2. The text reinforcement learning method according to claim 1, characterized in that, The calculation of one or more difference index values ​​between the replacement statement and the statement to be analyzed includes: Calculate the text fluency difference value between the replaced statement and the statement to be analyzed; wherein, the text fluency difference value is used to characterize the change in text fluency before and after the replacement; Calculate the word vector difference value between the word vector of the current candidate replacement word and the word vector of the target word; wherein, the word vector difference value is used to characterize the semantic uniqueness of the word; Calculate the overall semantic difference value between the semantic vector of the replacement statement and the semantic vector of the statement to be analyzed; wherein, the overall semantic difference value is used to characterize the overall semantic change before and after the replacement. Based on the semantic vectors preceding and following the target word, a contextual coherence difference value is calculated; wherein, the contextual coherence difference value is used to characterize the contextual coherence of the target word's location.

3. The text reinforcement learning method according to claim 2, characterized in that, The calculation of the text fluency difference value between the replaced statement and the statement to be analyzed includes: Calculate the perplexity of the original sentence of the statement to be analyzed, and calculate the average perplexity of all replacement sentences; Given the original sentence perplexity and the average perplexity, calculate the text fluency difference value; The text fluency difference value is the difference between the average perplexity and the original sentence perplexity.

4. The text reinforcement learning method according to claim 2, characterized in that, The step of calculating the word vector difference value between the word vector of the current candidate replacement word and the word vector of the target vocabulary includes: Obtain the word vector of the target word and the word vector of the current candidate replacement word; Calculate the cosine similarity between the word vectors of the target vocabulary and the word vectors of the current candidate replacement word; The word vector difference value is calculated based on the cosine similarity of the word vectors.

5. The text reinforcement learning method according to claim 2, characterized in that, The calculation of the overall semantic difference value between the semantic vector of the replaced statement and the semantic vector of the statement to be analyzed includes: Obtain the semantic vector of the statement to be analyzed, and obtain the semantic vector of the statement to be replaced; Calculate the semantic vector cosine similarity between the semantic vector of the statement to be analyzed and the semantic vector of the statement to be replaced; The overall semantic difference value is calculated based on the cosine similarity of the semantic vectors.

6. The text reinforcement learning method according to claim 2, characterized in that, The step of calculating the contextual coherence difference value based on the semantic vectors preceding and following the target vocabulary includes: The sentence to be analyzed is divided into a preceding text segment and a following text segment, with the target words as the boundary. Obtain the fragment vector of the preceding text segment and the fragment vector of the following text segment; Calculate the cosine similarity between the fragment vectors of the preceding text segment and the fragment vectors of the following text segment; The contextual coherence difference value is calculated based on the cosine similarity of the fragment vectors.

7. The text reinforcement learning method according to claim 2, characterized in that, The method further includes: The text fluency difference value, the word vector difference value, the overall semantic difference value, and the context coherence difference value are weighted and summed to obtain a comprehensive score; The overall score is normalized to obtain the importance weight.

8. A text reinforcement learning device, characterized in that, The device includes: an analysis module, a replacement module, a calculation module, and a determination module; The analysis module is used to identify one or more target words in the statement to be analyzed, and generate a set of replacement words based on the target words. The replacement module is used to replace the target word in the statement to be analyzed with the current candidate replacement word to obtain the replaced statement; The calculation module is used to calculate one or more difference index values ​​between the replacement sentence and the sentence to be analyzed; wherein, the difference index values ​​are used to quantify the impact of the replacement of the target words on the original sentence from different dimensions; The determining module is used to determine the importance weight of the target word in the sentence to be analyzed based on the difference index value; wherein the importance weight is used to redistribute the reward of a single character in the sentence.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores program instructions, and when the processor executes the program instructions, it performs the steps of the method according to any one of claims 1-7.

10. A computer storage medium, characterized in that, The computer storage medium stores computer program instructions, which, when executed by a processor, perform the steps of the method according to any one of claims 1-7.

Images