Text reinforcement learning method and device, electronic equipment and computer storage medium

Abstract

The application provides a text reinforcement learning method and device, electronic equipment and computer storage medium, relates to the technical field of text reinforcement learning, and is applied to a text generation model. The method comprises the following steps: determining one or more target words in a to-be-analyzed sentence, generating a set of replacement words according to the target words, replacing the target words in the to-be-analyzed sentence with a current candidate replacement word to obtain a replacement sentence, calculating one or more difference indicator values between the replacement sentence and the to-be-analyzed sentence, wherein the difference indicator values are used to quantify the influence of replacement of the target words on the original sentence from different dimensions, determining the importance weight of the target words in the to-be-analyzed sentence based on the difference indicator values, and wherein the importance weight is used to redistribute the rewards of individual characters in a sentence. The importance of the words is quantified through perturbation analysis and multi-dimensional evaluation, and then fine-grained allocation of rewards is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of text reinforcement learning, in particular to a text reinforcement learning method and device, an electronic device and a computer storage medium. BACKGROUND

[0002] With the rise of ChatGPT, large language models have gradually become a research hotspot in the field of natural language processing. In order to continuously improve the ability of the model, the training method has gradually expanded from the initial supervised fine-tuning (SFT) to include reinforcement learning (Reinforcement Learning), reward function learning (Reward Modeling), and chain-of-thought reasoning, etc. In reinforcement learning, especially human feedback-based reinforcement learning (RLHF), the value feedback mechanism of the reward function plays a crucial role, which directly affects the model optimization direction and generation quality.

[0003] Currently, in the text generation task, the widely used reward feedback mechanism is usually based on the whole generated text, and a total reward or advantage value (Advantage) is calculated as the global value estimate of the current sample. Specifically, whether using the REINFORCE algorithm, proximal policy optimization (PPO), or value function-based methods, the model often relies on the overall evaluation of the complete sentence or dialogue round, such as the paragraph-level score given by artificial annotation or pre-trained reward model, and then averages or aggregates a scalar reward representing the quality of the entire generated sequence. However, this reward allocation mechanism based on the whole text has obvious limitations. In the text generation task, the traditional reinforcement learning reward is usually only for the whole sentence (Sentence-level Reward), and the model cannot know which word or words in the sentence contribute most or least to the reward. This leads to low learning efficiency of the model and unclear optimization direction. The most prominent problem is the lack of fine-grained modeling of the internal local structure of the text, which cannot accurately depict the specific contribution of each token (word or character) to the overall reward, nor can it distinguish the value of the generated content at different positions. SUMMARY

[0004] Therefore, the purpose of the embodiments of the present application is to provide a text reinforcement learning method, device, electronic device, and computer storage medium to improve the above-mentioned problems existing in the prior art.

[0005] In a first aspect, the embodiments of the present application provide a text reinforcement learning method applied to a text generation model, the method comprising: determining one or more target words in a to-be-analyzed sentence, generating a set of replacement words according to the target words; replacing the target words in the to-be-analyzed sentence with a current candidate replacement word to obtain a replacement sentence; calculating one or more difference indicator values between the replacement sentence and the to-be-analyzed sentence; wherein the difference indicator values are used to quantify the influence of the replacement of the target words on the original sentence from different dimensions; determining the importance weight of the target words in the to-be-analyzed sentence based on the difference indicator values; wherein the importance weight is used to redistribute the reward of a single character in a sentence.

[0006] In the above implementation process, the influence of the replacement of the target words on the original sentence in multiple dimensions is quantified by introducing difference indicator values, so as to more finely evaluate the semantic, structural or style role of the words in the sentence. Compared with the traditional method which only relies on single similarity or gradient information, this multi-dimensional difference measurement can more accurately identify key words and avoid misjudging high-frequency but insignificant words as important. Finally, the calculated importance weight is used to redistribute the character-level reward, so that the reinforcement learning signal is more focused on the part with large semantic contribution, and the training efficiency and effect of the text generation model in controllability, coherence and target orientation are improved.

[0007] Optionally, the calculating one or more difference indicator values between the replacement sentence and the to-be-analyzed sentence comprises: calculating a text fluency difference value of the replacement sentence relative to the to-be-analyzed sentence; wherein the text fluency difference value is used to represent the change of text fluency before and after replacement; calculating a word vector difference value between the word vector of the current candidate replacement word and the word vector of the target word; wherein the word vector difference value is used to represent the semantic uniqueness of the word; calculating an overall semantic difference value between the semantic vector of the replacement sentence and the semantic vector of the to-be-analyzed sentence; wherein the overall semantic difference value is used to represent the overall semantic change before and after replacement; calculating a context coherence difference value based on the semantic vector of the context before the target word and the semantic vector of the context after the target word; wherein the context coherence difference value is used to represent the context coherence of the position of the target word.

[0008] In the above implementation process, four complementary difference indicator values are introduced from four dimensions of local fluency, semantic uniqueness of words, global semantic fidelity and context coherence, to prevent the model from misjudging harsh but semantically similar words as replaceable, to avoid assigning too high weight to synonymous words or general words, and to prevent the model from ignoring structural key words.

[0009] Optionally, the computing the text fluency difference value of the replacement sentence relative to the sentence to be analyzed comprises: computing a perplexity of the original sentence, computing an average perplexity of all replacement sentences; in the case of obtaining the perplexity of the original sentence and the average perplexity, computing the text fluency difference value; wherein the text fluency difference value is the difference between the average perplexity and the perplexity of the original sentence.

[0010] In the above implementation process, first, the same language model is used to score the original sentence to obtain the perplexity of the original sentence. Then, all candidate replacement words corresponding to the target word are filled in to obtain several replacement sentences, and the perplexity is calculated for each replacement sentence to obtain the average perplexity. If a word is replaced, the whole sentence becomes very incoherent, and the perplexity rises, indicating that the original word is very critical. If a word is replaced, the perplexity decreases, indicating that the candidate word makes the sentence more coherent, which can be regarded as a potential optimization point.

[0011] Optionally, the computing the word vector difference value between the word vector of the current candidate replacement word and the word vector of the target word comprises: obtaining the word vector of the target word and the word vector of the current candidate replacement word; calculating the cosine similarity between the word vector of the target word and the word vector of the current candidate replacement word; according to the cosine similarity between the word vectors, the word vector difference value is calculated.

[0012] In the above implementation process, first, the same word vector encoder outputs the vectors of the target word and all candidate replacement words, then the cosine similarity between the vectors of the target word and the candidate replacement words is calculated, and the average similarity is obtained; the word vector difference value is obtained by subtracting 1 from the average similarity. If the difference value is close to 0, it indicates that the semantic of the candidate word is almost the same as the original word, the sentence meaning does not change after replacement, and the original word has low replaceability; if the difference value is significantly greater than 0, it indicates that the candidate word deviates from the original word, and the original word has high uniqueness in the semantic level and should be given a greater importance weight; if the difference value is less than 0 for individual candidates, it indicates that there may be a replacement with closer semantics, and the original word can be used as an optimization candidate.

[0013] Optionally, the computing the overall semantic difference value between the semantic vector of the replacement sentence and the semantic vector of the sentence to be analyzed comprises: obtaining the semantic vector of the sentence to be analyzed and the semantic vector of the replacement sentence; calculating the semantic vector cosine similarity between the semantic vector of the sentence to be analyzed and the semantic vector of the replacement sentence; according to the semantic vector cosine similarity, the overall semantic difference value is calculated.

[0014] In the implementation process, first, the semantic vectors of the to-be-analyzed sentence and the replacement sentence are respectively generated by the same sentence-level encoder; then, the cosine similarity of the two vectors is calculated, and the overall semantic difference value is obtained by subtracting the similarity from 1. If the difference value is close to 0, it indicates that the replacement has little effect on the sentence-level semantics, and the target word has limited effect on the global semantics; if the difference value is significantly greater than 0, it indicates that the sentence semantics has been significantly shifted after the replacement, and the target word is crucial to maintaining the original sentence semantics, and the importance weight should be correspondingly increased; if the difference value abnormally decreases under a certain candidate replacement, it indicates that the candidate may make the sentence semantics closer to the potential intent, and the original word has the space for optimization.

[0015] Optionally, the calculating the context coherence difference value based on the semantic vector of the context before the target word and the semantic vector of the context after the target word comprises: dividing the to-be-analyzed sentence into a context before the target word and a context after the target word; obtaining a segment vector of the context before the target word and a segment vector of the context after the target word; calculating a segment vector cosine similarity between the segment vector of the context before the target word and the segment vector of the context after the target word; and calculating the context coherence difference value according to the segment vector cosine similarity.

[0016] In the implementation process, first, the to-be-analyzed sentence is divided into a context before the target word and a context after the target word, and then the segment vectors of the two contexts are obtained by using the same encoder; then, the cosine similarity of the two segment vectors is calculated, and the context coherence difference value is obtained by subtracting the similarity from 1. If the difference value is close to 0, it indicates that the context before and after the target word is closely connected, and the replacement of the target word has limited impact on the local coherence, and the structural role of the target word is weak; if the difference value is significantly greater than 0, it indicates that there is a fault in the context semantics after removing the target word, and the target word plays a key role in maintaining the context coherence, and should be given a higher importance weight; if the difference value decreases under a certain candidate replacement, it indicates that the candidate may better connect the context before and after, and the original word has the optimization space for reselection.

[0017] Optionally, the method further comprises: weighting and summing the text fluency difference value, the word vector difference value, the overall semantic difference value, and the context coherence difference value to obtain a comprehensive score; and performing normalization processing on the comprehensive score to obtain the importance weight.

[0018] In the implementation process, the text fluency difference value, the word vector difference value, the overall semantic difference value and the context coherence difference value are weighted and summed according to preset weight coefficients to obtain a comprehensive score; then the score is normalized by min-max or softmax to map to the interval [0, 1] to form the final importance weight. If the weight after normalization is close to 1, it indicates that the multi-dimension determines that the target word cannot be replaced, and a larger proportion should be obtained in the reinforcement learning reward allocation; if the weight is close to 0, it indicates that the dimensions show weak replacement impact, and the contribution of the word to the sentence is limited, and the reward proportion is correspondingly reduced; if the weight is systematically reduced under the candidate replacement, it indicates that the original word may be replaced by a better word, which can be used as an optimization direction for subsequent strategy update.

[0019] Optionally, the generating a replacement word set according to the target word comprises: replacing the target word in the sentence to be analyzed with a mask mark using a bidirectional mask language model; predicting K candidate words with the highest probability at the mask position through a mask language modeling head of the model to form the replacement word set; wherein the replacement word set comprises the target word itself.

[0020] In the implementation process, the target word in the sentence to be analyzed is replaced with a mask mark to obtain an input sequence containing a mask; then the sequence is input into a bidirectional mask language model, the probability distribution of the mask position is output by the mask language modeling head, and K candidate words with the highest probability are taken to form a replacement word set, while the target word itself is forced into the set. If the target word appears in a high-probability segment, it indicates that it conforms to the expected language habits of the model in the original sentence, and the replacement risk is low; if the target word is ranked last or even not recalled, it indicates that the word is conspicuous or has unique semantics in the sentence, and the sentence meaning may be significantly changed after replacement, so it should obtain a higher importance weight in the subsequent difference indicator calculation; by retaining the target word in the set, the weight calculation avoids empty reference.

[0021] In a second aspect, the embodiments of the present application provide a text reinforcement learning device, which comprises an analysis module, a replacement module, a calculation module and a determination module; the analysis module is used to determine one or more target words in a sentence to be analyzed, and generate a replacement word set according to the target words; the replacement module is used to replace the target words in the sentence to be analyzed with a current candidate replacement word to obtain a replacement sentence; the calculation module is used to calculate one or more difference indicator values between the replacement sentence and the sentence to be analyzed; wherein the difference indicator values are used to quantify the influence of replacement of the target words on the original sentence from different dimensions; the determination module is used to determine the importance weight of the target words in the sentence to be analyzed based on the difference indicator values; wherein the importance weight is used to redistribute the reward of a single character in the sentence.

[0022] In the implementation process, the analysis module outputs candidate words and their replacement sets; the replacement module generates corresponding rewritten sentences; the calculation module quantifies sentence-level differences; and the determination module converts the differences into importance weights of the words, which are used for subsequent reward redistribution.

[0023] Optionally, the difference indicator value includes a text fluency difference value, a word vector difference value, an overall semantic difference value, and a context coherence difference value; the text fluency difference value is used to represent the change in text fluency before and after replacement; the word vector difference value is used to represent the semantic uniqueness of the vocabulary; the overall semantic difference value is used to represent the overall semantic change before and after replacement; and the context coherence difference value is used to represent the context coherence of the position of the target vocabulary.

[0024] In the implementation process, the difference indicator value is introduced to quantify and evaluate the influence of replacement of a certain vocabulary in the text on the text quality, semantic fidelity, and language naturalness. The difference indicator value includes a text fluency difference value, a word vector difference value, an overall semantic difference value, and a context coherence difference value, wherein the text fluency difference value represents the change in text fluency before and after replacement, the word vector difference value represents the semantic uniqueness of the vocabulary, the overall semantic difference value represents the overall semantic change before and after replacement, and the context coherence difference value represents the context coherence of the position of the target vocabulary.

[0025] In a third aspect, the embodiments of the present application further provide an electronic device, which includes a memory and a processor, the memory stores program instructions, and the processor reads and runs the program instructions to execute the steps in any of the above implementation manners.

[0026] In a fourth aspect, the embodiments of the present application further provide a computer-readable storage medium, which stores computer program instructions, and the computer program instructions are read and run by a processor to execute the steps in any of the above implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0028] Figure 1 The first schematic diagram of the text reinforcement learning method provided by the embodiments of the present application;

[0029] Figure 2A second schematic diagram of the text reinforcement learning method provided by the embodiments of the present application is shown in FIG. 2.

[0030] Figure 3 A third schematic diagram of the text reinforcement learning method provided by the embodiments of the present application is shown in FIG. 3.

[0031] Figure 4 A fourth schematic diagram of the text reinforcement learning method provided by the embodiments of the present application is shown in FIG. 4.

[0032] Figure 5 A fifth schematic diagram of the text reinforcement learning method provided by the embodiments of the present application is shown in FIG. 5.

[0033] Figure 6 A sixth schematic diagram of the text reinforcement learning method provided by the embodiments of the present application is shown in FIG. 6.

[0034] Figure 7 A seventh schematic diagram of the text reinforcement learning method provided by the embodiments of the present application is shown in FIG. 7.

[0035] Figure 8 An eighth schematic diagram of the text reinforcement learning method provided by the embodiments of the present application is shown in FIG. 8.

[0036] Figure 9 A block schematic diagram of the text reinforcement learning device provided by the embodiments of the present application is shown in FIG. 9.

[0037] Figure 10 A block schematic diagram of the electronic device provided by the embodiments of the present application is shown in FIG. 10.

[0038] Icon: 01-analysis module; 02-replacement module; 03-computation module; 04-determination module; 100-electronic device; 111-memory; 112-storage controller; 113-processor; 114-peripheral interface; 115-input / output unit; 116-display unit. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0040] In a first aspect, the embodiments of the present application provide a text reinforcement learning method, which is applied to a text generation model. The text generation model can be used on electronic devices with logical computing functions, such as personal computers (PCs), tablet computers, smart phones, personal digital assistants (PDAs), etc.

[0041] Optionally, refer to Figure 1 , Figure 1 The first schematic diagram of the text reinforcement learning method provided by the embodiment of the application. The method comprises:

[0042] determining one or more target words in the sentence to be analyzed, and generating a set of replacement words according to the target words;

[0043] replacing the target words in the sentence to be analyzed with a current candidate replacement word to obtain a replacement sentence;

[0044] calculating one or more difference indicator values between the replacement sentence and the sentence to be analyzed; wherein the difference indicator values are used to quantify the influence of the replacement of the target words on the original sentence from different dimensions;

[0045] determining the importance weight of the target words in the sentence to be analyzed based on the difference indicator values; wherein the importance weight is used to redistribute the rewards of individual characters in the sentence.

[0046] In an embodiment of the application, the candidate replacement set of the target words is recalled first, and then each candidate is filled into the original sentence to generate a corresponding replacement sentence; subsequently, four types of difference indicators, namely fluency, word vector, overall semantics and context coherence, are calculated in parallel to quantify the multi-dimensional deviation caused by the replacement; finally, the importance weight of the target words is obtained by weighting and normalizing the four types of difference values. If the weight is high, it means that the sentence meaning, fluency or connection is significantly damaged after replacement, and the reinforcement learning stage will concentrate more rewards on the characters corresponding to the word, and inhibit the model from easily rewriting it; if the weight is low, it means that the candidate replacement is generally mild, and the rewards are diluted, and the model can safely explore better expressions; thereby achieving fine redistribution of character-level rewards, and improving the training efficiency and generation quality of the text generation model in terms of semantic fidelity, style consistency and logical coherence.

[0047] Optionally, the candidates can be recalled by using a mask language model (MLM), such as BERT, RoBERTa, etc. The target words in the original sentence are replaced with a [MASK] mark, and input into the MLM model. The model will predict the possible words at the mask position, and take the top-k (such as k=10~100) as the candidate set (which can be sorted according to probability or score). For each candidate replacement word, fill it into the mask position of the original sentence to generate a complete sentence.

[0048] Optionally, refer to Figure 2 , Figure 2 The second schematic diagram of the text reinforcement method provided by the embodiment of the application. The one or more difference indicator values between the replacement sentence and the sentence to be analyzed are calculated, comprising:

[0049] The text fluency difference value of the replacement sentence relative to the text fluency difference value of the sentence to be analyzed is calculated; wherein, the text fluency difference value is used to represent the change in text fluency before and after replacement;

[0050] The word vector difference value between the word vector of the current candidate replacement word and the word vector of the target word is calculated; wherein, the word vector difference value is used to represent the semantic uniqueness of the word;

[0051] The overall semantic difference value between the semantic vector of the replacement sentence and the semantic vector of the sentence to be analyzed is calculated; wherein, the overall semantic difference value is used to represent the overall semantic change before and after replacement;

[0052] Based on the semantic vector of the context before the target word and the semantic vector of the context after the target word, the context coherence difference value is calculated; wherein, the context coherence difference value is used to represent the context coherence of the position of the target word.

[0053] In the above implementation process, the same encoder is first called to extract the fluency, word vector, overall semantic and context coherence of the replacement sentence and the original sentence, and the four difference values are output in parallel; then the difference values are weighted and summed and normalized to obtain the importance weight of the target word. If the weight is high, it indicates that the sentence after replacement has obvious degradation in fluency, semantics or coherence, and the reinforcement learning will amplify the reward of the character corresponding to the word and suppress the rewriting in the subsequent generation; if the weight is low, it indicates that the multi-dimensional impact is weak, and the reward is diluted accordingly, and the model can continue to explore better candidates; thereby realizing the precise redistribution of character-level rewards, driving the text generation model to maintain the original meaning, smooth sentence reading and logical coherence while accelerating convergence and improving output quality.

[0054] Optionally, please refer to Figure 3 , Figure 3 The third schematic diagram of the text reinforcement method provided by the embodiment of the application. The text fluency difference value of the replacement sentence relative to the text fluency difference value of the sentence to be analyzed is calculated, comprising:

[0055] The perplexity of the original sentence of the sentence to be analyzed is calculated, and the average perplexity of all replacement sentences is calculated;

[0056] In the case of obtaining the perplexity of the original sentence and the average perplexity, the text fluency difference value is calculated; wherein, the text fluency difference value is the difference between the average perplexity and the perplexity of the original sentence.

[0057] In the above implementation process, perplexity is a metric used in natural language processing to measure the quality of a language model's predictions. The more accurate a model's predictions, the lower its perplexity. First, the same language model scores the original sentence to obtain its perplexity. Then, all candidate replacement words corresponding to the target word are sequentially added, and their perplexities are calculated for each word. The average perplexity is then obtained. The difference between the average perplexity and the original sentence's perplexity is used as the text fluency difference value. If the difference is significantly greater than zero, it indicates that the sentence becomes awkward after replacement, and the original word is crucial to fluency. If the difference is close to zero or even negative, it suggests that the candidate word is actually smoother, and the original word has room for optimization. This quantifies the change in surface language quality and serves as a direct basis for subsequent importance weighting.

[0058] Optionally, the difference between the perplexity level (PPL) of the original sentence and the average PPL of the replacement sentence can be calculated using a formula:

[0059]

[0060] Here, The original sentence is given, and S = [ ], For the first sentence One character, The length of the sentence; To replace the candidate word set, For set Each candidate word in the list is used for replacement. A list of all possible words, The number of all candidate words.

[0061] in, The perplexity difference value measures the difference between the fluency of the original sentence and the average fluency of the sentence after the replacement words. The larger the value, the less fluent the replaced sentence is, and the more important the original word is. Calculate the perplexity of the original sentence S. , This is the most common scenario, meaning the original sentence has a lower average perplexity than the replaced sentence. In other words, the original sentence flows better than the replaced sentence. The higher this value, the better the original sentence is perceived to be. The more important and irreplaceable something is in this context.

[0062] The above calculation process is as follows: in an original sentence In the middle, select a word to study. Prepare a set of candidate replacement words. , count as the original sentence level of confusion For sets Each candidate word , generate replacement sentences , calculate perplexity of the replacement sentences . Sum the perplexities of all replacement sentences and divide by the number of candidate words k to get the average replacement perplexity. Subtract this average perplexity from the original sentence perplexity to get the final .

[0063] In an embodiment of the present application, take "green" in "Spring wind again green Jiangnan shore, bright moon when I still" as an example. The input text is "Spring wind again green Jiangnan shore", and the system analyzes to replace "green" with candidate words ["dye", "over", "to", "full]. The calculation result is PPL(original sentence)=1.2, PPL(average replacement)=2.8. The score is (high score, indicating that the "green" word is extremely important). The system feedback: "green" word is the key to this sentence, and it is suggested to keep.

[0064] Optionally, refer to Figure 4 , Figure 4 the fourth schematic diagram of the text strengthening method provided in the embodiment of the present application. The word vector difference value between the word vector of the current candidate replacement word and the word vector of the target word is calculated, including:

[0065] obtain the word vector of the target word, and obtain the word vector of the current candidate replacement word;

[0066] calculate the cosine similarity between the word vector of the target word and the word vector of the current candidate replacement word; according to the cosine similarity, calculate the word vector difference value.

[0067] In the above implementation process, the context vectors of the target word and the current candidate replacement word are extracted by the same word vector model respectively, and the vector representations of the two are obtained; then the cosine similarity of the two vectors is calculated, and the difference value between 1 and the cosine similarity is taken as the word vector difference value. If the difference value is close to zero, it indicates that the candidate word and the original word are almost coincident in the semantic space, and the replacement risk is low; if the difference value is significantly greater than zero, it indicates that the candidate word deviates from the original word far away, and the target word has semantic uniqueness, which should obtain higher proportion in subsequent weight distribution; thus the semantic deviation of the word level is quantified as the key signal of importance evaluation.

[0068] Optionally, the cosine similarity of the target word embedding and the current candidate replacement word embedding is calculated, and the inverse value is taken:

[0069]

[0070] Here, is the original sentence, and S=[ ], is the character in the sentence, the length of the sentence, the set of replacement candidates, for each candidate word in the set , i.e. all the alternative words for replacing , the list of all candidate words. the number of all candidate words.

[0071] where Uniqueness is the evaluation of the irreplaceability of the target word in the semantic space by word uniqueness, and are the vector representations of the target word and the candidate word, respectively, for calculating the cosine similarity between them. This formula calculates the average semantic difference between the target word and all candidate words. If the uniqueness score is close to 1, it means that the target word is significantly different in semantics from its candidate word set, and it is a semantically unique word that lacks synonyms; on the contrary, if the score is close to 0, it means that the word has many semantically similar alternatives.

[0072] The more dissimilar the current candidate replacement word is, the more unique the semantic of the target word is, and the semantic in the context is irreplaceable.

[0073] In an embodiment of the present application, the input text is "Robustness Research of Machine Learning Algorithm"; the system analyzes the semantic uniqueness of "robustness"; the calculation process is the cosine similarity between emb( "robustness") and the candidate words ["stability", "reliability", "anti-interference"], and the result is Uniqueness( "robustness") = 1 0.3 = 0.7 (high uniqueness); the system feedback is that "robustness" is a professional term, which reflects the academic value of the paper, and is recommended to be retained.

[0074] Optionally, please refer to Figure 5 , Figure 5 the fifth schematic diagram of the text strengthening method provided in the embodiment of the present application. The overall semantic difference value between the semantic vector of the replacement sentence and the semantic vector of the sentence to be analyzed is calculated, including:

[0075] obtaining the semantic vector of the sentence to be analyzed, and obtaining the semantic vector of the replacement sentence;

[0076] calculating the semantic vector cosine similarity between the semantic vector of the sentence to be analyzed and the semantic vector of the replacement sentence;

[0077] According to the semantic vector cosine similarity, the overall semantic difference value is calculated.

[0078] In the above implementation process, first, a pre-trained language model needs to be selected to convert the text into a vector. Cosine similarity evaluates the similarity in the direction of two vectors by measuring the cosine value of the angle between them, while ignoring their size (modulus). This is very suitable for semantic vectors, because we are more concerned about the direction in the semantic space, rather than the expression strength.

[0079] Optionally, the similarity between the original sentence and the replacement sentence is calculated using a sentence encoder (such as sentence-BERT), and the original sentence is encoded as ; the replacement sentence is encoded as , where . Then the similarity between the original sentence and the replacement sentence is calculated:

[0080]

[0081] Here, is the original sentence S=[ ], is the th character in the sentence, is the length of the sentence, is the set of replacement candidate words, is each candidate word in the set , i.e., the list of all alternative words for replacing , is the number of all candidate words.

[0082] where, usually represents the amount of semantic change or semantic difference, and is used to measure the degree of semantic change before and after a certain operation (such as word replacement). The larger the value, the greater the impact of the target word on the overall sentence semantics.

[0083] In an embodiment of the present application, in the news title optimization scenario, the input text is "Technology giant stock price crash triggers investor panic"; the system analyzes the semantic change impact of "crash". The calculation process is (1) replacing "crash" with "down" has a semantic similarity of 0.45; (2) replacing "crash" with "callback" has a semantic similarity of 0.35. The average similarity is (0.45 + 0.35) / 2 = 0.4; the result is = 1 - 0.4 - 0.6 (high semantic impact); the system feedback is that "crash" contributes the most to the urgency and impact of the news, and it is recommended to be retained.

[0084] Optionally, refer to Figure 6 , Figure 6 is the sixth schematic diagram of the text strengthening method provided in the embodiment of the present application. Based on the semantic vector of the pre-text of the target word and the semantic vector of the post-text, the context coherence difference value is calculated, including:

[0085] The to-be-analyzed sentence is divided into a preceding text segment and a following text segment by the target vocabulary;

[0086] The segment vector of the preceding text segment and the segment vector of the following text segment are obtained;

[0087] The segment vector cosine similarity between the segment vector of the preceding text segment and the segment vector of the following text segment is calculated;

[0088] The context coherence difference value is calculated according to the segment vector cosine similarity.

[0089] In the implementation process described above, the semantic vectors of the to-be-analyzed sentence and the replacement sentence are first generated by the same sentence-level encoder to obtain the vector representations of the two sentences; then the cosine similarity between the two vectors is calculated, and the difference between 1 and the cosine similarity is taken as the overall semantic difference value. If the difference value is close to zero, it means that the meaning of the sentence before and after replacement has hardly changed, and the target word has limited impact on the overall semantics; if the difference value is significantly greater than zero, it means that the semantics of the sentence after replacement has shifted significantly, and the target word plays a key role in maintaining the original meaning and should obtain a higher share in the importance weight; thus, the semantic drift of the sentence is quantified as an important basis for subsequent reward allocation.

[0090] Optionally, first, the context is divided into a preceding text and a following text The preceding text is input into the BERT model, and the [CLS] vector of the last layer is extracted as The following text is input into the BERT model, and the [CLS] vector of the last layer is extracted as The cosine similarity of the two is calculated, and the greater the difference, the higher the penalty.

[0091] In an embodiment of the present application, the input text is "The deep learning algorithm performs excellently. This method has achieved the best results on multiple data sets." The system analyzes the context coherence of "this method", and the calculation process is E[preceding text] = emb("The deep learning algorithm performs excellently"), E[following text] = emb("has achieved the best results on multiple data sets"), cos(precoding text, following text) = 0.9 (highly coherent) - breakage degree = 1 - 0.9 = 0.1 (good coherence). The result is that the breakage penalty is small, and the "method" weight remains high. The system feedback is that "the method" effectively maintains the theme consistency of the preceding and following texts, and is a key referring word.

[0092] Optionally, please refer to Figure 7 , Figure 7 the seventh schematic diagram of the text reinforcement method provided in the embodiment of the present application. The method further comprises:

[0093] The weighted sum of the differences in text fluency, word vectors, overall semantics, and contextual coherence is used to obtain a comprehensive score.

[0094] The overall score is normalized to obtain the importance weights.

[0095] In one embodiment of this application, for characters in a sentence The overall score can be obtained as follows: Let the sentence be S=[ The target word is Its weighting function can be defined as:

[0096]

[0097] in, Indicates replacement word The resulting change in confusion reflects its contribution to sentence fluency. As mentioned above, it embodies its own semantic uniqueness. Indicates the word "will" The amount of semantic change in the entire sentence after replacement.

[0098] From the beginning of the sentence to The contextual representation of all previous words (e.g., the embedding vector of the first half of the sentence encoded using models such as BERT). From the sentence The contextual representation of all words from then until the end. Calculate the semantic coherence or relevance between the first and second halves of a sentence. The value is high, close to 1, even after removal. The two parts of the sentence are still highly related, indicating that... The connection is weak. The value is low, close to 0, indicating that [the element] has been removed. Afterwards, the two parts of the sentence lost their coherence, indicating... It is a key link connecting the preceding and following text.

[0099] In the weight function The weight parameters are normalized. Returning to the value function, This allows for the measurement of the dominant value of individual characters. During model learning, the focus shifts directly to the most dominant token. This learning process enables the model to generate more effective and meaningful responses. For example, given the input "Spring breeze has turned the south bank of the Yangtze River green again, when will the bright moon shine on my return?", the calculation process involves evaluating the character "green" as... =1.6, , , break analysis = 0.1. Weight calculation: Weight ("green") = 0.3 x 1.6 + 0.25 x 0.9 + 0.25 x 0.8 0.2 x 0.1 = 0.885; the system output is the "green" word weight 0.885 (full score 1.0), which is the most important word in the whole sentence, reflecting the poet's unique creativity.

[0100] Through multi-angle analysis at the linguistic level (fluency, lexical semantics, sentence semantics, and context structure), sparse sentence-level rewards are converted into dense lexical-level rewards, so as to enable reinforcement learning to more efficiently focus on learning those truly key "words", and further improve the quality and accuracy of the text generation model.

[0101] Optionally, please refer to Figure 8 , Figure 8 The eighth schematic diagram of the text reinforcement method provided by the embodiments of the present application. According to the target word, a replacement word set is generated, which includes:

[0102] Using a bidirectional mask language model, the target word in the sentence to be analyzed is replaced with a mask mark;

[0103] Through the mask language modeling head of the model, the K candidate words with the highest probability at the mask position are predicted to form a replacement word set; wherein the replacement word set includes the target word itself.

[0104] In the above implementation process, the target word in the sentence to be analyzed S=[ ] is replaced with a [MASK] mark to obtain an input sequence containing a mask; then the sequence is sent into a bidirectional mask language model, the probability distribution of the mask position is output by the mask language modeling head, and the K candidate words with the highest probability are taken to form an initial set; finally, the target word itself is added back to the set. If the target word ranks high in Top-K, it means that it conforms to the model language prior and has low replacement risk; if it does not enter Top-K or ranks low, it indicates that the word has very regular semantics and is more likely to cause sentence drift after replacement, so it is given a higher score in the subsequent importance weight; in this way, recall and benchmark are considered, ensuring the diversity and comparability of the replacement word set.

[0105] In a second aspect, the embodiments of the present application provide a text reinforcement learning device, please refer to Figure 9 , Figure 9 The block schematic diagram of the text reinforcement learning device provided by the embodiments of the present application.

[0106] The device comprises an analysis module 01, a replacement module 02, a calculation module 03, and a determination module 04; the analysis module 01 is used to determine one or more target words in a sentence to be analyzed, and generate a set of replacement words according to the target words; the replacement module 02 is used to replace the target words in the sentence to be analyzed with the current candidate replacement words to obtain a replacement sentence; the calculation module 03 is used to calculate one or more difference indicator values between the replacement sentence and the sentence to be analyzed; wherein the difference indicator values are used to quantify the influence of the replacement of the target words on the original sentence from different dimensions; the determination module 04 is used to determine the importance weight of the target words in the sentence to be analyzed based on the difference indicator values; wherein the importance weight is used to redistribute the rewards of individual characters in the sentence.

[0107] In the above implementation process, by integrating the analysis, replacement, calculation and determination of the four modules, the device can systematically evaluate the semantic and structural contributions of the target words in the sentence. Specifically, the analysis module 01 first locks the words to be investigated and generates a candidate replacement set; the replacement module 02 constructs a new sentence accordingly; the calculation module 03 quantifies the difference before and after replacement from multiple dimensions to form difference indicators; and finally the determination module 04 integrates these indicators to assign an "importance weight" to the target words. This weight can be directly used to redistribute the rewards of individual characters in the sentence, so that the reinforcement learning model focuses more rewards on key characters in the training stage, suppresses noise signals, and thus improves the accuracy and interpretability of downstream tasks such as text generation, text correction or key word extraction.

[0108] Optionally, wherein the difference indicator values include: text fluency difference value, word vector difference value, overall semantic difference value and context coherence difference value; wherein the text fluency difference value is used to represent the change of text fluency before and after replacement; the word vector difference value is used to represent the semantic uniqueness of the word; the overall semantic difference value is used to represent the overall semantic change before and after replacement; and the context coherence difference value is used to represent the context coherence of the position of the target word.

[0109] In the above implementation process, the text fluency difference value can immediately find out whether the replacement causes grammatical errors or awkward reading by means of language model perplexity or fluency scoring, avoiding misleading low-quality replacement sentences for subsequent weight calculation. The word vector difference value quantifies whether there is "irreplaceable exclusive semantics" at this position by calculating the distance between the target word and the candidate word in the distributed representation space, thereby highlighting high-information-entropy, low-frequency but key entity words or professional terms. The overall semantic difference value uses sentence-level embedding or cross-encoder scoring to measure whether the replacement causes semantic drift, preventing "smooth but off-topic" replacement sentences from obtaining a virtual high weight. The context coherence difference value uses the co-occurrence probability of the adjacent window or the reference resolution chain to check the dependency relationship between the target word and the context, ensuring that pronouns, omissions, collocations and other subtle coherence information are not damaged.

[0110] Optionally, refer to Figure 10 , Figure 10 A block schematic diagram of an electronic device provided by an embodiment of the present application is shown. The electronic device 100 can include a memory 111, a storage controller 112, a processor 113, a peripheral interface 114, an input / output unit 115, and a display unit 116. Those skilled in the art can understand that Figure 10 The structure shown is only schematic, and does not limit the structure of the electronic device 100. For example, the electronic device 100 can also include more or fewer components than those shown in the figure, or have a different configuration from that shown in the figure. Figure 10 Figure 10

[0111] The memory 111, the storage controller 112, the processor 113, the peripheral interface 114, the input / output unit 115, and the display unit 116 are directly or indirectly electrically connected to each other to realize data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines. The processor 113 is used to execute the executable modules stored in the memory.

[0112] The memory 111 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), etc. The memory 111 is used to store programs, and the processor 113 executes the programs after receiving execution instructions. The method performed by the electronic device 100 defined by the processes disclosed in any embodiment of the present application can be applied in the processor 113, or implemented by the processor 113.

[0113] ​​The processor 113 can be an integrated circuit chip having a signal processing capability. The processor 113 can be a general purpose processor, including a central processing unit (CPU), a network processor (NP), or the like; or can be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component. The processor 113 can implement or execute the various methods, steps, and logical block diagrams disclosed in the embodiments of the present application. The general purpose processor can be a microprocessor, or any conventional processor, or the like.

[0114] The peripheral interface 114 couples various input / output devices to the processor 113 and the memory 111. In some embodiments, the peripheral interface 114, the processor 113, and the memory controller 112 can be implemented in a single chip. In other embodiments, they can be implemented by independent chips, respectively.

[0115] The input / output unit 115 is configured to provide input data to a user. The input / output unit 115 can be, but is not limited to, a mouse, a keyboard, or the like.

[0116] The display unit 116 provides an interactive interface (e.g., a user operation interface) between the electronic device 100 and a user, or is configured to display image data for a user to refer. In the embodiments, the display unit can be a liquid crystal display or a touch display. If the display unit is a touch display, it can be a capacitive touch screen or a resistive touch screen supporting single-point and multi-point touch operations. The support of single-point and multi-point touch operations means that the touch display can sense a touch operation generated simultaneously from one or more positions on the touch display, and transmit the sensed touch operation to the processor for calculation and processing.

[0117] The embodiments of the present application further provide a computer readable storage medium, which stores computer program instructions. When the computer program instructions are read and executed by a processor, the steps of the text reinforcement learning method are performed.

[0118] To sum up, the application provides a text reinforcement learning method and device, electronic equipment and computer storage medium, relates to the technical field of text reinforcement learning, and is applied to a text generation model. The method comprises the following steps: determining one or more target words in a to-be-analyzed sentence, generating a replacement word set according to the target words; replacing the target words in the to-be-analyzed sentence with a current candidate replacement word to obtain a replacement sentence; calculating one or more difference indicator values between the replacement sentence and the to-be-analyzed sentence; wherein the difference indicator values are used to quantify the influence of the replacement of the target words on the original sentence from different dimensions; determining the importance weight of the target words in the to-be-analyzed sentence based on the difference indicator values; wherein the importance weight is used to redistribute the reward of a single character in a sentence. By introducing the difference indicator values to quantify the influence of the replacement of the target words on the original sentence in multiple dimensions, the semantic, structural or style role of the words in the sentence is more finely evaluated. Compared with the traditional method which only relies on single similarity or gradient information, this multi-dimensional difference measurement can more accurately identify key words and avoid misjudging high-frequency but unimportant words as important. Finally, the calculated importance weight is used to redistribute the character-level reward, so that the reinforcement learning signal is more focused on the part with great semantic contribution, and the training efficiency and effect of the text generation model in controllability, coherence and target orientation are improved.

[0119] In several embodiments provided in the present application, it should be understood that the disclosed devices can also be implemented in other ways. The device embodiments described above are only schematic, for example, the block diagram in the drawings shows the possible implementation architecture, function and operation of the device according to the embodiments of the present application. In this regard, each block in the block diagram can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementation ways, the functions marked in the blocks can also occur in different order from that marked in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram, and the combination of the block diagram, can be implemented by a dedicated hardware-based system for performing the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0120] In addition, the function modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0121] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0122] The above merely provides an example of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application. It should be noted that similar reference numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0123] The above merely provides an example of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application. It should be noted that similar reference numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0124] It should be noted that, in the present document, the relationship terms such as first and second are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the processes, methods, articles or devices including a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the elements defined by the statement "include" do not exclude the presence of other identical elements in the processes, methods, articles or devices including the elements.

Claims

1. A text reinforcement learning method applied to a text generation model, characterized in that, The method comprises: determining one or more target words in a sentence to be analyzed, generating a set of replacement words based on the target words; replacing the target words in the sentence to be analyzed with a current candidate replacement word to obtain a replacement sentence; calculating a plurality of difference indicator values between the replacement sentence and the sentence to be analyzed; wherein the difference indicator values are used to quantify the influence of the replacement of the target words on the original sentence from different dimensions; based on the difference indicator values, determining the importance weight of the target words in the sentence to be analyzed; wherein the importance weight is used to redistribute the rewards of individual characters in the sentence; wherein the calculation of the plurality of difference indicator values between the replacement sentence and the sentence to be analyzed comprises: calculating a text fluency difference value of the replacement sentence relative to the sentence to be analyzed; wherein the text fluency difference value is used to represent the change in text fluency before and after replacement; calculating a word vector difference value between the word vector of the current candidate replacement word and the word vector of the target word; wherein the word vector difference value is used to represent the semantic uniqueness of the word; calculating an overall semantic difference value between the semantic vector of the replacement sentence and the semantic vector of the sentence to be analyzed; wherein the overall semantic difference value is used to represent the overall semantic change before and after replacement; based on the semantic vector of the context before the target word and the semantic vector of the context after the target word, calculating a context coherence difference value; wherein the context coherence difference value is used to represent the context coherence of the position of the target word. wherein the method further comprises: weighting and summing the text fluency difference value, the word vector difference value, the overall semantic difference value and the context coherence difference value to obtain a comprehensive score; normalizing the comprehensive score to obtain the importance weight.

2. The text reinforcement learning method of claim 1, wherein, The calculation of the text fluency difference value of the replacement sentence relative to the sentence to be analyzed comprises: calculating the original sentence perplexity of the sentence to be analyzed, and calculating the average perplexity of all replacement sentences; under the condition of obtaining the original sentence perplexity and the average perplexity, calculating the text fluency difference value; wherein the text fluency difference value is the difference value between the average perplexity and the original sentence perplexity.

3. The text reinforcement learning method of claim 1, wherein, The calculation of the word vector difference value between the word vector of the current candidate replacement word and the word vector of the target word comprises: obtaining the word vector of the target word and obtaining the word vector of the current candidate replacement word; calculating the word vector cosine similarity between the word vector of the target word and the word vector of the current candidate replacement word; under the condition of the word vector cosine similarity, calculating the word vector difference value.

4. The text reinforcement learning method of claim 1, wherein, The calculation of the overall semantic difference value between the semantic vector of the replacement sentence and the semantic vector of the sentence to be analyzed comprises: obtaining the semantic vector of the sentence to be analyzed and obtaining the semantic vector of the replacement sentence; calculating the semantic vector cosine similarity between the semantic vector of the sentence to be analyzed and the semantic vector of the replacement sentence; under the condition of the semantic vector cosine similarity, calculating the overall semantic difference value.

5. The text reinforcement learning method of claim 1, wherein, The context coherence difference value is calculated based on the semantic vector of the context before the target vocabulary and the semantic vector of the context after the target vocabulary, including: The to-be-analyzed sentence is segmented into a context text segment before the target vocabulary and a context text segment after the target vocabulary by taking the target vocabulary as a boundary; A segment vector of the context text segment before the target vocabulary and a segment vector of the context text segment after the target vocabulary are obtained; A segment vector cosine similarity between the segment vector of the context text segment before the target vocabulary and the segment vector of the context text segment after the target vocabulary is calculated; The context coherence difference value is calculated according to the segment vector cosine similarity.

6. A text reinforcement learning apparatus characterized by comprising: The device includes an analysis module, a replacement module, a calculation module, and a determination module; The analysis module is configured to determine one or more target vocabularies in a to-be-analyzed sentence, and generate a replacement vocabulary set according to the target vocabularies; The replacement module is configured to replace the target vocabularies in the to-be-analyzed sentence with a current candidate replacement vocabulary to obtain a replacement sentence; The calculation module is configured to calculate a plurality of difference indicator values between the replacement sentence and the to-be-analyzed sentence; wherein the difference indicator values are used to quantify the influence of the replacement of the target vocabulary on the original sentence from different dimensions; The determination module is configured to determine the importance weight of the target vocabulary in the to-be-analyzed sentence based on the difference indicator values; wherein the importance weight is used to redistribute the rewards of individual characters in a sentence; The calculation module is specifically configured to: calculate a text fluency difference value of the replacement sentence relative to the to-be-analyzed sentence; wherein the text fluency difference value is used to represent the change in text fluency before and after replacement; calculate a word vector difference value between the word vector of the current candidate replacement vocabulary and the word vector of the target vocabulary; wherein the word vector difference value is used to represent the semantic uniqueness of the vocabulary; calculate an overall semantic difference value between the semantic vector of the replacement sentence and the semantic vector of the to-be-analyzed sentence; wherein the overall semantic difference value is used to represent the overall semantic change before and after replacement; calculate a context coherence difference value based on the semantic vector of the context before the target vocabulary and the semantic vector of the context after the target vocabulary; wherein the context coherence difference value is used to represent the context coherence of the position of the target vocabulary; The determination module is specifically configured to: perform weighted summation on the text fluency difference value, the word vector difference value, the overall semantic difference value, and the context coherence difference value to obtain a comprehensive score; perform normalization processing on the comprehensive score to obtain the importance weight.

7. An electronic device, comprising: The electronic device includes a memory and a processor, the memory stores program instructions, and the processor executes the program instructions to perform the steps in the method of any one of claims 1-5.

8. A computer storage medium, characterized in that The computer storage medium stores computer program instructions, and the computer program instructions are executed by a processor to perform the steps in the method of any one of claims 1-5.

Images