Government affair group relative strategy optimization method and device, equipment and medium

Abstract

The invention discloses a government affair group relative strategy optimization method and device, equipment and a medium. The method comprises the following steps: constructing a government affair sensitive word library and a training set comprising a plurality of government affair questions; determining a response set corresponding to the training set by using a natural language model and a government affair sensitive word library; determining a target type reward corresponding to each reply in the response set, and determining an advantage corresponding to each reply according to the target type reward; wherein the target type reward comprises a security reward, and the security reward is used for evaluating the security of the reply content; updating the natural language model according to the advantage corresponding to each reply and the KL divergence penalty deviation; wherein the KL divergence penalty deviation is established based on the basic reference model and the safety reference model. According to the technical scheme, on the premise of not depending on external capability and not additionally increasing the model training calculation amount as much as possible, multiple safety protection is carried out in the model training process, and the safety and reliability of the model output content are improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, equipment and medium for optimizing relative strategies for government groups. Background Technology

[0002] Group Relative Policy Optimization (GRPO), an innovative reinforcement learning algorithm, can be used to improve the reasoning ability of natural language models. It optimizes the policy by comparing samples within a group, avoiding the reliance on absolute reward values ​​and independent value models found in traditional methods. This reduces computational costs while improving model training efficiency and stability.

[0003] Due to its unique nature, the government sector has high security requirements. The current mainstream approach to applying the GRPO method in government involves privately deploying a natural language model within a local or departmental government cloud and network environment, and then building upper-layer applications on top of the model. For security reasons, the security of the model's output is usually controlled. For example, the output may contain sensitive words.

[0004] However, the above methods lack security safeguards during the GRPO-based model training process. This means that even after training, the model still requires additional security measures to perform real-time monitoring of its output, increasing resource consumption. Furthermore, GRPO's inherent intra-group evaluation model lacks an independent security assessment mechanism, making it difficult to effectively identify generated content with hidden risks and potentially exacerbating security vulnerabilities. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and medium for optimizing relative strategies for government groups. It does not rely on external capabilities and implements multiple security protections during model training without increasing the computational load of model training as much as possible, thereby improving the security and reliability of the model output.

[0006] According to one aspect of the present invention, a method for optimizing relative strategies of government groups is provided, the method comprising:

[0007] Construct a database of sensitive terms related to government affairs and a training set; wherein the training set includes multiple government-related questions;

[0008] The response set corresponding to the training set is determined using a natural language model and the aforementioned government sensitive word database; wherein, each government question corresponds to a response set, and each response set includes multiple responses;

[0009] Determine the target type reward for each response in the response set, and determine the advantage corresponding to each response based on the target type reward; wherein, the target type reward includes a security reward, which is used to evaluate the security of the response content;

[0010] The natural language model is updated based on the strength and KL divergence penalty bias corresponding to each response; wherein the KL divergence penalty bias is established based on a base reference model and a safety reference model, the base reference model is used to guide the overall training effect of the model, and the safety reference model is used to guide the safety of model training.

[0011] According to another aspect of the present invention, a relative strategy optimization device for government groups is provided, the device comprising:

[0012] The data creation module is used to construct a database of sensitive government terms and a training set; wherein, the training set includes multiple government-related questions;

[0013] The response set determination module is used to determine the response set corresponding to the training set using a natural language model and the government sensitive word database; wherein, each government question corresponds to a response set, and each response set includes multiple responses;

[0014] The response advantage determination module is used to determine the target type reward corresponding to each response in the response set, and to determine the advantage corresponding to each response based on the target type reward; wherein, the target type reward includes a security reward, and the security reward is used to evaluate the security of the response content;

[0015] The model update module is used to update the natural language model based on the advantage and KL divergence penalty bias corresponding to each response; wherein the KL divergence penalty bias is established based on a base reference model and a safety reference model, the base reference model is used to guide the overall training effect of the model, and the safety reference model is used to guide the safety of model training.

[0016] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0017] At least one processor; and,

[0018] A memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the relative strategy optimization method for government groups as described in any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the government group relative strategy optimization method according to any embodiment of the present invention.

[0021] The technical solution of this invention first constructs a government affairs sensitive word database and a training set; the training set includes multiple government affairs questions; then, a response set corresponding to the training set is determined using a natural language model and the government affairs sensitive word database; each government affairs question corresponds to a response set, and each response set includes multiple replies; furthermore, the target type reward corresponding to each reply in the response set is determined, and the advantage corresponding to each reply is determined based on the target type reward; the target type reward includes a security reward, which is used to evaluate the security of the reply content; finally, the natural language model is updated based on the advantage corresponding to each reply and the KL divergence penalty bias; the KL divergence penalty bias is established based on a base reference model and a security reference model, the base reference model is used to guide the overall training effect of the model, and the security reference model is used to guide the security of the model training. This technical solution does not rely on external capabilities and, without increasing the computational load of model training as much as possible, achieves multiple security protections during model training by improving three aspects: detecting and filtering government affairs sensitive words in the model's input and replies, adding a security reward evaluation mechanism in the reward calculation stage, and setting a security reference model in the KL divergence calculation stage, thereby improving the security and reliability of the model's output content.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of a relative strategy optimization method for government groups provided by an embodiment of the present invention;

[0025] Figure 2 This is a schematic diagram of a reference distance metric provided according to an embodiment of the present invention;

[0026] Figure 3 This is a schematic diagram of a relative strategy optimization method for government groups provided by an embodiment of the present invention;

[0027] Figure 4 This is a flowchart of another method for optimizing relative strategies for government groups according to an embodiment of the present invention;

[0028] Figure 5 This is a schematic diagram of the structure of a government affairs group relative strategy optimization device provided in an embodiment of the present invention;

[0029] Figure 6 This is a schematic diagram of the structure of an electronic device that implements a relative strategy optimization method for government groups according to an embodiment of the present invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first," "second," "target," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] Example 1

[0033] Figure 1 This is a flowchart of a relative strategy optimization method for government groups provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where multiple security protections are implemented for the model output during model training. This method can be executed by a relative strategy optimization device for government groups, which can be implemented in hardware and / or software. This device can be configured in an electronic device with data processing capabilities. Figure 1 As shown, the method includes:

[0034] S110, construct a database of sensitive government terms and a training set; the training set includes multiple government-related questions.

[0035] In this embodiment, the first step is to construct a sensitive term database containing multiple sensitive terms in the government affairs field and a training set including multiple government affairs questions. Specifically, sensitive terms in the government affairs field can be collected through the following four channels: First, extraction from legal and policy documents. Second, acquisition through integration with the systems of various business departments. For example, integrating with the business systems of the finance department and the public security department to obtain sensitive business terms. Third, capture of public opinion dynamics. For example, by constructing web crawlers to dynamically collect sensitive terms related to government affairs in social media and forums in real time. Fourth, multimodal data governance. For example, identifying sensitive areas in government affairs image files through OCR; collecting sensitive expressions in dialects through speech recognition.

[0036] S120, using a natural language model and a government sensitive word database to determine the response set corresponding to the training set; where each government question corresponds to a response set, and each response set includes multiple replies.

[0037] Natural language models (NLPs) are core technologies in the field of artificial intelligence used to process, understand, and generate human language. They learn the rules of language through statistical or deep learning methods, thereby enabling automatic analysis, generation, and interaction of text. For example, a NLP can be a Large Language Model (LLM), such as the DeepSeek-R1 model. It should be noted that GRPO provides a set of responses for each input. In cases where insecure government policy questions are input, it may amplify insecure signals in the training data, potentially guiding the model to train in the wrong direction.

[0038] In this embodiment, to improve the security and reliability of model training, after constructing the government sensitive word database and training set, a response set corresponding to the training set is determined using a natural language model and the government sensitive word database. This allows for the detection and filtering of government sensitive words in the input and responses of the natural language model. Optionally, determining the response set corresponding to the training set using a natural language model and the government sensitive word database includes: detecting each government question in the training set based on the government sensitive word database, removing government questions involving sensitive words from the training set to obtain a reference set; generating a response set corresponding to each government question in the reference set using a natural language model, and detecting each response in the response set based on the government sensitive word database; if a response containing sensitive words exists in the response set, a new response set is generated for the government question corresponding to the response containing sensitive words.

[0039] Specifically, for the training set Each government affairs question The system will detect sensitive government terms in a database. If any such term is found, the government-related question will be flagged. The invalidated government questions will be removed from the training set. The reference set can be obtained by removing elements from the set. For example, the DeepSeek-R1 model (671B) can be selected as the natural language model. Each government query in the reference set... Input into the natural language model in the current state In this process, a response set corresponding to each government inquiry is generated. .in, This indicates the number of responses in the response set. Indicates the first One reply. Regarding Each reply is checked against a database of sensitive government terms. If any of these terms are used, then... All replies are invalidated, and the natural language model is restarted from its current state. generate One reply.

[0040] S130, determine the target type reward for each response in the response set, and determine the advantage of each response based on the target type reward; wherein, the target type reward includes a security reward, which is used to evaluate the security of the response content.

[0041] For example, target-type rewards can include accuracy rewards, formatting rewards, labeling rewards, and security rewards. Accuracy rewards can be used to evaluate whether the model's response is correct. For instance, in a math problem with a definite answer, the model must provide the final result in a specified format to verify the answer's correctness according to predefined rules. Formatting rewards can be used to evaluate the reasonableness of the model's response format. For example, in reasoning, thought processes need to be strictly contained within a specific format. Labeling rewards can be used to evaluate the reasonableness of the key labels in the model's output. For example, in the output content... <think / > ,<end_of_sentence / > The presence and frequency of certain tags are considered. Security rewards can be used to assess the security of responses, such as whether they contain classified, politically sensitive, illegal, or unethical content. It's important to note that traditional GRPO relies on a relative evaluation mechanism within groups, incorporating only accuracy, formatting, and tagging rewards in the reward calculation. This lack of an independent security assessment mechanism makes it difficult to effectively identify generated content with hidden risks.

[0042] In this embodiment, to further improve the security and reliability of model training, a security reward evaluation mechanism is added specifically to the reward calculation stage, thereby realizing the security assessment of the response content. Optionally, determining the target type reward corresponding to each response in the response set includes: determining the reference distance corresponding to each response in the response set based on the government sensitive word library; wherein, the reference distance is used to characterize the closest distance between the response and all sensitive words in the government sensitive word library; determining the security value corresponding to each response based on the reference distance and a preset danger threshold, and using the security value as the security reward.

[0043] Figure 2 This is a schematic diagram of a reference distance measurement principle provided in Embodiment 1 of the present invention. Figure 2 The dangerous area in the text refers to the entire database of sensitive government terms, while the dangerous element refers to a specific sensitive government term. Indicates a preset danger threshold; Indicates the current reply The nearest distance (i.e., reference distance) to the high-dimensional semantic space of the danger zone can be specifically represented as: ,in, Indicates a danger zone. This indicates a hazardous element. For example, a preset hazardous threshold can be set to 0.5.

[0044] Furthermore, the security reward can be defined as the response of a corresponding security value, where the security value... Specifically, it can be expressed as: Understandably, when When the distance between the current response and the nearest sensitive word in the government affairs sensitive word database exceeds the preset danger threshold, it indicates that the current response poses no risk, and the corresponding safety value is [value missing]. .

[0045] This solution adds a security reward mechanism to the reward calculation process by using the nearest distance from the model to the high-dimensional semantic space of the dangerous region to measure the security reward, thus avoiding the increase in computational complexity of GRPO due to the use of complex security detection mechanisms.

[0046] In this embodiment, after obtaining multiple target type rewards for each response, the advantage corresponding to each response can be determined based on these multiple target type rewards. Optionally, determining the advantage corresponding to each response based on target type rewards includes: summing the accuracy reward, formatting reward, tagging reward, and security reward to obtain the comprehensive reward for each response; and determining the advantage corresponding to each response based on the comprehensive reward for each response and the average comprehensive reward of all responses in the response set to which the response belongs.

[0047] Specifically, for each reply First, the accuracy reward, formatting reward, tagging reward, and security reward calculated above are summed to obtain the comprehensive reward for each response. Then, the standard deviation of the overall reward for each reply relative to the average overall reward of all replies in the response set to which the reply belongs is calculated, which is taken as the advantage of each reply, and can be specifically expressed as follows: .in, In reply Advantages In response Calculated comprehensive reward, The average overall reward corresponding to the response set. This represents the standard deviation of the overall reward corresponding to the response set.

[0048] S140, the natural language model is updated based on the advantage and KL divergence penalty bias corresponding to each response; wherein, the KL divergence penalty bias is established based on the base reference model and the safety reference model, the base reference model is used to guide the overall training effect of the model, and the safety reference model is used to guide the safety of model training.

[0049] Using KL divergence as a penalty term ensures that the updated model does not deviate too far from the reference model. The penalty term calculates the distance between the current natural language model and the reference model. It should be noted that traditional GRPO only has one reference model and lacks a separate, security-related reference model, resulting in insufficient security guidance during model training. This invention expands the reference model to two types of domain reference models: one is a high-quality basic reference model for the government domain, fine-tuned from government corpora. One type can be used to guide the overall training effect of the natural language model in the current state; the other type is a high-security government domain security reference model after fine-tuning with government corpus. This can be used to guide the safety of natural language model training in the current state. The safety reference model and the base reference model each have their own functions but also cooperate with each other to guide the training of the current model.

[0050] In this embodiment, optionally, the process of determining the KL divergence penalty bias includes: determining multiple base reference models of different scales and multiple security reference models of different scales; determining a first penalty term based on the average KL divergence penalty bias of the natural language model relative to each base reference model; determining a second penalty term based on the average KL divergence penalty bias of the natural language model relative to each security reference model; and weighted summing the first penalty term and the second penalty term to obtain the KL divergence penalty bias corresponding to the natural language model.

[0051] For example, the base reference model can be and security reference model All set to Several models covering different scales, such as models with different parameter magnitudes (10B, 50B, 100B, etc.). Calculate the natural language models separately. Relative to each base reference model ( ,in KL divergence penalty bias And by dividing by the model parameter size to eliminate the effect of different model sizes, then calculate... indivual The average value is used as the first penalty term, which can be specifically expressed as: .in, Basic reference model The number of parameters, This is the first penalty item.

[0052] Similarly, we first compute the natural language models separately. Relative to each security reference model ( ,in KL divergence penalty bias And by dividing by the model parameter size to eliminate the effect of different model sizes, then calculate... indivual The average value is used as the second penalty term, specifically expressed as follows: .in, For safety reference model The number of parameters, This is the second penalty item. Therefore, the first penalty item can be... With the second penalty item We perform a weighted summation to obtain the natural language model. Corresponding KL divergence penalty bias Specifically, it can be expressed as: .in, and These are the weighting coefficients for the first penalty term and the second penalty term, respectively. By default, .

[0053] After obtaining the advantage and KL divergence penalty bias corresponding to each response, the objective function can be obtained by estimating the expectation of the advantage corresponding to each response based on the KL divergence penalty bias. The natural language model can then be updated by iteratively optimizing the objective function.

[0054] Figure 3 This is a schematic diagram of a relative strategy optimization method for government groups provided in Embodiment 1 of the present invention. Figure 3As shown, the training set is first analyzed using the constructed government sensitive word database. Each government affairs question Sensitive word detection and filtering are performed to remove government-related questions containing sensitive words from the training set. The remaining government-related questions are then removed from the list. The remaining questions are then input into the natural language model in the current state. In the process, each government inquiry is generated. Corresponding response set The response set includes Each response in the response set is then processed using a government sensitive word database. If a response contains a sensitive word, all responses in the response set to that response are invalidated. The current state of the natural language model is then utilized. Regenerate a new set of responses. Then, for each response in the response set, calculate multiple target type rewards (including accuracy reward, formatting reward, tagging reward, and security reward), and sum the rewards for each target type to obtain the comprehensive reward for each response. Then, the advantage of each response is obtained through within-group advantage calculation. Furthermore, a KL divergence penalty bias is established based on the fundamental reference model and the security reference model, and based on the advantage corresponding to each response... KL divergence penalty bias on the natural language model in the current state Update.

[0055] Compared with existing technologies, this invention has the following advantages: 1. Improved algorithm security: This invention adds a security assessment mechanism to multiple stages, including government question input and model response, reward calculation, and KL divergence calculation, making the algorithm's output credible and reliable. It is important to note that the security assessment mechanism proposed in this invention operates during model training, rather than relying on the security of the input training data or the output content for security detection and handling, further reducing the model's dependence on external security capabilities. 2. Maximum preservation of algorithm computational complexity: In this invention, the security assessment in the government question input, model response generation, and model response reward calculation stages primarily relies on the detection of sensitive government terms. When sensitive government terms are detected, the response generation or reward scoring is restarted without introducing complex calculation rules. This avoids GRPO losing its computational efficiency advantage compared to algorithms like PPO (Proximal Policy Optimization), thus preserving model training efficiency.

[0056] The technical solution of this invention first constructs a government affairs sensitive word database and a training set; wherein, the training set includes multiple government affairs questions; then, a response set corresponding to the training set is determined using a natural language model and the government affairs sensitive word database; wherein, each government affairs question corresponds to a response set, and each response set includes multiple replies; furthermore, the target type reward corresponding to each reply in the response set is determined, and the advantage corresponding to each reply is determined based on the target type reward; wherein, the target type reward includes a security reward, which is used to evaluate the security of the reply content; finally, the natural language model is updated based on the advantage corresponding to each reply and the KL divergence penalty bias; wherein, the KL divergence penalty bias is established based on a base reference model and a security reference model, the base reference model is used to guide the overall training effect of the model, and the security reference model is used to guide the security of the model training. This technical solution, without relying on external capabilities, achieves multiple security protections during model training by improving three aspects: detecting and filtering government-sensitive words in the model's input and response, adding a security reward evaluation mechanism in the reward calculation stage, and setting a security reference model in the KL divergence calculation stage. This fundamentally solves the low security problem of the GRPO algorithm and greatly improves the security and reliability of the model's output content.

[0057] Example 2

[0058] Figure 4 This is a flowchart of a relative strategy optimization method for government groups provided in Embodiment 2 of the present invention. This embodiment is based on the above embodiment and optimized accordingly. Specifically, the optimization involves updating the natural language model based on the advantage and KL divergence penalty bias corresponding to each response, including: determining the model change based on the current state's natural language model and the natural language model of the previous round's state; if the model change exceeds a preset range, then limiting the advantage update scale to the preset range based on the clip function; and updating the natural language model based on the advantage, advantage update scale, and KL divergence penalty bias corresponding to each response.

[0059] like Figure 4 As shown, the method in this embodiment specifically includes the following steps:

[0060] S210, construct a database of sensitive government terms and a training set; the training set includes multiple government-related questions.

[0061] S220: Use natural language models and government sensitive word databases to determine the response set corresponding to the training set; where each government question corresponds to a response set, and each response set includes multiple replies.

[0062] S230, determine the target type reward for each response in the response set, and determine the advantage of each response based on the target type reward; wherein, the target type reward includes a security reward, which is used to evaluate the security of the response content.

[0063] The specific implementation methods of S210-S230 can be referred to the relevant descriptions in the above embodiments, and will not be repeated here.

[0064] S240, determine the model changes based on the natural language model of the current state and the natural language model of the previous state.

[0065] For example, it can be done through formula Calculate the natural language model in the current state Compared to the natural language model in the previous round Model changes This ratio (i.e.) This can be used to measure questions raised regarding current government affairs. Natural language model in the current state generate The probability compared to the natural language model in the previous round The changes.

[0066] S250, if the model changes beyond the preset range, the advantage update scale will be limited to the preset range based on the clip function.

[0067] Specifically, if the model changes beyond the preset range, it indicates that the advantage update is too large, which may lead to instability in the model training process. In this case, it is necessary to limit the advantage update scale based on the clip function, and the limiting rules are as follows: .in, To preset the range of variation, It is an extremely small number, such as 0.0005. That is, it is achieved through the `clip` function. Update scale is limited to Its purpose is to prevent drastic changes from causing the model to collapse. This ensures that new strategies are improved gradually, rather than through radical updates.

[0068] S260 updates the natural language model based on the strength, strength update scale, and KL divergence penalty bias corresponding to each response.

[0069] In this embodiment, after determining the strength and KL divergence penalty corresponding to each response and limiting the strength update scale to a preset range, the natural language model can be updated based on the strength, strength update scale, and KL divergence penalty bias corresponding to each response. Optionally, updating the natural language model based on the strength, strength update scale, and KL divergence penalty bias corresponding to each response includes: estimating the expected strength of each response based on the strength update scale and KL divergence penalty bias to obtain an objective function; and iteratively optimizing the objective function using gradient descent until the objective function meets the iteration stopping condition.

[0070] For example, the objective function It can be represented as follows:

[0071] ;

[0072] in, It is an extremely small number, such as 0.05. The objective function is iteratively optimized using gradient descent. : ,until Decrease to an acceptable range (e.g.) At this point, it can be determined that the iteration stopping condition is met. and They represent and The values ​​of all trainable parameters at two time points, It is the learning rate, taken as , Indicates one iteration. This represents the gradient.

[0073] The technical solution of this invention, when updating the natural language model based on the advantage and KL divergence penalty bias corresponding to each response, first determines the model change based on the current state of the natural language model and the natural language model of the previous state; if the model change exceeds a preset range, the advantage update scale is limited to the preset range based on the clip function; then, the natural language model is updated based on the advantage, advantage update scale, and KL divergence penalty bias corresponding to each response. This technical solution, by limiting the advantage update scale to the preset range based on the clip function when the model change exceeds the preset range, effectively prevents drastic changes from causing model collapse, thereby ensuring that the new strategy is gradually improved rather than undergoing aggressive updates.

[0074] Example 3

[0075] Figure 5This is a schematic diagram of a relative strategy optimization device for government groups provided in Embodiment 3 of the present invention. This device can execute the relative strategy optimization method for government groups provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. For example... Figure 5 As shown, the device includes:

[0076] The data creation module 310 is used to construct a database of sensitive government terms and a training set; wherein, the training set includes multiple government-related questions;

[0077] The response set determination module 320 is used to determine the response set corresponding to the training set using a natural language model and the government sensitive word database; wherein, each government question corresponds to a response set, and each response set includes multiple responses;

[0078] The response advantage determination module 330 is used to determine the target type reward corresponding to each response in the response set, and to determine the advantage corresponding to each response based on the target type reward; wherein, the target type reward includes a security reward, and the security reward is used to evaluate the security of the response content;

[0079] The model update module 340 is used to update the natural language model according to the advantage and KL divergence penalty bias corresponding to each response; wherein the KL divergence penalty bias is established based on the base reference model and the safety reference model, the base reference model is used to guide the overall training effect of the model, and the safety reference model is used to guide the safety of model training.

[0080] Optionally, the response set determination module 320 is specifically used for:

[0081] The government affairs sensitive word database is used to detect each government affairs question in the training set, and the government affairs questions involving sensitive words are removed from the training set to obtain a reference set;

[0082] The natural language model is used to generate a response set corresponding to each government inquiry in the reference set, and the responses in the response set are detected according to the government sensitive word database.

[0083] If the response set contains responses involving sensitive words, a new response set will be generated for the government affairs questions corresponding to the responses involving sensitive words.

[0084] Optionally, the response advantage determination module 330 is used for:

[0085] The reference distance for each response in the response set is determined based on the government sensitive word database; wherein, the reference distance is used to characterize the nearest distance between the response and all sensitive words in the government sensitive word database;

[0086] The safety value corresponding to each response is determined based on the reference distance and the preset danger threshold, and the safety value is used as a safety reward.

[0087] Optionally, the target type reward may also include accuracy reward, formatting reward, and tag reward;

[0088] Accordingly, the response advantage determination module 330 is also used for:

[0089] The accuracy reward, formatting reward, tagging reward, and security reward are summed to obtain the comprehensive reward for each response;

[0090] The advantage of each reply is determined based on the overall reward corresponding to each reply and the average overall reward of all replies in the response set to which the reply belongs.

[0091] Optionally, the model update module 340 is used for:

[0092] Determine the model changes based on the current state's natural language model and the natural language model from the previous state;

[0093] If the changes in the model exceed the preset range, the advantage update scale will be limited to the preset range based on the clip function.

[0094] The natural language model is updated based on the advantage corresponding to each response, the advantage update scale, and the KL divergence penalty bias.

[0095] Optionally, the device further includes: a deviation determination module, used for:

[0096] Establish multiple basic reference models of different scales and multiple security reference models of different scales;

[0097] The first penalty term is determined based on the mean KL divergence penalty deviation of the natural language model relative to each of the base reference models;

[0098] The second penalty term is determined based on the mean KL divergence penalty deviation of the natural language model relative to each of the security reference models;

[0099] The weighted sum of the first penalty term and the second penalty term yields the KL divergence penalty bias corresponding to the natural language model.

[0100] Optionally, the model update module 340 is further configured to:

[0101] The objective function is obtained by estimating the expected advantage of each response based on the advantage update scale and KL divergence penalty bias.

[0102] The objective function is iteratively optimized using gradient descent until the objective function meets the iteration stopping condition.

[0103] The relative strategy optimization device for government groups provided in this embodiment of the invention can execute the relative strategy optimization method for government groups provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0104] Example 4

[0105] Figure 6 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0106] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0107] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0108] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the relative policy optimization method for government groups.

[0109] In some embodiments, the relative strategy optimization method for government groups can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the relative strategy optimization method for government groups described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to execute the relative strategy optimization method for government groups by any other suitable means (e.g., by means of firmware).

[0110] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0111] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0112] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0113] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0114] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0115] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0116] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0117] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A relative strategy optimization method for government groups, characterized in that, The method includes: Construct a database of sensitive terms related to government affairs and a training set; wherein, the training set includes multiple government-related questions; The response set corresponding to the training set is determined using a natural language model and the aforementioned government sensitive word database; wherein, each government question corresponds to a response set, and each response set includes multiple responses; Determine the target type reward for each response in the response set, and determine the advantage corresponding to each response based on the target type reward; wherein, the target type reward includes a security reward, which is used to evaluate the security of the response content; The natural language model is updated based on the strength and KL divergence penalty bias corresponding to each response; wherein the KL divergence penalty bias is established based on a base reference model and a safety reference model, the base reference model is used to guide the overall training effect of the model, and the safety reference model is used to guide the safety of model training.

2. The method according to claim 1, characterized in that, Determining the response set corresponding to the training set using a natural language model and the aforementioned government sensitive word database includes: The government affairs sensitive word database is used to detect each government affairs question in the training set, and the government affairs questions involving sensitive words are removed from the training set to obtain a reference set; The natural language model is used to generate a response set corresponding to each government inquiry in the reference set, and the responses in the response set are detected according to the government sensitive word database. If the response set contains responses involving sensitive words, a new response set will be generated for the government affairs questions corresponding to the responses involving sensitive words.

3. The method according to claim 1, characterized in that, Determine the target type reward for each response in the response set, including: The reference distance for each response in the response set is determined based on the government sensitive word database; wherein, the reference distance is used to characterize the nearest distance between the response and all sensitive words in the government sensitive word database; The safety value corresponding to each response is determined based on the reference distance and the preset danger threshold, and the safety value is used as a safety reward.

4. The method according to claim 1 or 3, characterized in that, The target type rewards also include accuracy rewards, formatting rewards, and tagging rewards; Accordingly, the advantage corresponding to each response is determined based on the target type reward, including: The accuracy reward, formatting reward, tagging reward, and security reward are summed to obtain the comprehensive reward for each response; The advantage of each reply is determined based on the overall reward corresponding to each reply and the average overall reward of all replies in the response set to which the reply belongs.

5. The method according to claim 1, characterized in that, The natural language model is updated based on the strength and KL divergence penalty bias corresponding to each response, including: Determine the model changes based on the current state's natural language model and the natural language model from the previous state; If the changes in the model exceed the preset range, the advantage update scale will be limited to the preset range based on the clip function. The natural language model is updated based on the advantage corresponding to each response, the advantage update scale, and the KL divergence penalty bias.

6. The method according to claim 1 or 5, characterized in that, The process for determining the KL divergence penalty bias includes: Establish multiple basic reference models of different scales and multiple security reference models of different scales; The first penalty term is determined based on the mean KL divergence penalty deviation of the natural language model relative to each of the base reference models; The second penalty term is determined based on the mean KL divergence penalty deviation of the natural language model relative to each of the security reference models; The weighted sum of the first penalty term and the second penalty term yields the KL divergence penalty bias corresponding to the natural language model.

7. The method according to claim 5, characterized in that, The natural language model is updated based on the strength of each response, the strength update scale, and the KL divergence penalty bias, including: The objective function is obtained by estimating the expected advantage of each response based on the advantage update scale and KL divergence penalty bias. The objective function is iteratively optimized using the gradient descent method until the objective function meets the iteration stopping condition.

8. A relative strategy optimization device for government groups, characterized in that, The device includes: The data creation module is used to construct a database of sensitive government terms and a training set; wherein, the training set includes multiple government-related questions; The response set determination module is used to determine the response set corresponding to the training set using a natural language model and the government sensitive word database; wherein, each government question corresponds to a response set, and each response set includes multiple responses; The response advantage determination module is used to determine the target type reward corresponding to each response in the response set, and to determine the advantage corresponding to each response based on the target type reward; wherein, the target type reward includes a security reward, and the security reward is used to evaluate the security of the response content; The model update module is used to update the natural language model based on the advantage and KL divergence penalty bias corresponding to each response; wherein the KL divergence penalty bias is established based on a base reference model and a safety reference model, the base reference model is used to guide the overall training effect of the model, and the safety reference model is used to guide the safety of model training.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the relative strategy optimization method for government groups as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the relative strategy optimization method for government groups as described in any one of claims 1-7.

Images