Document generation quality optimization method and device, equipment and storage medium
By combining reinforcement learning training with GRPO and DPO algorithms and the Athene-RM-8B data scoring component, the problems of insufficient understanding and speed limitation in document generation quality assessment and optimization are solved, achieving efficient and fast document quality optimization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA PING AN PROPERTY INSURANCE CO LTD
- Filing Date
- 2026-01-13
- Publication Date
- 2026-05-08
AI Technical Summary
Existing technologies suffer from insufficient text topic comprehension and limited optimization speed in document generation quality assessment and feedback optimization, resulting in a slow document quality optimization process and unsatisfactory optimization results.
A pre-trained document generation quality assessment model is used, which is trained by reinforcement learning using GRPO and DPO algorithms. Through the combination of online and offline reinforcement learning algorithms and the Athene-RM-8B data scoring component, efficient optimization of the generated documents is achieved.
It enables high-quality and rapid optimization of generated documents, ensuring that the final document meets preset requirements and improving the efficiency and effectiveness of document generation quality assessment and optimization.
Smart Images
Figure CN121998663A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology and is applied to scenarios involving the evaluation and optimization of the generation quality of target documents. It relates to a document generation quality optimization method, apparatus, device, and storage medium. Background Technology
[0002] With breakthroughs in artificial intelligence technology, the automatic generation of financial business documents based on large language models has entered a practical stage. However, significant shortcomings remain in the content quality assessment and feedback optimization mechanisms for the generated documents, which directly restricts the quality of the final document content. Existing assessment and optimization systems mainly employ the following four methods: n-gram-based similarity algorithms, deep learning semantic matching models, intelligent assessment systems based on large models, and traditional manual assessment methods.
[0003] However, these current evaluation and optimization methods still have shortcomings when optimizing the generation quality of target documents, such as insufficient understanding of text topics and limited speed of text content optimization. This results in a slow document quality optimization process and less than ideal optimization results. Summary of the Invention
[0004] The purpose of this application is to provide a document generation quality optimization method, apparatus, device, and storage medium to achieve high-quality and rapid optimization of generated documents, so that the final output document meets the corresponding requirements.
[0005] In a first aspect, embodiments of this application provide a document generation quality optimization method, which adopts the following technical solution: A method for optimizing document generation quality includes the following steps: Obtain the generated document output by the preset document generation model; The generated document is then input into a pre-trained document generation quality assessment model; Obtain the evaluation results output by the document generation quality assessment model; Determine whether the evaluation result meets the preset requirements; If the preset requirements are not met, the generated document is optimized according to the preset generation quality optimization strategy and the evaluation results to obtain an optimized document. The optimized document is then used to replace the generated document to repeat the document generation quality evaluation and optimization operation. The final document will be output once the latest evaluation results meet the preset requirements.
[0006] Secondly, embodiments of this application also provide a document generation quality optimization device, which adopts the following technical solution: A document generation quality optimization device, comprising: The document generation module is used to obtain the generated document output by the preset document generation model; An evaluation model input module is used to input the generated document into a pre-trained document generation quality evaluation model; The evaluation result acquisition module is used to acquire the evaluation results output by the document generation quality evaluation model; The requirement condition judgment module is used to determine whether the evaluation result meets the preset requirement conditions; The document duplication optimization module is used to optimize the generated document according to the preset generation quality optimization strategy and the evaluation result if the preset requirements are not met, so as to obtain an optimized document and use the optimized document to replace the generated document to repeatedly perform document generation quality evaluation and optimization operations. The final document output module is used to output the final document until the latest evaluation results meet the preset requirements.
[0007] Thirdly, embodiments of this application also provide a computer device that adopts the technical solution described below: A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the document generation quality optimization method described above.
[0008] Fourthly, embodiments of this application also provide a computer-readable storage medium, which adopts the technical solutions described below: A computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the steps of the document generation quality optimization method described above.
[0009] Compared with the prior art, the embodiments of this application have the following main advantages: The document generation quality optimization method described in this application can be widely applied to scenarios involving quality assessment and optimization of generated target documents. It involves obtaining the generated document output by a preset document generation model; inputting it into a document generation quality assessment model; obtaining the assessment result; determining whether the assessment result meets preset requirements; if not, optimizing the generated document according to a preset quality optimization strategy and the assessment result to obtain an optimized document; and repeatedly performing document generation quality assessment and optimization operations with the optimized document instead of the generated document until the latest assessment result meets the preset requirements, at which point the final document is output. By performing quality assessment and optimization operations on the generated document, the final document is guaranteed to meet the corresponding document quality requirements. Applying this document generation quality optimization method to financial business scenarios can assist in quality auditing during the generation of financial contract documents; applying it to customer service script document generation scenarios can also assist in quality auditing of script document generation. Attached Figure Description
[0010] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0011] Figure 1 This is an exemplary system architecture diagram to which this application can be applied; Figure 2 This is a flowchart of an embodiment of a document generation quality optimization method according to this application; Figure 3 This is a flowchart of a specific embodiment of reinforcement learning training of a document generation quality assessment model in the document generation quality optimization method described in this application; Figure 4 yes Figure 3 A flowchart of a specific embodiment of step 303 shown; Figure 5 yes Figure 3 A flowchart of a specific embodiment of step 307 is shown; Figure 6 This is a flowchart of a specific embodiment of the document generation quality optimization method described in this application, which evaluates the generated document; Figure 7 yes Figure 2 A flowchart of a specific embodiment of optimizing the generated document in step 205 is shown; Figure 8 This is a schematic diagram of an embodiment of a document generation quality optimization device according to this application; Figure 9 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation
[0012] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.
[0013] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0014] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.
[0015] like Figure 1 As shown, system architecture 100 may include terminal device 101, network 102, and server 103. Terminal device 101 may be a laptop 1011, tablet 1012, or mobile phone 1013. Network 102 is used as a medium to provide a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables.
[0016] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.
[0017] Terminal device 101 can be various electronic devices with a display screen and support web browsing. In addition to laptops 1011, tablets 1012, or mobile phones 1013, terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, and a desktop computer, etc.
[0018] Server 103 can be a server that provides various services, such as a backend server that provides support for the pages displayed on terminal device 101.
[0019] It should be noted that the document generation quality optimization method provided in this application embodiment is generally executed by a server, and correspondingly, a document generation quality optimization device is generally set in the server.
[0020] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.
[0021] Continue to refer to Figure 2 The diagram illustrates a flowchart of an embodiment of a document generation quality optimization method according to this application. The document generation quality optimization method includes the following steps: Step 201: Obtain the generated document output by the preset document generation model.
[0022] In this embodiment, the preset document generation model includes, for example, a document generation model for generating financial business contract documents and a document generation model for generating customer service script guidance documents. The generated documents include documents generated based on a template and documents generated based on reference materials. The generated documents include financial business contract documents and customer service script guidance documents. Here, the generated documents may differ depending on the actual business needs.
[0023] Specifically, obtaining the generated document output by the preset document generation model includes: receiving the generated document output by the output layer of the preset document generation model.
[0024] Step 202: Input the generated document into the pre-trained document generation quality assessment model.
[0025] In this embodiment, the pre-trained document generation quality assessment model refers to a document generation quality assessment model trained through semi-online reinforcement learning. Specifically, in the reinforcement learning of the document generation quality assessment model, an offline reinforcement learning algorithm is introduced on top of the online reinforcement learning algorithm. For example, during online reinforcement learning, the GRPO (Group Relative Policy Optimization) algorithm is used, while during offline reinforcement learning, the DPO (Direct Preference Optimization) algorithm is used. During online reinforcement learning, the GRPO algorithm optimizes the actual business processing model through relative rewards within groups until the actual business processing model reaches certain training conditions, completing the model reinforcement learning training and obtaining the pre-trained model. First, compared to the traditional PPO online reinforcement learning algorithm, GRPO significantly reduces memory usage and computational cost during training. Furthermore, GRPO estimates the advantage function through within-group comparisons, reducing the variance of policy updates and ensuring a more stable learning process, thus improving training stability. GRPO also introduces KL divergence constraints to prevent overly drastic policy updates, thereby maintaining the stability of the policy distribution. Finally, by introducing the DPO offline reinforcement learning algorithm, which directly selects appropriate optimization options based on the optimization expectation, the computational cost is lower and more stable. This ensures that a document generation quality assessment model trained through reinforcement learning is obtained more quickly while reducing the resource consumption of online reinforcement learning.
[0026] Step 203: Obtain the evaluation results output by the document generation quality assessment model.
[0027] In this embodiment, the evaluation result includes an evaluation score or an evaluation level.
[0028] Step 204: Determine whether the evaluation result meets the preset requirements.
[0029] In this embodiment, the preset requirements are, for example, preset evaluation scores or evaluation levels.
[0030] Step 205: If the preset requirements are not met, optimize the generated document according to the preset generation quality optimization strategy and the evaluation results to obtain an optimized document. Replace the generated document with the optimized document and repeat the document generation quality evaluation and optimization operation.
[0031] Specifically, by repeatedly performing document generation quality assessment and optimization operations, it is ensured that the final document can meet the quality assessment and preset requirements.
[0032] Step 206: Continue until the latest evaluation results meet the preset requirements, then output the final document.
[0033] In this embodiment, the generated document output by the preset document generation model is obtained; input into the document generation quality assessment model; assessment results are obtained; it is determined whether the assessment results meet the preset requirements; if not, the generated document is optimized according to the preset generation quality optimization strategy and assessment results to obtain an optimized document. The optimized document replaces the generated document, and the document generation quality assessment and optimization operations are repeated until the latest assessment result meets the preset requirements, at which point the final document is output. By performing generation quality assessment and optimization operations on the generated document, it is ensured that the final document meets the corresponding document quality requirements. Applying the document generation quality optimization method to financial business scenarios can assist in the quality review of financial contract document generation; applying it to customer service script document generation scenarios can also assist in the quality review of script document generation.
[0034] Continue to refer to Figure 3 In some specific implementations, a step of reinforcement learning training for the document generation quality assessment model is included before step 202. Figure 3 This is a flowchart of a specific embodiment of reinforcement learning training of a document generation quality assessment model in the document generation quality optimization method described in this application, including: Step 301: Obtain the training document set, wherein each document in the training document set is pre-labeled with a document partitioning structure and the document classification topic, document classification similar topics, and quality score corresponding to each partitioning structure; In this embodiment, each document in the training document set and the generated document are generated documents under the same business. The difference is that each document in the training document set is pre-labeled with a document division structure and the document category topic, document category similar topics, and quality score corresponding to each division structure; for example, a financial business contract document includes a section for filling in the main information (Party A and Party B), a section for introducing business matters, and a section for legal terms and responsibilities.
[0035] Step 302: Input the training document set into a document generation quality assessment model consisting of N quality assessment agents, wherein the N quality assessment agents include 1 master agent and N-1 slave agents, and N is a positive integer greater than 1; In this embodiment, the document generation quality assessment model consists of N quality assessment agents, including one master agent and N-1 slave agents. The purpose of the master agent's reinforcement learning is to divide the entire generated document into various parts and identify the processing parts corresponding to each slave agent. The purpose of the slave agents' reinforcement learning is to optimize and replace the actual text content of different parts in the generated document, thereby obtaining an optimized generated document.
[0036] Step 303: Based on the document partitioning structure pre-annotated for each document in the training document set, the document classification topic corresponding to each partitioning structure, and the document classification similar topic, the main agent is used to perform structural partitioning and classification processing on the documents in the training document set, so as to obtain a main agent that has completed both structural partitioning and classification processing. Specifically, the document segmentation structure, the document classification topic, and the document classification similar topic for each document in the training document set are pre-annotated and input into the main agent, so that the main agent learns to perform structural segmentation and classification of the generated documents.
[0037] Step 304: Assign the classification results to different agents according to the different document classification topics; Specifically, different classification results are assigned to different agents based on different document classification topics, so that each agent only performs reinforcement learning for the optimization of the target document classification topic, avoiding confusion among the agents and achieving a decoupled optimization effect when actually optimizing the generated documents.
[0038] Step 305: In the agent, the classification results under the corresponding topic are sorted by quality according to the quality score corresponding to each partitioning structure; Specifically, by ranking the classification results under the corresponding topic according to the quality score of each partition structure in the agent, it is ensured that the agent trained by reinforcement learning can use the quality ranking result in the actual document quality optimization and optimize the quality of the corresponding partition in the generated document according to the quality ranking result.
[0039] Step 306: Obtain the mapping relationship between document classification topics and agents, obtain the quality ranking result corresponding to each agent, and the quality score corresponding to each partition structure in each agent; Step 307: Based on the mapping relationship between the document classification topics and the agents, the quality ranking results corresponding to each agent, the quality scores corresponding to all partitioning structures in each agent, and the document classification topics and document classification similar topics corresponding to each partitioning structure, the document generation quality assessment model is trained by reinforcement learning to obtain the pre-trained document generation quality assessment model.
[0040] In this embodiment, the main agent and slave agents in the document generation quality assessment model are trained using a labeled training document set, ensuring that the pre-trained document generation quality assessment model can optimize newly generated documents.
[0041] Continue to refer to Figure 4 , Figure 4 yes Figure 3 A flowchart of a specific embodiment of step 303 shown includes: Step 401: Based on the document segmentation structure pre-annotated for each document in the training document set, perform structural segmentation on all documents to obtain the structural document fragment corresponding to each document; Step 402: Combining the document classification theme corresponding to each partition structure, the structured document fragments are classified according to different document classification themes to obtain a set of structured document fragments corresponding to different document classification themes. Specifically, steps 401 and 402 involve collecting and organizing all structured document fragments corresponding to the same document category topic in the training document set.
[0042] Step 403: Perform association tagging on the set of structured document fragments corresponding to similar topics in document classification; Specifically, the system implements association labeling for the aggregated results of document classification with similar topics. Subsequently, since different agents learn to optimize document content under different topics, this association labeling process facilitates the master agent's learning of the association relationship between different agents. That is, when the document classification topics corresponding to two agents are similar topics, it is easier for the master agent to identify them.
[0043] Step 404: The association tagging results between the sets of structured document fragments corresponding to different document classification topics and the sets of structured document fragments corresponding to similar document classification topics are used as the classification results.
[0044] Continue to refer to Figure 5 , Figure 5 yes Figure 3 A flowchart of a specific embodiment of step 307 shown includes: Step 501: Based on the quality scores corresponding to all partition structures in each agent, a document generation quality scoring component is trained for each agent, wherein the document generation quality scoring component includes a data scoring component based on Athene-RM-8B. Specifically, Athene-RM-8B is a reward model based on the RLHF (Human Feedback Reinforcement Learning) framework, designed for alignment with Large Language Models (LLM). It aims to optimize the quality of model output through user interaction data. It scores the quality of generated data at each generation stage to obtain a generation quality score, which represents the generation quality of that part.
[0045] In this embodiment, the data scoring component based on Athene-RM-8B is trained using the quality scores corresponding to all partition structures in each agent, i.e., the quality scores labeled in the training sample set, so that the data scoring component can perform quality scoring on the newly input partition structure content.
[0046] Step 502: Based on the mapping relationship between document classification topics and slave agents, the document classification topics and document classification similar topics corresponding to each partitioning structure, fit a first-order optimization strategy for preliminary optimization of document generation quality. The first-order optimization strategy includes using the online reinforcement learning algorithm GRPO combined with the document generation quality scoring component, so that the main agent learns to select slave agents corresponding to the same or similar document classification topics to optimize document generation quality. Specifically, the online reinforcement learning algorithm GRPO is a group relative optimization algorithm. Its input is a given prompt word, and its output is a set of target answers. Then, it performs intra-group optimization based on this set of target answers. Afterward, it iteratively optimizes by obtaining the group target answers based on different given prompt words. In this embodiment, it can be understood as inputting a document classification topic and outputting a set of agents corresponding to the same or similar document classification topics. Then, it uses the selected agents within the group to perform actual document generation quality optimization. Here, combined with the document generation quality scoring component, each time group relative optimization is performed using different document classification topics, the replacement document portion of the selected agents is scored, ensuring that the quality score of the selected replacement document is higher than the quality score of the original replacement document in each document generation quality optimization.
[0047] Step 503: Based on the quality ranking result corresponding to each agent and the quality score corresponding to all partition structures in each agent, fit a second-order optimization strategy for optimizing document generation quality on the basis of the preliminary optimization strategy. The second-order optimization strategy includes the optimization method of using the offline reinforcement learning algorithm DPO, so that the agent learns to select the partition documents with the highest scores in the quality ranking results for document generation quality optimization. Specifically, the optimization method using the offline reinforcement learning algorithm DPO refers to replacing documents with higher quality scores directly from the quality ranking results learned from the agent when optimizing the documents to be replaced, based on the calculated quality scores.
[0048] In this embodiment, the optimization method of the offline reinforcement learning algorithm and the data scoring component are finally integrated into each round of group optimization in the group relative optimization algorithm. That is, the offline optimization algorithm and the data scoring component are called once every time the group relative optimization algorithm performs group relative optimization, which ensures efficient and fast document quality optimization of the generated document.
[0049] Step 504: Deploy the first-order optimization strategy and the second-order optimization strategy together as a preset generation quality optimization strategy into the document generation quality assessment model to obtain the pre-trained document generation quality assessment model.
[0050] Continue to refer to Figure 6 In some specific embodiments, an evaluation step of the generated document is included before step 203. Figure 6 This is a flowchart of a specific embodiment of the document generation quality optimization method described in this application, which evaluates the generated document, including: Step 601: Use the main agent in the document generation quality assessment model to perform structural division and classification of the generated document to obtain the structural division results of the generated document and the document classification topic corresponding to each division part; Specifically, since the main intelligent agent learns to perform structural division and classification of documents during the learning and training phase, it can directly perform structural division and classification on the generated documents during actual processing to obtain the structural division results of the generated documents and the document classification topics corresponding to each division part.
[0051] Step 602: According to the document classification topic corresponding to each segment, all the segments in the structured segmentation result are passed to the corresponding slave agents one by one; Specifically, since each agent processes document segmentation parts corresponding to different document classification topics during the learning and training phase, in actual processing, all the segmentation parts in the structured segmentation result can be directly passed to the corresponding agent according to the document classification topic corresponding to each segmentation part.
[0052] Step 603: Use each document generation quality scoring component learned and trained from the agent to score the generation quality of all the partitions, and obtain the generation quality score corresponding to each partition. Specifically, during the learning and training phase, the agents ultimately rank the quality of the partitions under the same document category topic based on the quality scores. Therefore, the document generation quality scoring component, namely the Athene-RM-8B-based data scoring component, is introduced into each agent to ensure that in actual processing, different agents can generate quality scores for different document category topics.
[0053] Step 604: Compile the generated quality scores corresponding to all the divisions as the evaluation results.
[0054] Continue to refer to Figure 7 , Figure 7 yes Figure 2 A flowchart of a specific embodiment of optimizing the generated document in step 205 is shown, including: Step 701: Based on the evaluation results, identify the document category topics and generate quality scores corresponding to each of the divisions; Step 702: Based on the document classification topics corresponding to all the partitioned parts, the first-order optimization strategy in the preset generation quality optimization strategy is adopted to determine the agents corresponding to the same or similar document classification topics as target agents online. Specifically, based on the document category topics corresponding to each of the divisions, and using the document category topic as a prompt, a group of agents corresponding to that document category topic is selected. The group of agents includes agents corresponding to the same or similar document category topics.
[0055] Step 703: Based on the quality ranking results of the target from the agent, the second-order optimization strategy in the preset generation quality optimization strategy is adopted to offline filter the partition document with the highest generation quality score from the target from the agent to replace the target partition part. Specifically, the document with the highest quality score from the target agent is offline filtered to replace the target partition. Here, the document with the highest quality score is selected directly from the quality ranking results based on the correspondence between the quality score and the quality score of each target agent. If there are multiple target agents, the quality scores of the selected replacement documents need to be compared, and only one replacement document is ultimately selected.
[0056] Step 704: Obtain the document after replacement as the optimized document.
[0057] In this embodiment, the step of outputting the final document until the latest evaluation result meets the preset requirements includes: if the generation quality scores corresponding to all the partitions exceed the corresponding generation quality score thresholds, then the document that was last input into the pre-trained document generation quality evaluation model is obtained; and the document is output as the final document.
[0058] In this embodiment, the document generation quality optimization method improves the document generation quality assessment model. The main improvement is that, in terms of generation quality assessment and optimization, it no longer uses a single reinforcement learning algorithm for model training. Instead, based on the GRPO algorithm, it introduces the DPO algorithm optimization method and the Athene-RM-8B-based data scoring component during each in-group optimization. This ultimately ensures high-quality and rapid optimization of the generated documents, making the final output document meet the corresponding requirements, and the optimization process is relatively stable and efficient.
[0059] In this embodiment, the document generation quality optimization method can be widely applied to scenarios where the generation quality of a target document is evaluated and optimized. It involves obtaining the generated document output by a preset document generation model; inputting it into a document generation quality evaluation model; obtaining the evaluation result; determining whether the evaluation result meets preset requirements; if not, optimizing the generated document according to a preset generation quality optimization strategy and the evaluation result to obtain an optimized document; and repeating the document generation quality evaluation and optimization operation with the optimized document instead of the generated document until the latest evaluation result meets the preset requirements, at which point the final document is output. By performing generation quality evaluation and optimization operations on the generated document, the final document is ensured to meet the corresponding document quality requirements. Applying this document generation quality optimization method to financial business scenarios can assist in quality review during the generation of financial contract documents; applying it to customer service script document generation scenarios can also assist in quality review of script document generation.
[0060] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.
[0061] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.
[0062] In this embodiment, the document generation quality optimization method can be widely applied to scenarios where the generation quality of a target document is evaluated and optimized. It involves obtaining the generated document output by a preset document generation model; inputting it into a document generation quality evaluation model; obtaining the evaluation result; determining whether the evaluation result meets preset requirements; if not, optimizing the generated document according to a preset generation quality optimization strategy and the evaluation result to obtain an optimized document; and repeating the document generation quality evaluation and optimization operation with the optimized document instead of the generated document until the latest evaluation result meets the preset requirements, at which point the final document is output. By performing generation quality evaluation and optimization operations on the generated document, the final document is ensured to meet the corresponding document quality requirements. Applying this document generation quality optimization method to financial business scenarios can assist in quality review during the generation of financial contract documents; applying it to customer service script document generation scenarios can also assist in quality review of script document generation.
[0063] Further reference Figure 8 As a response to the above Figure 2 The implementation of the method shown in this application provides an embodiment of a document generation quality optimization device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.
[0064] like Figure 8 As shown, the document generation quality optimization device 800 described in this embodiment includes: a document acquisition module 801, an evaluation model input module 802, an evaluation result acquisition module 803, a requirement condition judgment module 804, a document duplication optimization module 805, and a final document output module 806. Wherein: The document generation acquisition module 801 is used to acquire the generated document output by the preset document generation model; The evaluation model input module 802 is used to input the generated document into the pre-trained document generation quality evaluation model; The evaluation result acquisition module 803 is used to acquire the evaluation results output by the document generation quality evaluation model; The requirement condition judgment module 804 is used to determine whether the evaluation result meets the preset requirement conditions; The document duplication optimization module 805 is used to optimize the generated document according to the preset generation quality optimization strategy and the evaluation result if the preset requirements are not met, so as to obtain an optimized document and use the optimized document to replace the generated document to repeatedly perform document generation quality evaluation and optimization operations. The final document output module 806 is used to output the final document until the latest evaluation results meet the preset requirements.
[0065] This application obtains the generated document output by a preset document generation model; inputs it into a document generation quality assessment model; obtains the assessment result; determines whether the assessment result meets preset requirements; if not, optimizes the generated document according to a preset generation quality optimization strategy and the assessment result to obtain an optimized document; and repeats the document generation quality assessment and optimization operation with the optimized document instead of the generated document; until the latest assessment result meets the preset requirements, the final document is output. By performing generation quality assessment and optimization operations on the generated document, the final document is guaranteed to meet the corresponding document quality requirements. Applying the aforementioned document generation quality optimization method to financial business scenarios can assist in the quality review of financial contract document generation; applying it to customer service script document generation scenarios can also assist in the quality review of script document generation.
[0066] In this embodiment, the document generation quality optimization device 800 further includes a training document set acquisition module, a training input module, a master agent learning module, a slave agent processing and allocation module, a quality ranking module, a slave agent reinforcement correlation acquisition module, and a slave agent reinforcement learning training module. Wherein: The training document set acquisition module is used to acquire the training document set, wherein each document in the training document set is pre-labeled with a document partitioning structure and the document category topic, document category similar topics, and quality score corresponding to each partitioning structure; The training input module is used to input the training document set into a document generation quality assessment model consisting of N quality assessment agents, wherein the N quality assessment agents include 1 master agent and N-1 slave agents, and N is a positive integer greater than 1; The main agent learning module is used to perform structural division and classification processing on the documents in the training document set based on the pre-annotated document division structure, the document classification topic corresponding to each division structure and the document classification similar topic, and to obtain a main agent that has completed both structural division and classification processing. The agent processing and allocation module is used to allocate the classification processing results to different agents according to the different document classification topics; The quality ranking module is used to rank the classification results under the corresponding topic in the agent according to the quality score corresponding to each partitioning structure. The module for enhancing the relevant information from the agent is used to obtain the mapping relationship between document classification topics and agents, obtain the quality ranking result corresponding to each agent, and obtain the quality score corresponding to each partition structure in each agent. The agent reinforcement learning training module is used to perform reinforcement learning training on the document generation quality assessment model based on the mapping relationship between the document classification topic and the agent, the quality ranking result corresponding to each agent, the quality score corresponding to each partition structure in each agent, and the document classification topic and document classification similar topic corresponding to each partition structure, so as to obtain the pre-trained document generation quality assessment model.
[0067] In this embodiment, the main intelligent agent learning module includes a structured document fragment segmentation unit, a structured document fragment classification processing unit, an association tagging processing unit, and a classification processing result setting unit. Wherein: The structured document segmentation unit is used to structurally segment all documents according to the document segmentation structure pre-annotated for each document in the training document set, and obtain the structured document segment corresponding to each document. The structured document fragment classification processing unit is used to classify the structured document fragments according to different document classification themes, based on the document classification theme corresponding to each partition structure, to obtain a set of structured document fragments corresponding to different document classification themes. The association tagging processing unit is used to perform association tagging processing on a set of structured document fragments corresponding to similar topics in document classification; The classification processing result setting unit is used to take the association tagging processing results between the set of structured document fragments corresponding to different document classification topics and the set of structured document fragments corresponding to similar document classification topics as the classification processing results.
[0068] In this embodiment, the agent reinforcement learning training module includes a document generation quality scoring component training unit, a first-order optimization strategy fitting unit, a second-order optimization strategy fitting unit, and a generation quality optimization strategy deployment unit. Wherein: The document generation quality scoring component training unit is used to learn and train a document generation quality scoring component for each agent based on the quality scores corresponding to all partition structures in each agent. The document generation quality scoring component includes a data scoring component based on Athene-RM-8B. The first-order optimization strategy fitting unit is used to fit a first-order optimization strategy for preliminary optimization of document generation quality based on the mapping relationship between document classification topics and slave agents, the document classification topics corresponding to each partitioning structure and document classification similar topics. The first-order optimization strategy includes using the online reinforcement learning algorithm GRPO combined with the document generation quality scoring component, so that the master agent learns to select slave agents corresponding to the same or similar document classification topics for document generation quality optimization. The second-order optimization strategy fitting unit is used to fit a second-order optimization strategy for optimizing document generation quality based on the quality ranking result corresponding to each agent and the quality score corresponding to all partition structures in each agent, on the basis of the preliminary optimization strategy. The second-order optimization strategy includes the optimization method of using the offline reinforcement learning algorithm DPO, so that the agent learns to select the partition documents with the highest scores in the quality ranking results for document generation quality optimization. A quality optimization strategy deployment unit is used to deploy the first-order optimization strategy and the second-order optimization strategy as a preset quality optimization strategy into the document generation quality evaluation model to obtain the pre-trained document generation quality evaluation model.
[0069] In this embodiment, the document generation quality optimization device 800 further includes a main agent processing module, a classification result transmission module, a slave agent scoring module, and an evaluation result processing module. Wherein: The main agent processing module is used to perform structural division and classification processing on the generated document using the main agent in the document generation quality assessment model, so as to obtain the structural division results of the generated document and the document classification topic corresponding to each division part; The classification result transmission module is used to transmit all the division parts in the structured division result to the corresponding slave agents according to the document classification topic corresponding to each division part; The agent scoring module is used to score the generation quality of all the partitions by using the document generation quality scoring components learned and trained from each agent, and to obtain the generation quality score corresponding to each partition. The evaluation result processing module is used to process the generated quality scores corresponding to each of the divisions as the evaluation results.
[0070] In this embodiment, the document duplication optimization module 805 includes an evaluation result recognition unit, a target agent determination unit, a document partitioning and replacement unit, and an optimized document acquisition unit. Wherein: The evaluation result identification unit is used to identify the document category topics and generate quality scores corresponding to each of the divisions based on the evaluation results. The target agent determination unit is used to determine, based on the document classification topics corresponding to all the partitioned parts, the first-order optimization strategy in the preset generation quality optimization strategy, and online the agents corresponding to the same or similar document classification topics as target agents. The document replacement unit is used to replace the target partition part by offline filtering of the document with the highest generation quality score from the target from the intelligent agent based on the quality ranking result of the target from the intelligent agent and using the second-order optimization strategy in the preset generation quality optimization strategy. An optimized document acquisition unit is used to obtain the document after replacement as the optimized document.
[0071] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).
[0072] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.
[0073] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 9 , Figure 9 This is a basic structural block diagram of the computer device in this embodiment.
[0074] The computer device 9 includes a memory 9a, a processor 9b, and a network interface 9c that are interconnected via a system bus. It should be noted that... Figure 9 Only a computer device 9 with component memory 9a, processor 9b, and network interface 9c is shown. However, it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Those skilled in the art will understand that the computer device described herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.
[0075] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.
[0076] The memory 9a includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 9a may be an internal storage unit of the computer device 9, such as the hard disk or memory of the computer device 9. In other embodiments, the memory 9a may also be an external storage device of the computer device 9, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 9. Of course, the memory 9a may also include both the internal storage unit and its external storage device of the computer device 9. In this embodiment, the memory 9a is typically used to store the operating system and various application software installed on the computer device 9, such as computer-readable instructions for a document generation quality optimization method. In addition, the memory 9a can also be used to temporarily store various types of data that have been output or will be output.
[0077] In some embodiments, the processor 9b may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 9b is typically used to control the overall operation of the computer device 9. In this embodiment, the processor 9b is used to execute computer-readable instructions stored in the memory 9a or to process data, for example, to execute computer-readable instructions for the document generation quality optimization method described above.
[0078] The network interface 9c may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 9 and other electronic devices.
[0079] The computer device proposed in this embodiment belongs to the field of artificial intelligence technology and is applied to scenarios involving the evaluation and optimization of the generation quality of target documents. This application obtains the generated document output by a preset document generation model; inputs it into a document generation quality evaluation model; obtains the evaluation result; determines whether the evaluation result meets preset requirements; if not, optimizes the generated document according to a preset generation quality optimization strategy and the evaluation result to obtain an optimized document; and repeats the document generation quality evaluation and optimization operation with the optimized document instead of the generated document; until the latest evaluation result meets the preset requirements, the final document is output. By performing generation quality evaluation and optimization operations on the generated document, it ensures that the final document meets the corresponding document quality requirements. Applying the aforementioned document generation quality optimization method to financial business scenarios can assist in the quality review of financial contract document generation; applying it to customer service script document generation scenarios can also assist in the quality review of script document generation.
[0080] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by a processor to cause the processor to perform the steps of the document generation quality optimization method described above.
[0081] The computer-readable storage medium proposed in this embodiment belongs to the field of artificial intelligence technology and is applied to scenarios involving the evaluation and optimization of the generation quality of target documents. This application obtains the generated document output by a preset document generation model; inputs it into a document generation quality evaluation model; obtains the evaluation result; determines whether the evaluation result meets preset requirements; if not, optimizes the generated document according to a preset generation quality optimization strategy and the evaluation result to obtain an optimized document; and repeats the document generation quality evaluation and optimization operation with the optimized document instead of the generated document; until the latest evaluation result meets the preset requirements, the final document is output. By performing generation quality evaluation and optimization operations on the generated document, it ensures that the final document meets the corresponding document quality requirements. Applying the document generation quality optimization method to financial business scenarios can assist in the quality review of financial contract document generation; applying it to customer service script document generation scenarios can also assist in the quality review of script document generation.
[0082] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0083] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to make the disclosure of this application more thorough and comprehensive. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application. Software tools or components not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.
Claims
1. A method for optimizing document generation quality, characterized in that, Includes the following steps: Obtain the generated document output by the preset document generation model; The generated document is then input into a pre-trained document generation quality assessment model; Obtain the evaluation results output by the document generation quality assessment model; Determine whether the evaluation result meets the preset requirements; If the preset requirements are not met, the generated document is optimized according to the preset generation quality optimization strategy and the evaluation results to obtain an optimized document. The optimized document is then used to replace the generated document to repeat the document generation quality evaluation and optimization operation. The final document will be output once the latest evaluation results meet the preset requirements.
2. The document generation quality optimization method according to claim 1, characterized in that, Before performing the step of inputting the generated document into the pre-trained document generation quality assessment model, the method further includes: Obtain a training document set, wherein each document in the training document set is pre-labeled with a document partitioning structure and the document classification topic, document classification similar topics, and quality score corresponding to each partitioning structure; The training document set is input into a document generation quality assessment model consisting of N quality assessment agents, wherein the N quality assessment agents include 1 master agent and N-1 slave agents, and N is a positive integer greater than 1; Based on the document partitioning structure, document classification topics, and document classification similar topics pre-annotated for each document in the training document set, the main agent is used to perform structural partitioning and classification processing on the documents in the training document set, resulting in a main agent that has completed both structural partitioning and classification processing. The classification results are assigned to different agents based on the different document categories and topics. In the aforementioned intelligent agent, the classification results under the corresponding topic are sorted by quality according to the quality score corresponding to each partitioning structure; Obtain the mapping relationship between document classification topics and agents, obtain the quality ranking result corresponding to each agent, and the quality score corresponding to each partition structure in each agent; Based on the mapping relationship between the document classification topics and the agents, the quality ranking results corresponding to each agent, the quality scores corresponding to all partitioning structures in each agent, and the document classification topics and document classification similar topics corresponding to each partitioning structure, the document generation quality assessment model is trained by reinforcement learning to obtain a pre-trained document generation quality assessment model.
3. The document generation quality optimization method according to claim 2, characterized in that, The step of structurally partitioning and classifying documents in the training document set using the main agent, based on the pre-annotated document partitioning structure of each document in the training document set, the document classification topic corresponding to each partitioning structure, and the document classification similar topics, includes: Based on the document segmentation structure pre-annotated for each document in the training document set, all documents are structurally segmented to obtain the structural document fragments corresponding to each document. Based on the document category theme corresponding to each partition structure, the structured document fragments are classified according to different document category themes to obtain a set of structured document fragments corresponding to different document category themes. Perform association tagging on sets of structured document fragments corresponding to similar topics in document classification; The association labeling results between the sets of structured document fragments corresponding to different document classification topics and the sets of structured document fragments corresponding to similar document classification topics are used as the classification results.
4. The document generation quality optimization method according to claim 2, characterized in that, The step of training the document generation quality assessment model through reinforcement learning based on the mapping relationship between the document classification topics and the agents, the quality ranking results corresponding to each agent, the quality scores corresponding to all partitioning structures in each agent, and the document classification topics and similar topics corresponding to each partitioning structure, to obtain the pre-trained document generation quality assessment model, includes: Based on the quality scores corresponding to all partition structures in each agent, a document generation quality scoring component is trained for each agent. Based on the mapping relationship between document classification topics and slave agents, the document classification topics and similar topics corresponding to each partitioning structure, a first-order optimization strategy for preliminary optimization of document generation quality is fitted. The first-order optimization strategy includes using the document generation quality scoring component in an online reinforcement learning algorithm, so that the master agent learns to select slave agents corresponding to the same or similar document classification topics to optimize document generation quality. Based on the quality ranking result corresponding to each agent and the quality score corresponding to all partition structures in each agent, a second-order optimization strategy for optimizing document generation quality is fitted on the basis of the first-order optimization strategy. The second-order optimization strategy includes an optimization method using an offline reinforcement learning algorithm, so that the agent learns to select the partitioned documents with the highest scores in the quality ranking results for document generation quality optimization. The first-order optimization strategy and the second-order optimization strategy are combined as preset generation quality optimization strategies and deployed into the document generation quality evaluation model to obtain the pre-trained document generation quality evaluation model.
5. The document generation quality optimization method according to claim 1, characterized in that, Before performing the step of obtaining the evaluation results output by the document generation quality assessment model, the method further includes: The generated documents are structurally divided and categorized using the main agent in the document generation quality assessment model, resulting in the structural division results of the generated documents and the document classification topics corresponding to each division. According to the document classification topic corresponding to each segment, all segments in the structural segmentation result are passed to the corresponding slave agents one by one; The document generation quality scoring component, which is learned and trained from the intelligent agent, is used to score the generation quality of all the partitions, and the generation quality score corresponding to each partition is obtained. The quality scores corresponding to each of the divisions are compiled as the evaluation results.
6. The document generation quality optimization method according to claim 5, characterized in that, The step of optimizing the generated document according to the preset generation quality optimization strategy and the evaluation results to obtain an optimized document includes: Based on the evaluation results, the document category topics and quality scores corresponding to each of the divisions were identified; Based on the document classification topics corresponding to each of the division parts, the first-order optimization strategy in the preset generation quality optimization strategy is adopted to determine the agents corresponding to the same or similar document classification topics as target agents online. Based on the quality ranking results of the target from the agent, the second-order optimization strategy in the preset generation quality optimization strategy is adopted to offline filter the partitioned document with the highest generation quality score from the target from the agent to replace the target partition. The document after the replacement is completed is obtained as the optimized document.
7. The document generation quality optimization method according to claim 5, characterized in that, The step of outputting the final document until the latest evaluation result meets the preset requirements includes: If the generation quality scores corresponding to all the partitions exceed the corresponding generation quality score thresholds, then the document that is finally input into the pre-trained document generation quality assessment model is obtained. The document is output as the final document.
8. A document generation quality optimization device, characterized in that, include: The document generation module is used to obtain the generated document output by the preset document generation model; An evaluation model input module is used to input the generated document into a pre-trained document generation quality evaluation model; The evaluation result acquisition module is used to acquire the evaluation results output by the document generation quality evaluation model; The requirement condition judgment module is used to determine whether the evaluation result meets the preset requirement conditions; The document duplication optimization module is used to optimize the generated document according to the preset generation quality optimization strategy and the evaluation result if the preset requirements are not met, so as to obtain an optimized document and use the optimized document to replace the generated document to repeatedly perform document generation quality evaluation and optimization operations. The final document output module is used to output the final document until the latest evaluation results meet the preset requirements.
9. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the document generation quality optimization method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the document generation quality optimization method as described in any one of claims 1 to 7.