Report generation method and device based on reward mechanism, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides a report generation method and device based on a reward mechanism, equipment and a medium, on one hand, a reward model is constructed with correlation, fluency, diversity, length control, safety, order description and term utilization rate as reward and punishment dimensions, and report quality can be improved from multiple dimensions; on the other hand, the direct preference optimization algorithm is adopted, the non-inference model is intensively trained based on the question-answer pair and the reward model, and the supervised fine tuning algorithm is utilized to perform model fine tuning based on the annotation training set, so that the non-inference model can have thinking ability based on the direct preference optimization algorithm in the first stage; the generated thinking chain can improve the logicality and orderliness of the report generation process, the model output can be further optimized through the supervision fine tuning algorithm in the second stage, it is ensured that the generated report meets the professional standard, and the quality of the generated report is improved while the generation efficiency is improved and the operation cost is reduced based on the direct preference optimization algorithm.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a report generation method, apparatus, device, and medium based on a reward mechanism. Background Technology

[0002] Currently, intelligent report generation technology has gradually become an important tool for improving decision-making efficiency and quality across various fields. For example, in higher education, with the increasing informatization of universities, massive amounts of educational, research, and management data are constantly emerging, making traditional manual report generation methods insufficient to meet the increasingly complex decision-making needs. Intelligent report generation technology, by combining cutting-edge technologies such as natural language processing, data analysis, and machine learning, can automatically generate various reports, promptly reflecting the dynamics of university development and providing scientific basis for think tank decision-making. The development of this technology is of great significance in the strategic planning, policy formulation, and academic research of higher education.

[0003] In existing technologies, intelligent report generation methods mainly fall into the following categories: 1. Template-based generation method: This method generates a report by pre-setting a report template and filling the corresponding data into the template. Although this method is simple to operate and highly efficient, it lacks flexibility, making it difficult to cope with complex reporting needs in different scenarios. Furthermore, the generated reports are rather mechanical and lack personalization. 2. Rule-based generation methods: These methods guide the organization of report content and the construction of sentences through pre-defined rules. For example, rules can be used to determine data trends and generate corresponding conclusions and recommendations based on those trends. While rules can ensure the accuracy and consistency of report content to a certain extent, this method has limited flexibility and scalability when faced with complex data or changing reporting needs. 3. Statistical Model-Based Approaches: Statistical models analyze historical data and report content to build models and generate matching report content. These methods can improve the accuracy of report generation to some extent, but they still struggle to fully adapt to highly customized report needs. 4. Deep Learning-Based Approaches: Deep learning technology, especially pre-trained models in the field of natural language processing, provides powerful support for intelligent report generation. Through training on large-scale text data, deep learning models can understand semantics, reasoning logic, and generate fluent and content-rich reports. However, these methods typically require a large amount of training data, and the generated reports may sometimes lack accuracy, especially when dealing with complex data in specialized fields. Testing of publicly available large language models reveals that mainstream model vendors can only provide users with relatively general answers regarding chart interpretation and report generation, lacking the ability to offer effective solutions in specific vertical domains. These products typically only paraphrase data in tables and offer vague and unsubstantial suggestions such as "It is recommended to strengthen the recruitment of faculty." Furthermore, the survey found that these large models often make common-sense errors when describing rankings or positions. For example, when a position changes from 3 to 5, these large models typically describe the data as "the ranking has increased," which is clearly contrary to common sense. Although this phenomenon has been somewhat alleviated by using engineered prompt words, it cannot be completely eliminated. 5. Multimodal Learning Methods: In some complex scenarios, report generation relies not only on text data but also on various data formats such as charts and images. Multimodal learning methods can integrate different types of data to generate comprehensive reports containing rich information. These methods present significant challenges in data processing and model fusion, and require substantial computational resources.

[0004] Given the shortcomings of the above methods, it is necessary to provide a more efficient, low-cost, and higher-quality report generation method. Summary of the Invention

[0005] In view of the above, it is necessary to provide a report generation method, apparatus, device and medium based on a reward mechanism, which aims to solve the problems of low report generation quality, low efficiency and high computational cost.

[0006] A report generation method based on a reward mechanism, the report generation method based on a reward mechanism includes: In response to a report generation instruction triggered based on target organization document information, the organization document information is collected, and the organization document information is parsed using a ternary relation group as a data structure to obtain question-answer pairs; A reward model is constructed based on relevance, fluency, diversity, length control, security, positional description, and terminology usage as reward and penalty dimensions. The direct preference optimization algorithm is used to reinforce the non-reasoning model based on the question-answer pair and the reward model to obtain the initial model. Obtain a labeled training set, and use a supervised fine-tuning algorithm to fine-tune the initial model based on the labeled training set to obtain a report generation model; The target organization's document information is input into the report generation model to obtain the target report.

[0007] According to a preferred embodiment of the present invention, the step of collecting organizational file information and parsing the organizational file information using a ternary relation group as the data structure to obtain question-answer pairs includes: Obtain the address of the organization to be collected; According to the crawler protocol, obtain the organization's file information based on the address of the organization to be collected; The organizational document information is converted into the target format to obtain the target file; Extract the table data and the corresponding interpretation text from the target file; The table data and the interpretation text are parsed using a ternary relation group that includes the question content, the thought process, and the answer content to obtain the question-answer pair.

[0008] According to a preferred embodiment of the present invention, the construction of the reward model based on relevance, fluency, diversity, length control, security, positional description, and terminology usage as reward and penalty dimensions includes: Based on the model output and model input, construct the following reward functions: relevance reward function, fluency reward function, diversity reward function, length control reward function, security reward function, position description optimization function, and term usage rate reward function; The reward model is constructed based on the relevance reward function, the fluency reward function, the diversity reward function, the length control reward function, the security reward function, the position description optimization function, and the term usage rate reward function.

[0009] According to a preferred embodiment of the present invention, constructing the reward model based on the relevance reward function, the fluency reward function, the diversity reward function, the length control reward function, the security reward function, the position description optimization function, and the term usage rate reward function includes: The task requirement data is obtained by parsing the report generation instruction; Configure a weight coefficient matrix based on the task requirement data; The reward model is obtained by calculating the weighted sum of the relevance reward function, the fluency reward function, the diversity reward function, the length control reward function, the security reward function, the position description optimization function, and the term usage rate reward function based on the weight coefficient matrix.

[0010] According to a preferred embodiment of the present invention, the step of employing a direct preference optimization algorithm to perform reinforcement training on the non-reasoning model based on the question-answer pair and the reward model to obtain an initial model includes: The non-inference model is initialized to the baseline state, and the model parameters of the non-inference model are initialized using random parameters to obtain the first model; Construct cue words for training; The direct preference optimization algorithm is used to train the first model by inputting the prompt words and the question-answer pair into it. During the training process, the reward model is used to process the output data of the model obtained after each round of training to obtain the reward function score matrix and the total reward score. The reward function score matrix and the total reward score are then fed back to the model obtained after each round of training to adjust the model parameters of the model obtained after each round of training. After a preset number of training rounds, the current model is determined as the initial model; The reward function score matrix is ​​used to record the values ​​of each reward function corresponding to each question-answer pair; the total reward score includes the weighted sum of each reward function calculated according to the weight coefficient.

[0011] According to a preferred embodiment of the present invention, the step of fine-tuning the initial model based on the labeled training set using a supervised fine-tuning algorithm to obtain the report generation model includes: The data from the labeled training set is input into the initial model for training. During the training process, the model loss of the model obtained after each training round is continuously obtained, and the model parameters of the model obtained after each training round are updated based on the model loss of the model obtained after each training round. When the model loss converges, training stops, and the currently obtained model is determined as the report generation model.

[0012] According to a preferred embodiment of the present invention, after obtaining the target report, the method further includes: Obtain a pre-configured report quality assessment system; The target report is evaluated according to the aforementioned report quality assessment system, and the evaluation results are obtained. The report generation model is optimized based on the evaluation results.

[0013] A report generation device based on a reward mechanism, the report generation device based on a reward mechanism includes: The parsing unit is used to respond to a report generation instruction triggered based on target organization document information, collect organization document information, and parse the organization document information using a ternary relation group as a data structure to obtain question-answer pairs; Building units are used to construct reward models based on relevance, fluency, diversity, length control, security, positional description, and terminology usage as reward and penalty dimensions; The training unit is used to perform reinforcement training on the non-reasoning model based on the question-answer pair and the reward model using the direct preference optimization algorithm to obtain the initial model. The fine-tuning unit is used to acquire a labeled training set and use a supervised fine-tuning algorithm to fine-tune the initial model based on the labeled training set to obtain a report generation model. The input unit is used to input the target organization's document information into the report generation model to obtain the target report.

[0014] A computer device, the computer device comprising: A memory for storing at least one instruction; and a processor for executing the instructions stored in the memory to implement the reward-based report generation method.

[0015] A computer-readable storage medium storing at least one instruction, which is executed by a processor in a computer device to implement the reward-based report generation method.

[0016] As can be seen from the above technical solutions, on the one hand, constructing a reward model based on relevance, fluency, diversity, length control, security, positional description, and terminology usage as reward and punishment dimensions can improve report quality from multiple dimensions. On the other hand, using the direct preference optimization algorithm, the non-reasoning model is reinforced and trained based on question-answer pairs and the reward model to obtain an initial model. Then, the report generation model is obtained by fine-tuning the initial model based on the labeled training set using a supervised fine-tuning algorithm. In the first stage, the non-reasoning model can be endowed with thinking ability based on the direct preference optimization algorithm. The generated thought chain can improve the logic and coherence of the report generation process. In the second stage, the model output can be further optimized through the supervised fine-tuning algorithm to ensure that the generated report meets professional standards. While improving generation efficiency and reducing computational costs based on the direct preference optimization algorithm, the quality of the generated report is also improved. Attached Figure Description

[0017] Figure 1 This is a flowchart of a preferred embodiment of the report generation method based on a reward mechanism of the present invention.

[0018] Figure 2 This is a functional block diagram of a preferred embodiment of the report generation device based on a reward mechanism of the present invention.

[0019] Figure 3 This is a schematic diagram of the structure of a computer device that implements a preferred embodiment of the report generation method based on a reward mechanism according to the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] like Figure 1 The diagram shown is a flowchart of a preferred embodiment of the report generation method based on a reward mechanism according to the present invention. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different needs.

[0022] The reward-based report generation method is applied to one or more computer devices. The computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0023] The computer device can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, interactive network television (IPTV), smart wearable device, etc.

[0024] The computer equipment may also include network equipment and / or user equipment. The network equipment includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0025] The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0026] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0027] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0028] The network in which the computer device is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, and virtual private network (VPN).

[0029] S10, in response to a report generation instruction triggered based on target organization document information, collect organization document information and parse the organization document information using a ternary relation group as a data structure to obtain question-answer pairs.

[0030] In this embodiment, the target organization file information may include relevant file information of the organization corresponding to the report generation instruction.

[0031] For example, in the field of higher education, the target institution's document information may include university development documents, annual reports, research achievements, discipline development, etc.

[0032] In this embodiment, the report generation instruction can be automatically triggered when the target organization's file information is detected to be uploaded to the designated interface.

[0033] In this embodiment, the process of collecting organizational file information and parsing the information using ternary relation groups as the data structure to obtain question-answer pairs includes: Obtain the address of the organization to be collected; According to the crawler protocol (robot.txt), obtain the organization file information based on the address of the organization to be collected; The organizational document information is converted into the target format to obtain the target file; Extract the table data and the corresponding interpretation text from the target file; The table data and the interpretation text are parsed using a ternary relation group that includes the question content, the thought process, and the answer content to obtain the question-answer pair.

[0034] For example, in the field of higher education, the address of the institution to be collected may include the homepage address of the relevant school, etc.

[0035] Specifically, publicly available institutional document information can be collected according to the aforementioned crawler protocol and based on the developed general crawler framework.

[0036] The target format can be a format such as docx, to facilitate subsequent text parsing.

[0037] Through the above embodiments, effective content can be parsed from the collected institutional document information as the basis for subsequent model training.

[0038] S11 constructs a reward model based on relevance, fluency, diversity, length control, security, positional description, and terminology usage as reward and penalty dimensions.

[0039] In this embodiment, the construction of the reward model based on relevance, fluency, diversity, length control, security, positional description, and terminology usage as reward and penalty dimensions includes: Based on the model output and model input, construct the following reward functions: relevance reward function, fluency reward function, diversity reward function, length control reward function, security reward function, position description optimization function, and term usage rate reward function; The reward model is constructed based on the relevance reward function, the fluency reward function, the diversity reward function, the length control reward function, the security reward function, the position description optimization function, and the term usage rate reward function.

[0040] Specifically, the word overlap rate and cosine similarity between the model output and the model input are calculated, the first sub-weight corresponding to the word overlap rate and the second sub-weight corresponding to the cosine similarity are obtained, and the relevance reward function is obtained by weighting the word overlap rate, the cosine similarity, the first sub-weight and the second sub-weight. Calculate the number of errors output by the model, and call the pre-trained model to calculate the perplexity of the model output. Obtain the third sub-weight corresponding to the number of errors and the fourth sub-weight corresponding to the perplexity. Perform a weighted calculation based on the number of errors, the perplexity, the third sub-weight, and the fourth sub-weight to obtain the fluency reward function. The n-gram repetition rate of the model output is calculated. The difference between 1 and the n-gram repetition rate is used to obtain the lexical non-repetition rate. The type-to-total word ratio (TTR) of the number of unique words in the model output is calculated as the unique word ratio. The fifth sub-weight corresponding to the lexical non-repetition rate and the sixth sub-weight corresponding to the unique word ratio are obtained. The diversity reward function is obtained by weighting the lexical non-repetition rate, the unique word ratio, the fifth sub-weight, and the sixth sub-weight. Calculate the difference between the actual length output by the model and the ideal length, calculate the quotient of the difference and the ideal length to obtain a first value, calculate the difference between 1 and the first value to obtain a length score; construct a length control reward function based on the length score; Obtain a pre-maintained list of sensitive words, detect the proportion of sensitive words output by the model based on the list of sensitive words, and obtain a security reward function; Identify the position description in the model output, detect whether the position description is correct to obtain a first detection result, and construct a position description optimization function based on the first detection result; Obtain a pre-maintained term reward list, detect whether the model output includes terms from the term reward list to obtain a second detection result, and construct a term usage rate reward function based on the second detection result.

[0041] The word overlap rate can be calculated using metrics such as BLEU (Bilingual Evaluation Understudy) and ROUGE (Recall-Oriented Understudy for Gisting Evaluation).

[0042] In this process, pre-trained models (such as BERT (Bidirectional Encoder Representations from Transformers) or Sentence-BERT (Sentence Embeddings using Siamese and Triplet Network Designs)) can be used to encode the model output and model input into vectors, thereby calculating the cosine similarity.

[0043] The system can call tools such as LanguageTool and Grammarly API (Grammarly Application Programming Interface) to count the number of errors in the model's output.

[0044] The perplexity of the model output can be calculated using a pre-trained model; a lower value indicates greater fluency.

[0045] The n-gram repetition rate is used to reflect the proportion of repeated n-grams (such as 2-grams and 3-grams) in the model output.

[0046] If the generated text (i.e. the model output) is too short or too long, the model is given a negative reward.

[0047] The ranking description may include, but is not limited to, the following: ranking, position, ranking percentage, etc. Points will be deducted if the generated text contains an incorrect ranking description, and points will be added if the description is correct.

[0048] The terminology reward list can be configured based on a summary and scoring of authoritative public reports. Specifically, a reward is given when the model's output contains terms or phrases from the terminology reward list; a slight penalty is applied if no terms or phrases from the list appear. If a phrase similar to a given sample in the terminology reward list (high similarity) appears and the data description is accurate, double the score can be awarded. If the context of the sentence differs from the given learning sample in the terminology reward list, 2.5 times the score is awarded.

[0049] In this embodiment, constructing the reward model based on the relevance reward function, the fluency reward function, the diversity reward function, the length control reward function, the security reward function, the position description optimization function, and the term usage rate reward function includes: The task requirement data is obtained by parsing the report generation instruction; Configure a weight coefficient matrix based on the task requirement data; The reward model is obtained by calculating the weighted sum of the relevance reward function, the fluency reward function, the diversity reward function, the length control reward function, the security reward function, the position description optimization function, and the term usage rate reward function based on the weight coefficient matrix.

[0050] The weighting coefficient matrix can be adjusted according to the needs and priorities of different report generation tasks, avoiding inaccuracies caused by a one-size-fits-all approach.

[0051] Through the above embodiments, the quality of the generated reports can be effectively controlled based on a multi-dimensional reward and punishment mechanism. Specifically, the reward model can effectively control the accuracy of the terminology used in the generated reports, effectively alleviating the problem that large models on the market cannot accurately interpret rankings or ranking percentages; the reward model can effectively control the professionalism of the terminology used in the generated reports, reducing the problem of large models using unprofessional terminology when generating reports; the reward model can effectively control the practicality of the suggestions in the generated reports, avoiding the problem that large models on the market make overly general suggestions when generating reports.

[0052] S12, the Direct Preference Optimization (DPO) algorithm is used to reinforce the non-reasoning model based on the question-answer pair and the reward model to obtain the initial model.

[0053] In this embodiment, the direct preference optimization algorithm is used to reinforce the non-reasoning model based on the question-answer pair and the reward model to obtain the initial model, which includes: The non-inference model is initialized to the baseline state, and the model parameters of the non-inference model are initialized using random parameters to obtain the first model; Construct cue words for training; The direct preference optimization algorithm is used to train the first model by inputting the prompt words and the question-answer pair into it. During the training process, the reward model is used to process the output data of the model obtained after each round of training to obtain the reward function score matrix and the total reward score. The reward function score matrix and the total reward score are then fed back to the model obtained after each round of training to adjust the model parameters of the model obtained after each round of training. After a preset number of training rounds, the current model is determined as the initial model; The reward function score matrix is ​​used to record the values ​​of each reward function corresponding to each question-answer pair; the total reward score includes the weighted sum of each reward function calculated according to the weight coefficient.

[0054] Among them, the prompt words enable the model to... <think>The thought process is generated within the tags, and in <answer>The final answer will be given in the middle.

[0055] Through experimentation, the model gradually improved its performance after training for the following number of steps (i.e., rounds): 50 steps: The model has learned the basic format and output structure, and can roughly generate a reasonable text framework based on the input.

[0056] 100 steps: The model begins at <think>Describe your thought process within the tags and try to combine the analysis results with actual data.

[0057] 200 steps: The model can accurately identify and interpret the data in the table and generate text with preliminary explanations.

[0058] 450 steps: The model has been fully trained and is able to correctly analyze and generate reports in relatively complex situations, basically meeting the design requirements.

[0059] The generation of Chain-of-Thought (COT) is continuously optimized through multiple rounds of training, reducing resource consumption and maintaining high generation quality.

[0060] As can be seen, reinforcement learning training using the direct preference optimization algorithm can spontaneously generate thought chains through multiple rounds of training, effectively producing high-quality report thought chains without relying on high computing resources. Compared to existing technologies using GRPO (Group Relative Policy Optimization) or complex cluster computing methods, this significantly reduces the demand for GPU (Graphics Processing Unit) computing resources and can be trained on a single 4090 graphics card, achieving results comparable to industry-leading technologies, thus improving generation efficiency and operability. This embodiment can efficiently generate high-standard professional reports with lower resource costs, demonstrating significant computational efficiency advantages.

[0061] In this reward function scoring matrix, each row of elements can be the values ​​of the reward functions corresponding to each question-answer pair. For example, when there are 100 question-answer pairs, the first row of the reward function scoring matrix can be the values ​​of the reward functions corresponding to the first question-answer pair (i.e., the values ​​of the 7 reward functions provided in this embodiment, with each row of the reward function scoring matrix containing 7 elements), the second row of the reward function scoring matrix can be the values ​​of the reward functions corresponding to the second question-answer pair, and so on, ultimately resulting in a reward function scoring matrix with a dimension of 100*7.

[0062] During training, the model is rewarded or penalized based on the reward function score matrix, with the goal of increasing the total reward score.

[0063] As can be seen, this embodiment, based on a reward model and reinforcement learning, can precisely control the accuracy of word choice and the professionalism of terminology in the report. When generating the report, the combination of reinforcement learning and rule control effectively alleviates the ambiguity in the expression of precise data such as position and percentage in existing large language models, and ensures that the use of terminology conforms to the professional standards of the relevant field. This embodiment can generate highly professional reports while ensuring that the recommendations are specific and actionable, avoiding overly generalized suggestions. Furthermore, this embodiment, by combining domain knowledge and rule control, optimizes the generation of recommendations in the report. Compared to the overly generalized recommendations generated by existing large language models, it ensures that the generated recommendations are specific, actionable, and have high practicality and relevance. Through the collaborative work of reinforcement learning and the rule system, this embodiment can automatically adjust the content of the recommendations in the report, ensuring that each recommendation meets the needs of the actual application scenario, thus improving the practical value of the report.

[0064] Through the above embodiments, combined with the direct preference optimization algorithm, non-reasoning models that originally lacked thinking ability can generate thought chains, no longer relying on the traditional GRPO with high computing resource requirements. This reduces costs while ensuring the logic and orderliness of the report generation process, thus guaranteeing the quality of the generated reports.

[0065] S13, obtain the labeled training set, and use the supervised fine-tuning algorithm to fine-tune the initial model based on the labeled training set to obtain the report generation model.

[0066] In this embodiment, the labeled training set may include labels for inappropriate wording, incorrect formatting, unclear logic, and analytical bias.

[0067] In this embodiment, the step of using the Supervised Fine-Tuning (SFT) algorithm to fine-tune the initial model based on the labeled training set to obtain the report generation model includes: The data from the labeled training set is input into the initial model for training. During the training process, the model loss of the model obtained after each training round is continuously obtained, and the model parameters of the model obtained after each training round are updated based on the model loss of the model obtained after each training round. When the model loss converges, training stops, and the currently obtained model is determined as the report generation model.

[0068] Among these methods, the model loss of the model obtained after each round of training can be determined using methods such as cross-entropy loss.

[0069] Through the above embodiments, based on the labeled training set and the supervised fine-tuning algorithm, it is possible to further control the accuracy of word choice, format accuracy, logical rationality, and interpretation of the model output content, thereby improving the model's generation effect in highly specialized vertical fields.

[0070] This embodiment employs a generation scheme combining reinforcement learning and SFT, which offers significant advantages, particularly in vertical domains such as higher education. By directly using DPO for reinforcement learning in the first stage of the generation process, and then using SFT to control format and language in the second stage, the generated reports are more professional and accurate, avoiding the generalization problem that existing large language models encounter when handling complex domain-specific terminology and data. Furthermore, compared to existing rule-based or simple supervised learning models, this embodiment can more precisely control the professional terminology and data expression in the generated content, ensuring that the report results have higher domain-specificity.

[0071] S14, input the target organization document information into the report generation model to obtain the target report.

[0072] In this embodiment, after the target organization's document information is input into the report generation model, the report generation model can generate a structured report based on the input data and the knowledge learned during previous training.

[0073] For example, the content of the target report may include, but is not limited to: 1) Data Analysis: Extract key trends based on the input tables, charts, and other data.

[0074] 2) Conclusion generation: Based on the results of data analysis, generate specific conclusions or recommendations.

[0075] 3) Terminology: The model can accurately use relevant domain terminology to ensure that the report content is professional and credible.

[0076] In this embodiment, after obtaining the target report, the method further includes: Obtain a pre-configured report quality assessment system; The target report is evaluated according to the aforementioned report quality assessment system, and the evaluation results are obtained. The report generation model is optimized based on the evaluation results.

[0077] The report quality assessment system can be configured according to actual task requirements. For example, the report quality assessment system may include the following assessment indicators: 1) Accuracy: Does the report accurately reflect the information in the data, and are the conclusions based on reasonable analysis?

[0078] 2) Fluency: Is the language of the report fluent and the logic clear?

[0079] 3) Security: Report whether there are sensitive words or inappropriate content.

[0080] The above evaluation allows for further optimization and improvement of the model, ensuring the continued delivery of high-quality intelligent reports.

[0081] Unlike traditional report generation models that require large-scale training, this embodiment can be trained on a single 4090 24G VRAM and achieve industry-leading report generation results. The solution is also universal and can be applied to various vertical industries. It can obtain a highly professional vertical domain model with fewer resources, and has good scalability and transferability. It can provide stable and efficient report generation services under different hardware configurations.

[0082] The architecture of the system upon which this embodiment relies may include a data storage layer, a processing layer, and a result display layer. These layers collaborate through interfaces to ensure the stability and scalability of the overall system.

[0083] As can be seen from the above technical solutions, on the one hand, constructing a reward model based on relevance, fluency, diversity, length control, security, positional description, and terminology usage as reward and punishment dimensions can improve report quality from multiple dimensions. On the other hand, using the direct preference optimization algorithm, the non-reasoning model is reinforced and trained based on question-answer pairs and the reward model to obtain an initial model. Then, the report generation model is obtained by fine-tuning the initial model based on the labeled training set using a supervised fine-tuning algorithm. In the first stage, the non-reasoning model can be endowed with thinking ability based on the direct preference optimization algorithm. The generated thought chain can improve the logic and coherence of the report generation process. In the second stage, the model output can be further optimized through the supervised fine-tuning algorithm to ensure that the generated report meets professional standards. While improving generation efficiency and reducing computational costs based on the direct preference optimization algorithm, the quality of the generated report is also improved.

[0084] like Figure 2 The diagram shown is a functional block diagram of a preferred embodiment of the report generation device based on a reward mechanism of the present invention. The report generation device 11 based on the reward mechanism includes a parsing unit 110, a construction unit 111, a training unit 112, a fine-tuning unit 113, and an input unit 114. The module / unit referred to in this invention is a series of computer program segments that can be executed by a processor and perform a fixed function, and which are stored in memory. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0085] The parsing unit 110 is used to collect the organization file information in response to a report generation instruction triggered based on the target organization file information, and parse the organization file information using a ternary relation group as a data structure to obtain question-answer pairs; The construction unit 111 is used to construct a reward model based on relevance, fluency, diversity, length control, security, position description, and term usage rate as reward and punishment dimensions. The training unit 112 is used to perform reinforcement training on the non-reasoning model based on the question-answer pair and the reward model using the direct preference optimization algorithm to obtain an initial model. The fine-tuning unit 113 is used to acquire a labeled training set and use a supervised fine-tuning algorithm to fine-tune the initial model based on the labeled training set to obtain a report generation model. The input unit 114 is used to input the target organization document information into the report generation model to obtain the target report.

[0086] As can be seen from the above technical solutions, on the one hand, constructing a reward model based on relevance, fluency, diversity, length control, security, positional description, and terminology usage as reward and punishment dimensions can improve report quality from multiple dimensions. On the other hand, using the direct preference optimization algorithm, the non-reasoning model is reinforced and trained based on question-answer pairs and the reward model to obtain an initial model. Then, the report generation model is obtained by fine-tuning the initial model based on the labeled training set using a supervised fine-tuning algorithm. In the first stage, the non-reasoning model can be endowed with thinking ability based on the direct preference optimization algorithm. The generated thought chain can improve the logic and coherence of the report generation process. In the second stage, the model output can be further optimized through the supervised fine-tuning algorithm to ensure that the generated report meets professional standards. While improving generation efficiency and reducing computational costs based on the direct preference optimization algorithm, the quality of the generated report is also improved.

[0087] like Figure 3 The diagram shown is a structural schematic of a computer device that implements a preferred embodiment of the report generation method based on a reward mechanism according to the present invention.

[0088] The computer device 1 may include a memory 12, a processor 13, and a bus (the arrow in the figure represents the bus), and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a report generation program based on a reward mechanism.

[0089] Those skilled in the art will understand that the schematic diagram is merely an example of computer device 1 and does not constitute a limitation on computer device 1. Computer device 1 can be either a bus topology or a star topology. Computer device 1 may also include more or fewer other hardware or software than shown in the diagram, or different component arrangements. For example, computer device 1 may also include input / output devices, network access devices, etc.

[0090] It should be noted that the computer device 1 described is merely an example. Other existing or future electronic products that are adaptable to this invention should also be included within the scope of protection of this invention and are incorporated herein by reference.

[0091] The memory 12 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the computer device 1, such as a portable hard drive of the computer device 1. In other embodiments, the memory 12 can be an external storage device of the computer device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the computer device 1. Furthermore, the memory 12 can include both internal and external storage units of the computer device 1. The memory 12 can be used not only to store application software and various types of data installed on the computer device 1, such as the code of a reward-based report generation program, but also to temporarily store data that has been output or will be output.

[0092] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the computer device 1, connecting various components of the computer device 1 via various interfaces and lines. It performs various functions of the computer device 1 and processes data by running or executing programs or modules stored in the memory 12 (e.g., executing a report generation program based on a reward mechanism) and accessing data stored in the memory 12.

[0093] The processor 13 executes the operating system of the computer device 1 and various installed applications. The processor 13 executes the applications to implement the steps in the various reward-based report generation method embodiments described above, for example... Figure 1 The steps are shown.

[0094] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing a specific function, which describe the execution process of the computer program in the computer device 1. For example, the computer program may be divided into a parsing unit 110, a construction unit 111, a training unit 112, a fine-tuning unit 113, and an input unit 114.

[0095] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute portions of the reward-based report generation method described in the various embodiments of the present invention.

[0096] If the modules / units integrated in the computer device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware devices. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above.

[0097] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, etc.

[0098] Furthermore, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store the operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of blockchain nodes, etc.

[0099] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0100] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, in... Figure 3 The bus is represented by only one straight line, but this does not mean that there is only one bus or one type of bus. The bus is configured to enable communication between the memory 12 and at least one processor 13, etc.

[0101] Although not shown, the computer device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 13 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The computer device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0102] Furthermore, the computer device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the computer device 1 and other computer devices.

[0103] Optionally, the computer device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the computer device 1 and to display a visual user interface.

[0104] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0105] It will be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the computer device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0106] Combination Figure 1 The memory 12 in the computer device 1 stores multiple instructions to implement a report generation method based on a reward mechanism, and the processor 13 can execute the multiple instructions to achieve: In response to a report generation instruction triggered based on target organization document information, the organization document information is collected, and the organization document information is parsed using a ternary relation group as a data structure to obtain question-answer pairs; A reward model is constructed based on relevance, fluency, diversity, length control, security, positional description, and terminology usage as reward and penalty dimensions. The direct preference optimization algorithm is used to reinforce the non-reasoning model based on the question-answer pair and the reward model to obtain the initial model. Obtain a labeled training set, and use a supervised fine-tuning algorithm to fine-tune the initial model based on the labeled training set to obtain a report generation model; The target organization's document information is input into the report generation model to obtain the target report.

[0107] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0108] It should be noted that all data involved in this case was legally obtained. Software tools or components not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

[0109] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0110] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0111] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0112] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0113] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0114] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0115] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in this invention can also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.< / think> < / answer> < / think>

Claims

1. A report generation method based on a reward mechanism, characterized in that, The report generation method based on the reward mechanism includes: In response to a report generation instruction triggered based on target organization document information, the organization document information is collected, and the organization document information is parsed using a ternary relation group as a data structure to obtain question-answer pairs; A reward model is constructed based on relevance, fluency, diversity, length control, security, positional description, and terminology usage as reward and penalty dimensions. The direct preference optimization algorithm is used to reinforce the non-reasoning model based on the question-answer pair and the reward model to obtain the initial model. Obtain a labeled training set, and use a supervised fine-tuning algorithm to fine-tune the initial model based on the labeled training set to obtain a report generation model; The target organization's document information is input into the report generation model to obtain the target report.

2. The report generation method based on a reward mechanism as described in claim 1, characterized in that, The collection of organizational file information, and the parsing of the organizational file information using a ternary relation group as the data structure to obtain question-answer pairs, include: Obtain the address of the organization to be collected; According to the crawler protocol, obtain the organization's file information based on the address of the organization to be collected; The organizational document information is converted into the target format to obtain the target file; Extract the table data and the corresponding interpretation text from the target file; The table data and the interpretation text are parsed using a ternary relation group that includes the question content, the thought process, and the answer content to obtain the question-answer pair.

3. The report generation method based on a reward mechanism as described in claim 1, characterized in that, The reward model, constructed using relevance, fluency, diversity, length control, security, positional description, and terminology usage as reward and penalty dimensions, includes: Based on the model output and model input, construct the following reward functions: relevance reward function, fluency reward function, diversity reward function, length control reward function, security reward function, position description optimization function, and term usage rate reward function; The reward model is constructed based on the relevance reward function, the fluency reward function, the diversity reward function, the length control reward function, the security reward function, the position description optimization function, and the term usage rate reward function.

4. The report generation method based on a reward mechanism as described in claim 3, characterized in that, The step of constructing the reward model based on the relevance reward function, the fluency reward function, the diversity reward function, the length control reward function, the security reward function, the position description optimization function, and the term usage rate reward function includes: The task requirement data is obtained by parsing the report generation instruction; Configure a weight coefficient matrix based on the task requirement data; The reward model is obtained by calculating the weighted sum of the relevance reward function, the fluency reward function, the diversity reward function, the length control reward function, the security reward function, the position description optimization function, and the term usage rate reward function based on the weight coefficient matrix.

5. The report generation method based on a reward mechanism as described in claim 4, characterized in that, The direct preference optimization algorithm is used to reinforce the non-reasoning model based on the question-answer pair and the reward model, resulting in an initial model including: The non-inference model is initialized to the baseline state, and the model parameters of the non-inference model are initialized using random parameters to obtain the first model; Construct cue words for training; The direct preference optimization algorithm is used to train the first model by inputting the prompt words and the question-answer pair into it. During the training process, the reward model is used to process the output data of the model obtained after each round of training to obtain the reward function score matrix and the total reward score. The reward function score matrix and the total reward score are then fed back to the model obtained after each round of training to adjust the model parameters of the model obtained after each round of training. After a preset number of training rounds, the current model is determined as the initial model; The reward function score matrix is ​​used to record the values ​​of each reward function corresponding to each question-answer pair; the total reward score includes the weighted sum of each reward function calculated according to the weight coefficient.

6. The report generation method based on a reward mechanism as described in claim 1, characterized in that, The step of using a supervised fine-tuning algorithm to fine-tune the initial model based on the labeled training set to obtain the report generation model includes: The data from the labeled training set is input into the initial model for training. During the training process, the model loss of the model obtained after each training round is continuously obtained, and the model parameters of the model obtained after each training round are updated based on the model loss of the model obtained after each training round. When the model loss converges, training stops, and the currently obtained model is determined as the report generation model.

7. The report generation method based on a reward mechanism as described in claim 1, characterized in that, After obtaining the target report, the method further includes: Obtain a pre-configured report quality assessment system; The target report is evaluated according to the aforementioned report quality assessment system, and the evaluation results are obtained. The report generation model is optimized based on the evaluation results.

8. A report generation device based on a reward mechanism, characterized in that, The report generation device based on the reward mechanism includes: The parsing unit is used to respond to a report generation instruction triggered based on target organization document information, collect organization document information, and parse the organization document information using a ternary relation group as a data structure to obtain question-answer pairs; Building units are used to construct reward models based on relevance, fluency, diversity, length control, security, positional description, and terminology usage as reward and penalty dimensions; The training unit is used to perform reinforcement training on the non-reasoning model based on the question-answer pair and the reward model using the direct preference optimization algorithm to obtain the initial model. The fine-tuning unit is used to acquire a labeled training set and use a supervised fine-tuning algorithm to fine-tune the initial model based on the labeled training set to obtain a report generation model. The input unit is used to input the target organization's document information into the report generation model to obtain the target report.

9. A computer device, characterized in that, The computer device includes: A memory for storing at least one instruction; and a processor for executing the instructions stored in the memory to implement the reward-based report generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, which is executed by a processor in a computer device to implement the reward-based report generation method as described in any one of claims 1 to 7.

Images