Method and device for evaluating character innovation ability of GPT class model
Through data processing based on public database and GPT model training, combined with the evaluation method of accuracy and repetition rate, the problem of evaluating the innovation ability of GPT-like models in the existing technology is solved, and the accuracy of text generation of GPT model is achieved. Objective evaluation improves the efficiency and reliability of evaluation and is suitable for a variety of text generation tasks.
Patent Information
- Application Number
- CN202411312515.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-20
- Publication Date
- 2024-12-06
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
It is difficult for the existing technology to effectively evaluate the innovative ability of GPT-like models to generate texts. Manual evaluation is highly subjective and inefficient. Traditional automatic evaluation indicators cannot accurately measure the innovation and creativity of texts, especially in long texts and complex contexts.
By collecting and processing text data in public databases, establishing a GPT model, and evaluating the generated text innovation ability based on the accuracy and repetition rate, it provides a GPT model text innovation ability evaluation method and device, using data acquisition, training and Evaluate the system, generate standardized data sets, train the model, calculate the accuracy and repetition rate to evaluate the innovative capabilities of the model.
Implement objective and accurate evaluation of text generated by GPT model, improve the efficiency and reliability of the evaluation, provide comprehensive analysis results and improvement suggestions, and are suitable for text generation tasks in different types and fields, improving the practicality of the method and applicability.
Smart Images

Figure CN119089902A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of natural language processing, and specifically relates to a method and device for evaluating the text innovation ability of a GPT-type model. Background Art
[0002] In the field of modern natural language processing, generative pre-training models (GPT) have demonstrated their powerful capabilities in text generation and language understanding. This type of model can generate high-quality natural language text and is widely used in many fields such as automatic writing, dialogue systems, and content creation. However, with the continuous advancement of text generation technology, evaluating the innovative ability of generated text has become an important topic. At present, the methods for evaluating the innovative ability of GPT-type models to generate text mainly rely on manual evaluation and traditional automatic evaluation indicators:
[0003] 1. Manual evaluation: Experts or users score and comment on the generated text to evaluate its innovation and quality. Although this method can provide a certain reference value, it has problems such as strong subjectivity, low efficiency, and high cost, making it difficult to apply on a large scale.
[0004] 2. Traditional evaluation indicators: Use traditional indicators such as BLEU and ROUGE to evaluate the generated text. These indicators are mainly based on the literal similarity between texts and cannot effectively measure the innovation and creativity of the text. Especially for long texts and complex contexts, the evaluation results are often not accurate and comprehensive enough. Summary of the invention
[0005] The present invention aims to address the deficiencies of the prior art and proposes a text innovation ability evaluation method based on a GPT-type model. By analyzing the similarity between the text generated by the model and the existing text, the innovation ability of the GPT-type model is objectively evaluated.
[0006] To achieve the above object, the present invention provides the following scheme: a method for evaluating the text innovation ability of a GPT model, comprising the following steps:
[0007] S1. Collect text data from public databases, clean, classify and annotate the text data to obtain a data set in a standardized format;
[0008] S2. Establishing a GPT model, and training the GPT model based on the data set;
[0009] S3. Provide a specific topic and generate text content based on the trained GPT model; evaluate the innovation ability of the text content based on the accuracy and repetition rate to obtain an evaluation result.
[0010] Further preferably, S2 comprises the following steps:
[0011] S21, constructing the GPT model and performing initial settings on the GPT model;
[0012] S22. Providing standardized text data to the GPT model to train the GPT model.
[0013] Further preferably, in S3, the quality of the generated content is controlled by setting generation parameters; the generation parameters include: text length and generation quantity.
[0014] Further preferably, the method for calculating the accuracy rate includes:
[0015] Correctness = (predicted words / real words) × 100%;
[0016] The calculation method of the repetition rate includes:
[0017] Repeat rate = (predicted word / provided word) x 100%.
[0018] The present invention also provides a GPT model text innovation ability evaluation device, including: a data acquisition system, a training system and an evaluation system;
[0019] The data collection system is used to collect text data from public databases, and clean, classify and annotate the text data to obtain a data set in a standardized format;
[0020] The training system is used to establish a GPT model and train the GPT model based on the data set;
[0021] The evaluation system is used to provide specific topics and generate text content based on the trained GPT model; and evaluate the innovation ability of the text content based on the accuracy and repetition rate to obtain an evaluation result.
[0022] Further preferably, the training system comprises: a model building module and a training module;
[0023] The model building module is used to build the GPT model and perform initial settings on the GPT model;
[0024] The training module is used to provide standardized text data to the GPT model to train the GPT model.
[0025] Further preferably, the evaluation system controls the quality of the generated content by setting generation parameters; the generation parameters include: text length and generation quantity.
[0026] Further preferably, the method for calculating the accuracy rate includes:
[0027] Correctness = (predicted words / real words) × 100%;
[0028] The calculation method of the repetition rate includes:
[0029] Repeat rate = (predicted word / provided word) x 100%.
[0030] Compared with the prior art, the present invention has the following beneficial effects:
[0031] The present invention selects a public text data database, sets the search scope and conditions, and collects text data within a certain time limit. This process efficiently and accurately collects a large amount of high-quality data, ensuring the integrity and diversity of the basic data required for evaluation. Keyword and word frequency statistics are performed, and the data is organized into a standardized format, which provides high-quality input data for subsequent analysis and improves the automation and efficiency of data processing. Providing standardized data and performing keyword analysis enables the GPT model to be accurately trained based on data within a specific time limit, thereby improving the ability to generate new texts. Data processing and analysis during model training ensure the high quality and consistency of training data and optimize the training effect of the model. The calculation of repetition rate and similarity analysis provide quantitative evaluation indicators, making the evaluation results of the model's innovation ability more accurate and reliable. The lower the repetition rate, the stronger the model's innovation ability. Through this evaluation method, the innovative performance of the model when generating new texts can be objectively reflected. By saving the generated text and similarity calculation results, structured data is formed, and a comprehensive analysis is performed to evaluate the innovation ability of the model under different conditions. A detailed model text innovation ability evaluation report is generated, providing researchers and developers with comprehensive analysis results and improvement suggestions. The method of the present invention is applicable to text generation tasks of different types and fields, including news writing, academic papers, novel creation, etc., and has broad application prospects. By providing a universal text innovation ability evaluation method, it provides support for various text generation application scenarios, thereby improving the practicality and applicability of the method. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0033] Figure 1 This is a schematic diagram of a flow chart of a method for evaluating text innovation ability of a GPT model according to an embodiment of the present invention;
[0034] Figure 2 A schematic diagram of a situation in which predicted keywords are obtained based on provided keywords according to an embodiment of the present invention;
[0035] Figure 3 This is a schematic diagram of the situation in which the abstract part of an embodiment of the present invention uses four years of data to predict the fifth year's words. DETAILED DESCRIPTION
[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0037] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0038] Embodiment 1:
[0039] like Figure 1 As shown, this embodiment provides a method for evaluating the text innovation ability of a GPT model, comprising the following steps:
[0040] S1. Collect text data from public databases, clean, classify and annotate the text data to obtain a data set in a standardized format.
[0041] According to the public database, all papers, articles and books published in a specific journal from 2000 to 2019 are collected, and the retrieved records are exported as plain text files, including the full record and cited references. Then, the collected text materials are cleaned, classified and annotated to generate a data set in a standardized format; a part of the preprocessed data is selected as the training data set. At the same time, the words that appear most frequently in the title, keywords and abstracts of all the documents in the journal each year are counted.
[0042] S2. Establish a GPT model and train the GPT model based on the dataset.
[0043] Further, S2 comprises the following steps:
[0044] S21. Build a GPT model and perform initial settings on the GPT model.
[0045] In this embodiment, the GPT model uses ChatGPT 4. The initial settings include: telling GPT the source of the training text and the purpose of the training to ensure that the model understands the background and purpose of the analysis.
[0046] S22. Provide standardized text data to the GPT model to train the GPT model.
[0047] Provide standardized text data to the GPT model, showing the most frequently appearing words in the titles, keywords, and abstracts of all documents. Ensuring that GPT understands the source of the data helps understand the significance and academic value of the input data, and ensures that the prediction is based on the academic standards of the field. Use the training data set to train the GPT model to improve its ability to generate new text. Require the GPT model to predict keywords that may appear in the future environmental field based on the provided historical data, trends, and patterns. Repeat the data provision and hotspot prediction process, perform multiple cycles of prediction, and comprehensively analyze the results of each prediction. Comprehensively analyze the saved prediction results to obtain the most likely predicted keywords. For example, provide data from 2000-2009 and ask it to predict the most frequently appearing words from 2010-2019. See details. Figure 2 and Table 1. For example, if data is provided every 4 years and the data for the fifth year is predicted, see Figure 3 and Table 2.
[0048] Table 1
[0049]
[0050] Table 2
[0051]
[0052] S3. Provide specific topics and generate text content based on the trained GPT model; evaluate the innovation ability of the text content based on the accuracy and repetition rate to obtain the evaluation results.
[0053] Furthermore, in this embodiment, the quality of the generated content is controlled by setting generation parameters; the generation parameters include: text length and number of generated content, etc. Provide the GPT model with a title and keywords to regenerate the summary. Compare the generated new text content with other collected information, and save the generated text and similarity calculation results. Calculate its accuracy and repetition rate, and evaluate the innovation ability of the model by the repetition rate level. The lower the repetition rate, the stronger the innovation ability of the model; the higher the repetition rate, the weaker the innovation ability of the model. Comprehensively analyze the results of multiple generations to evaluate the innovation ability of the model under different conditions. Generate a model text innovation ability evaluation report based on the analysis results.
[0054] In this embodiment, the calculation method of the accuracy rate includes:
[0055] Correctness = (predicted words / real words) × 100%;
[0056] The calculation method of repetition rate includes:
[0057] Repeat rate = (predicted word / provided word) x 100%.
[0058] Embodiment 2:
[0059] This embodiment provides a device for evaluating the text innovation ability of a GPT model, including: a data collection system, a training system, and an evaluation system.
[0060] The data collection system is used to collect text data from public databases, and clean, classify and annotate the text data to obtain data sets in a standardized format.
[0061] According to the public database, all papers, articles and books published in a specific journal from 2000 to 2019 are collected, and the retrieved records are exported as plain text files, including the full record and cited references. Then, the collected text materials are cleaned, classified and annotated to generate a data set in a standardized format; a part of the preprocessed data is selected as the training data set. At the same time, the words that appear most frequently in the title, keywords and abstracts of all the documents in the journal each year are counted.
[0062] The training system is used to build a GPT model and train the GPT model based on the data set.
[0063] In this embodiment, the training system includes: a model construction module and a training module; the model construction module is used to construct a GPT model and perform initial settings on the GPT model; the training module is used to provide standardized text data to the GPT model and train the GPT model.
[0064] Among them, the initial settings include: telling GPT the source of the training text and the purpose of training to ensure that the model understands the context and purpose of the analysis.
[0065] Provide standardized text data to the GPT model, showing the most frequently appearing words in the titles, keywords, and abstracts of all documents. Ensure that GPT understands the source of the data, which helps understand the meaning and academic value of the input data and ensures that the predictions are based on the academic standards of the field. Use the training data set to train the GPT model to improve its ability to generate new text. Require the GPT model to predict keywords that may appear in the future environmental field based on the historical data, trends, and patterns provided. Repeat the data provision and hot spot prediction process, perform multiple cycles of prediction, and comprehensively analyze the results of each prediction. Comprehensively analyze the saved prediction results to obtain the most likely predicted keywords.
[0066] The evaluation system is used to provide specific topics and generate text content based on the trained GPT model; and evaluate the innovation ability of the text content based on the accuracy and repetition rate to obtain evaluation results.
[0067] Further preferably, the evaluation system controls the quality of generated content by setting generation parameters; generation parameters include: text length and number of generated content, etc. Provide the GPT model with a title and keywords to regenerate the summary. Compare the generated new text content with other collected information, and save the generated text and similarity calculation results. Calculate its accuracy and repetition rate, and evaluate the innovation ability of the model by the repetition rate level. The lower the repetition rate, the stronger the innovation ability of the model; the higher the repetition rate, the weaker the innovation ability of the model. Comprehensively analyze the results of multiple generations to evaluate the innovation ability of the model under different conditions. Generate a model text innovation ability evaluation report based on the analysis results.
[0068] The calculation method of accuracy rate includes:
[0069] Correctness = (predicted words / real words) × 100%;
[0070] The calculation method of repetition rate includes:
[0071] Repeat rate = (predicted word / provided word) x 100%.
[0072] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.
Claims
1. A method for evaluating the text innovation ability of a GPT model, characterized in that: The following steps are involved: S1. Collect text data from public databases, clean, classify and annotate the text data to obtain a data set in a standardized format; S2. Establishing a GPT model, and training the GPT model based on the data set; S3. Provide a specific topic and generate text content based on the trained GPT model; evaluate the innovation ability of the text content based on the accuracy and repetition rate to obtain an evaluation result.
2. According to claim 1, a method for evaluating the text innovation ability of a GPT model is characterized in that: S2 includes the following steps: S21, constructing the GPT model and performing initial settings on the GPT model; S22. Providing standardized text data to the GPT model to train the GPT model.
3. According to claim 1, a method for evaluating the text innovation ability of a GPT model is characterized in that: In S3, the quality of the generated content is controlled by setting generation parameters; The generation parameters include: text length and generation quantity.
4. According to claim 1, a method for evaluating the text innovation ability of a GPT model is characterized by: The calculation method of the accuracy rate includes: Correctness = (predicted words / real words) × 100%; The calculation method of the repetition rate includes: Repeat rate = (predicted word / provided word) x 100%.
5. A device for evaluating the text innovation ability of a GPT model, the device being used to implement the method according to any one of claims 1 to 4, characterized in that: include: Data collection system, training system and evaluation system; The data collection system is used to collect text data from public databases, and clean, classify and annotate the text data to obtain a data set in a standardized format; The training system is used to establish a GPT model and train the GPT model based on the data set; The evaluation system is used to provide specific topics and generate text content based on the trained GPT model; and evaluate the innovation ability of the text content based on the accuracy and repetition rate to obtain an evaluation result.
6. According to claim 5, a GPT model text innovation ability evaluation device is characterized by: The training system comprises: a model building module and a training module; The model building module is used to build the GPT model and perform initial settings on the GPT model; The training module is used to provide standardized text data to the GPT model to train the GPT model.
7. According to claim 5, a GPT model text innovation ability evaluation device is characterized by: The evaluation system controls the quality of the generated content by setting generation parameters; The generation parameters include: text length and generation quantity.
8. According to claim 5, a GPT model text innovation ability evaluation device is characterized by: The calculation method of the accuracy rate includes: Correctness = (predicted words / real words) × 100%; The calculation method of the repetition rate includes: Repeat rate = (predicted word / provided word) x 100%.
Citation Information
Patent Citations
Single-document abstract generation model construction method and device based on multi-task learning
CN111723196A
Text quality evaluation model training method and text quality determination method
CN113392218A
Text abstract generation method based on keyword information and multi-head attention mechanism
CN116010590A
Outpatient service electronic medical record generation method based on Chinese medical big model
CN117253576A
Comment generation method based on natural language processing
CN118410169A
