A framework and method for generating full-text news commentary articles based on RAG large language model

Abstract

The invention discloses a full-text generation framework of a news review article based on a large language model of RAG, and relates to the technical field of large language model text generation, comprising a comment head generation module: using news event information and keywords input by a user, searching a news fact database, obtaining complete news event information, and generating a comment head of the news review article through a large language model; a thesis and evidence generation module: on the basis of the comment head, after the user confirms the article position and / or article direction of the news review article, the large language model generates a core thesis; on this basis, the user inputs an outline type and a text length, and the large language model generates an thesis outline, and the large language model generates the thesis and evidence of the news review article according to the thesis outline and searching a news database; a viewpoint novelty sorting module: real-time collection of real-time new thesis, establishment of a novelty sorting model, and generation of innovative and novel thesis; ensuring that the core thesis generated by the thesis and evidence generation module meets the requirements, is written in the current popular language paradigm, and meets the current era characteristics; an ending and title generation module: the large language model generates the ending and title of the news review article in combination with the thesis and evidence of the news review article.

Description

Technical Field

[0001] The present invention relates to the technical field of large language model text generation, and in particular to a framework and method for generating full-text news commentary articles based on a large language model of RAG. Background Art

[0002] Since the Turing test was proposed in the 1950s, humans have been exploring the problem of machines mastering language intelligence. Large language models (LLMs) refer to pre-trained language models with large-scale parameters. By pre-training on a large corpus, it can understand and generate natural language and demonstrate strong capabilities in a variety of natural language processing tasks. The research and development of LLMs has made significant progress. One of the famous examples is ChatGPT, a powerful artificial intelligence chatbot developed based on LLM technology. Its core algorithmic principle is based on the Transformer network structure, which is a revolutionary architecture in the field of natural language processing (NLP) and was first introduced in the paper "Attention is All You Need" proposed by Vaswani et al. in 2017. The core feature of the Transformer architecture is the Self-Attention Mechanism, which enables the Transformer to effectively process sequence data, especially when dealing with long-distance dependencies.

[0003] The current mainstream approach to writing news commentary articles based on large models is to continue training news forecasts based on Transformer on the open source large model to obtain a base model. This process usually includes two stages: pre-training and fine-tuning. In the pre-training stage, the model captures the general rules and knowledge of language by learning a large amount of unlabeled text data. In the fine-tuning stage, the model is trained on labeled data in a specific field to make it better adapted to specific tasks.

[0004] Then, based on the base model, we construct the instruction task of news commentary articles, fine-tune the instructions of the base model, and obtain a large news model that can write news. This process mainly includes task definition, data collection and processing, model training and evaluation. In the task definition stage, it is necessary to clarify the specific requirements for writing news commentary articles, such as article structure, content and style. In the data collection and processing stage, it is necessary to collect a certain amount of news commentary articles as training data and perform preprocessing, such as word segmentation and removal of stop words. In the model training stage, through instruction fine-tuning, the model can generate corresponding commentary articles based on the given news information. Finally, in the evaluation stage, the news commentary articles generated by the model are evaluated through certain evaluation indicators, such as ROUGE, BLEU, etc., to ensure their quality and accuracy.

[0005] Finally, the generative capabilities of the large model are used to create news articles. At this stage, users can provide the model with key information about the news event, such as time, location, and people, and the model will generate a complete news review article based on this information. This process involves multiple technologies such as text generation, summary generation, and logical reasoning, which enables the model to generate news articles with a certain logical structure and readability.

[0006] Directly using big models to generate commentary articles will result in the lack of a lead or main point in the generated news articles due to the uncontrollable sampling of big models. Big models use historical data to produce new texts, so they may tend to repeat previous views and opinions, and lack innovation and novelty. Currently, big models are trained using historical data, and the data generated lacks timeliness and is prone to hallucinations. Summary of the invention

[0007] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is how to generate news commentary articles that are similar in "form" to ensure that the structure of the article meets the requirements of news commentary articles, and generate news commentary articles that are similar in "spirit" to ensure that the content of the article is interesting.

[0008] To achieve the above objectives, the present invention provides a framework for generating full-text news commentary articles based on a large language model of RAG, including

[0009] Comment header generation module: uses the news event information and keywords input by the user to search the news fact database, obtain the complete information of the news event, and generate the comment header of the news comment article through the large language model;

[0010] Thesis and evidence generation module: Based on the commentary, after the user confirms the article position and / or article direction of the news commentary article, the large language model generates the core thesis; on this basis, the user inputs the outline type and text length, and the large language model generates the thesis outline. The large language model generates the thesis and evidence of the news commentary article based on the thesis outline and searches the news database;

[0011] Opinion novelty ranking module: collect new arguments in real time, establish a novelty ranking model, and generate innovative and novel arguments; ensure that the core arguments generated by the argument and evidence generation module meet the requirements, are written in the currently popular language paradigm, and are in line with the current era characteristics;

[0012] Ending and title generation module: The ending and title of the news commentary article are generated by combining the large language model with the arguments and evidence of the news commentary article.

[0013] Preferably, after obtaining the complete information of the news event, the comment header generation module can choose to extract markers and / or highlights by a large language model, and then generate the comment header of the news comment article by the large language model.

[0014] Preferably, after the user confirms the article stance and / or article direction of the news commentary article and before the large language model generates the core argument, the argument and evidence generation module allows the user to input his own argument and / or evidence.

[0015] Preferably, after the large language model generates the argument outline, the argument and evidence generation module uses the large language model to control the length of the generated evidence in combination with the argument outline, recalls the evidence in combination with the news database, and regenerates the evidence until it meets the requirements.

[0016] Preferably, the real-time collection of new arguments means that the opinion novelty ranking module crawls hot events from the hot list pages of major portal websites at different times, stores them in the database, and then crawls the hot comments of the hot events at regular intervals according to these hot events.

[0017] Preferably, a large language model is called to extract opinions from the original texts of popular user comments, construct data pairs of "event-direction-argument", and store them in a basic opinion library.

[0018] Preferably, after obtaining a large number of opinions on the same event, the opinions should be sorted to obtain uniqueness and innovation.

[0019] Preferably, for the ranking of viewpoints, a novelty model is constructed, and its training data is constructed by using GPT4 for scoring, scoring from a total of 9 dimensions, including the perspective of topic selection, the novelty of the main viewpoint, whether the analysis of the problem is profound, whether there is a logical hierarchy between the arguments, whether it can resonate with the majority of readers, whether an interdisciplinary perspective is used, whether the emotional tendency of the viewpoint itself is positive, the value tendency of the viewpoint itself, and whether the proposed suggestions are constructive. Finally, the average score is calculated, and then a reward model is used to generalize the GPT 4 scoring results; the loss function is as follows:

[0020]

[0021] Where x represents instructions, y expresses opinions, and r θ represents the output value of the reward model with parameter θ, and σ represents the sigmoid function.

[0022] Preferably, by constructing an argument database and adding search engine retrieval methods, multiple rounds of content authenticity verification are performed on the text, which greatly reduces the illusion of the large model and ensures the authenticity of the arguments and their support for the thesis.

[0023] Preferably, the argument library includes a basic argument library, an event library and a retrieval engine; the basic argument library contains a library of traditional classics, basic economic principles and basic legal data, which are used to support the recall of rational arguments; the event library mainly calls the six elements of the event from the big model of news and forum data, which is used to proofread the six elements one by one after generating the argument to reduce the illusion of the model; the retrieval engine uses the cause and argument to recall relevant documents through retrieval, and generates arguments based on the recalled reference materials.

[0024] Preferably, historical news commentary articles published by Xinhua News Agency are used for data tagging to obtain the reasons and arguments, and then the arguments are recalled through search engine data to construct reference materials, which are then used as input and the arguments of the original text as output to conduct supervised fine-tuning training corpus of the own model, and a large model for citation generation is trained to improve the quality of model generation and further reduce hallucinations.

[0025] Another aspect of the present invention provides a method for generating a full text of a news commentary article based on a large language model of RAG, the method comprising the following steps:

[0026] (1) Using the news event information and keywords input by the user, the news event database is searched to obtain the complete information of the news event, and the comment head of the news comment article is generated through the large language model;

[0027] (2) Based on the commentary, after the user confirms the article's stance and / or direction, the large language model generates the core argument;

[0028] (3) By collecting new arguments in real time and establishing a novelty ranking model, we can generate arguments that are innovative and novel; ensure that the generated core arguments meet the requirements, are written in the currently popular language paradigm, and are in line with the current era characteristics;

[0029] (5) The user inputs the outline type and text length, and the large language model generates an outline of the argument based on the core argument. The large language model generates the argument and evidence of the news commentary article based on the outline of the argument and by searching the news database;

[0030] (6) By building an argument database and using search engine retrieval methods, the text is subjected to multiple rounds of content authenticity verification, which greatly reduces the illusion of the large model and ensures the authenticity of the argument and its support for the thesis;

[0031] (7) The large language model combines the arguments and evidence of the news commentary article to generate the ending and title of the news commentary article;

[0032] (8) Finally, the full text of the news commentary article is formed to meet the user's requirements.

[0033] Technical effects of the present invention:

[0034] (1) Build a structured review article generation process, combine it with human assistance, and implement AI (Artificial Intelligence) + HI (Human Intervene), so that the generated article structure is clearer and the content is more controllable;

[0035] (2) Use online crowdsourcing to collect new arguments in real time and establish a novelty ranking model to generate innovative and novel arguments;

[0036] (3) By building an event library and combining it with an online search engine to conduct multiple rounds of content authenticity verification, the hallucination phenomenon of large models can be reduced.

[0037] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a structural schematic and flow chart of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following describes several preferred embodiments of the present invention with reference to the drawings in the specification, so that the technical content is clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0040] In the drawings, components with the same structure are indicated by the same numerical reference numerals, and components with similar structures or functions are indicated by similar numerical reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present invention does not limit the size and thickness of each component. In order to make the illustration clearer, the thickness of the components is appropriately exaggerated in some places in the drawings.

[0041] like Figure 1 As shown, this preferred embodiment provides a news review article full text generation framework based on the RAG large language model, including

[0042] Comment header generation module: uses the news event information and keywords input by the user to search the news fact database, obtain the complete information of the news event, and generate the comment header of the news comment article through the large language model;

[0043] Thesis and evidence generation module: Based on the commentary, after the user confirms the article position and / or article direction of the news commentary article, the large language model generates the core thesis; on this basis, the user inputs the outline type and text length, and the large language model generates the thesis outline. The large language model generates the thesis and evidence of the news commentary article based on the thesis outline and searches the news database;

[0044] Opinion novelty ranking module: collect new arguments in real time, establish a novelty ranking model, and generate innovative and novel arguments; ensure that the core arguments generated by the argument and evidence generation module meet the requirements, are written in the currently popular language paradigm, and are in line with the current era characteristics;

[0045] Ending and title generation module: The ending and title of the news commentary article are generated by combining the large language model with the arguments and evidence of the news commentary article.

[0046] Large-scale language models (LLMs) are more powerful than anything we have seen before in natural language processing (NLP). GPT series models, Llama series models, Gemini, and other large language models have demonstrated impressive language and knowledge mastery in multiple evaluation benchmarks, surpassing human baselines.

[0047] However, large language models also exhibit many shortcomings. They often make up facts and lack knowledge when dealing with domain-specific or highly specialized queries. For example, when the information sought is beyond the model’s training data or when the latest data is required, LLMs may not provide accurate answers. This limitation poses a challenge to deploying generative AI in actual production environments, as blindly using black-box LLMs may not be sufficient.

[0048] Retrieval-Augmented Generation (RAG),RAG is a technique that uses an external knowledge base to supplement the context of a large language model (LLM) and generate responses. Notably, RAG combines parameterized knowledge in LLM and non-parametric external knowledge, alleviates the hallucination problem, identifies timely information through retrieval technology, and enhances the accuracy of responses. In addition, by citing sources, RAG increases the transparency and user trust of model output.

[0049] RAG enhances the reliability of these models by basing their responses on factual data retrieved from a vector database. This approach not only guarantees the accuracy of the generated information, but also provides a reference point for users to verify the data. In addition, by focusing on factual data related to a specific domain, RAG allows large language models to focus on domain-specific tasks more effectively. This produces more accurate, reliable, and contextually relevant outputs.

[0050] Retrieval-augmented generation (RAG) is a sophisticated approach that combines the strengths of retrieval-based models with seq2seq generation. Unlike traditional seq2seq (text-to-text) models, RAG introduces an intermediate step that involves retrieving relevant documents before generating a response.

[0051] Here’s how it works:

[0052] Document Retrieval: Initially, an input sequence (usually a question or prompt) is used to perform a semantic search in the vector database. This search retrieves a set of relevant documents. These documents serve as an external source of knowledge that the model can refer to. Note that any type of data can be stored in the vector database as long as they are converted into vector embeddings.

[0053] Combine input with retrieved documents: The input sequence is combined with the retrieved documents to form an extended context. This extended context contains additional information from the original input and the retrieved documents.

[0054] Pass to Decode Transformer: The extended context is then passed to the Decode Transformer. It is important to emphasize that RAG is compatible with any Decode Transformer, not just a specific one. This flexibility enables it to be integrated into a variety of architectures and applications.

[0055] Generate Response: The Decode transformer processes the extended context and generates a response. The response is based not only on the input sequence but also on the information in the retrieved document. This ensures that the output is factually based and relevant to the input.

[0056] By incorporating external knowledge sources through document retrieval and using a decoder-transformer, RAG effectively bridges the gap between retrieval-based models and generative models. This can provide more informed and accurate responses for your specific task compared to traditional seq2seq models.

[0057] On this basis, after obtaining the complete information of the news event, the comment header generation module can choose to extract markers and / or highlights through the large language model, display the highlights in bold in the text, and then generate the comment header of the news comment article through the large language model.

[0058] On this basis, after the user confirms the article stance and / or direction of the news commentary article, the argument and evidence generation module allows the user to input his or her own arguments and / or evidence before the large language model generates the core arguments.

[0059] On this basis, after the big language model generates the argument outline, the argument and evidence generation module uses the big language model to control the length of the generated arguments based on the argument outline, recalls the arguments based on the news database, and regenerates the arguments until they meet the requirements.

[0060] On this basis, real-time collection of new arguments means that the opinion novelty ranking module crawls hot events from the hot list pages of major portals such as Zhihu, NetEase, Toutiao, and Weibo in different time periods, and then stores them in the Mongodb database, and then regularly crawls hot comments on these hot topics for these hot topics. Call a large language model (such as Qianwen 14B) to extract opinions from the original text of users' hot comments, build "event-direction-argument" data pairs, and store them in the basic opinion library. For the same event, after obtaining a large number of opinions, it is necessary to sort the opinions to obtain uniqueness and innovation. For the sorting of opinions, a novelty model is constructed, and its training data is constructed by using GPT4 for scoring, scoring from a total of 9 dimensions, including the perspective of topic selection, the novelty of the main point of view, whether the analysis of the problem is profound, whether there is a logical level between the arguments, whether it can resonate with the majority of readers, whether the cross-disciplinary perspective is used, whether the emotional tendency of the opinion itself is positive, the value tendency of the opinion itself, and whether the suggestions are constructive. Finally, the average score is calculated, and then a reward model is used to generalize the GPT 4 scoring results; the loss function is as follows:

[0061]

[0062] Where x represents instructions, y expresses opinions, and r θ represents the output value of the reward model with parameter θ, and σ represents the sigmoid function.

[0063] By building an argument database and adding a search engine search method, the text is verified for multiple rounds of content authenticity, which greatly reduces the illusion of the big model and ensures the authenticity of the argument and the support of the argument for the thesis. The argument database includes a basic argument database, an event database, and a search engine; the basic argument database contains a traditional classic library (such as the Analects, Mencius, Doctrine of the Mean, Tang poetry and Song lyrics, etc.), basic economic principles (microeconomics and macroeconomics, etc.) and basic legal data (such as the Civil Code, etc.), which are used to support the recall of those rational arguments; the event database mainly calls the six elements of the big model for events from news and forum data, which is used to proofread the six elements one by one after generating the argument to reduce the illusion of the model; the search engine uses the cause and the thesis to recall relevant documents through retrieval, and generates arguments based on the recalled reference materials.

[0064] Furthermore, we use historical news commentary articles published by Xinhua News Agency to mark data, obtain the reasons and arguments, and then recall the arguments through search engine data to build reference materials. Then, we use the original arguments as input and the original text as output to conduct supervised fine-tuning training corpus of our own model, train a large model for citation generation, improve the quality of model generation, and further reduce hallucinations.

[0065] By recalling similar arguments, we can make the results generated by the large model more persuasive rather than repetitive. Through the above algorithm framework, we can make the review articles similar in both "form" and "spirit".

[0066] like Figure 1 As shown, this preferred embodiment also provides a method for generating the full text of a news commentary article based on a large language model of RAG, the method comprising the following steps:

[0067] (1) Using the news event information and keywords input by the user, the news event database is searched to obtain the complete information of the news event, and the comment head of the news comment article is generated through the large language model;

[0068] (2) Based on the commentary, after the user confirms the article's stance and / or direction, the large language model generates the core argument;

[0069] (3) By collecting new arguments in real time and establishing a novelty ranking model, we can generate arguments that are innovative and novel; ensure that the generated core arguments meet the requirements, are written in the currently popular language paradigm, and are in line with the current era characteristics;

[0070] (5) The user inputs the outline type and text length, and the large language model generates an outline of the argument based on the core argument. The large language model generates the argument and evidence of the news commentary article based on the outline of the argument and by searching the news database;

[0071] (6) By building an argument database and using search engine retrieval methods, the text is subjected to multiple rounds of content authenticity verification, which greatly reduces the illusion of the large model and ensures the authenticity of the argument and its support for the thesis;

[0072] (7) The large language model combines the arguments and evidence of the news commentary article to generate the ending and title of the news commentary article;

[0073] (8) Finally, the full text of the news commentary article is formed to meet the user's requirements.

[0074] In this method, the full text is generated step by step, and the user can intervene at each step of the generation, such as modifying the main argument. The algorithm can generate subsequent sub-arguments, arguments and endings based on the user's modified results, realizing AI (Artificial Intelligence) + HI (Human Intervene) to improve the generation quality.

[0075] The preferred specific embodiments of the present invention are described in detail above. It should be understood that ordinary technicians in the field can make many modifications and changes based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by technicians in the technical field based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A news review article full-text generation system based on a large language model of RAG, characterized in that: include Comment header generation module: uses the news event information and keywords input by the user to search the news fact database to obtain the complete information of the news event. The large language model generates the comment header of the news comment article based on the complete information of the news event. Thesis and evidence generation module: Based on the commentary, after the user confirms the article stance and / or article direction of the news commentary, the large language model generates the core thesis according to the commentary, article stance and / or article direction; on this basis, the user inputs the outline type and text length, and the large language model generates thesis outline according to the core thesis. The large language model generates the thesis and evidence of the news commentary based on the basic thesis library and basic evidence library, according to the thesis outline and retrieves the news fact library; Opinion novelty ranking module: collect new arguments in real time, establish a novelty ranking model, and generate innovative and novel arguments to ensure that the core arguments generated by the argument and evidence generation module meet the requirements, are written in the currently popular language paradigm, and are in line with the current era characteristics; The real-time collection of new arguments means that the opinion novelty ranking module crawls hot events from the hot list pages of major portal websites in different time periods and stores them in the event library, and then crawls the hot comments of hot events in a regular manner; Call the large language model to extract opinions from the original text of users' popular comments, build data pairs of "event-direction-argument", and store them in the basic argument library; After obtaining a large number of opinions on the same event, sort the opinions and obtain innovative and novel arguments; For the ranking of opinions, a novelty ranking model was constructed. The training data was constructed by using GPT4 to score the opinions. The opinions were scored from nine dimensions: the perspective of topic selection, the novelty of the main opinion, whether the analysis was profound, whether there was a logical hierarchy between the arguments, whether they could resonate with the readers, whether they used an interdisciplinary perspective, whether the sentiment of the opinion itself was positive, the value orientation of the opinion itself, and whether the suggestions were constructive. The average score was calculated, and a reward model was used to generalize the GPT 4 scoring results. Ending and title generation module: The ending and title of the news commentary article are generated by combining the large language model with the arguments and evidence of the news commentary article.

2. The news review article full-text generation system based on the RAG large language model according to claim 1 is characterized in that: After obtaining the complete information of the news event, the comment header generation module extracts the markers and / or highlights by the large language model, and then generates the comment header of the news comment article by the large language model.

3. The news review article full-text generation system based on the RAG large language model according to claim 1 is characterized in that: After the user confirms the article stance and / or article direction of the news commentary article, the argument and evidence generation module allows the user to input his / her own argument and / or evidence before the large language model generates the core argument.

4. The news review article full-text generation system based on the RAG large language model according to claim 1 is characterized in that: After the large language model generates the argument outline, the argument and evidence generation module uses the large language model to control the length of the generated evidence in combination with the argument outline, recalls the evidence in combination with the news fact database, and regenerates the evidence until it meets the requirements.

5. The news review article full-text generation system based on the RAG large language model according to claim 1 is characterized in that: By building an argument database and using search engine retrieval methods, we conduct multiple rounds of content authenticity verification on the text to significantly reduce the illusion of the big model and ensure the authenticity of the argument and its support for the thesis. The argument database includes a basic argument database, an event database and a search engine; the basic argument database contains a traditional classic book database, basic economic principles and basic legal data, which are used to support the recall of rational arguments; the event database uses a large model to extract the six elements of an event from news and forum data, which is used to proofread the six elements one by one after generating the argument to reduce the illusion of the model; the search engine uses the cause and argument to recall relevant documents through retrieval, and generates arguments based on the recalled reference materials; Historical news commentary articles are used for data labeling to obtain the reasons and arguments. The arguments are then recalled through search engine data. Reference materials are constructed as input and the original arguments are used as output. Supervised fine-tuning is performed to train a large language model for generating arguments.

6. A method for generating full-text news commentary articles based on a large language model of RAG, applied to the full-text news commentary article generation system based on a large language model of RAG according to any one of claims 1 to 5, characterized in that: The method comprises the following steps: (1) Using the news event information and keywords input by the user, the news fact database is searched to obtain the complete information of the news event, and the comment head of the news comment article is generated through the large language model; (2) Based on the comment head, after the user confirms the article stance and / or article direction of the news comment article, the large language model generates the core argument based on the comment head and the article stance and / or article direction; (3) By collecting new arguments in real time and establishing a novelty ranking model, we can generate arguments that are innovative and novel; to ensure that the generated core arguments meet the requirements, are written in the current popular language paradigm, and are in line with the current era characteristics; (4) The user inputs the outline type and text length, and the large language model generates an outline of the argument based on the core argument. The large language model generates the argument and evidence of the news commentary article based on the outline of the argument and by searching the news database; (5) By building an argument database and using search engine retrieval methods, the text is subjected to multiple rounds of content authenticity verification to significantly reduce the illusion of the large model and ensure the authenticity of the argument and its support for the thesis; (6) The large language model combines the arguments and evidence of the news commentary article to generate the ending and title of the news commentary article; (7) Finally, the full text of the news commentary article is formed to meet the user's requirements.

Images