Data generation method and device based on artificial intelligence, computer equipment and medium

Abstract

The invention belongs to the technical field of artificial intelligence, and relates to a data generation method and device based on artificial intelligence, computer equipment and a storage medium, and the method comprises the steps: obtaining news data based on a crawling module; inputting the news data into the target large language model, and calling a performance function corresponding to the target large language model; performing news classification on the news data based on the classification function to obtain news classification data; screening the news data based on the topic screening function and the news classification data to obtain first news data; performing de-duplication screening on the first news data based on a de-duplication function to obtain second news data; performing abstract generation on the second news data based on the summary function to obtain initial news abstract data; and optimizing the initial news abstract data to obtain target news abstract data. In addition, the target news summary data may be stored in the block chain. The processing efficiency of news abstract generation is improved, and meanwhile, the accuracy of the generated target news abstract data is ensured.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence development technology and financial technology, and in particular to artificial intelligence-based data generation methods, devices, computer equipment and storage media. Background Art

[0002] In today's era of rapid development of information technology, the Internet, as the main channel for information dissemination, covers a wide range of information and updates at a fast speed, posing an unprecedented challenge to the information processing capabilities of all walks of life. The insurance industry, as an important part of the financial market, is also facing the profound impact of this change. With the rapid changes in the market environment and the continuous emergence of technological innovations, insurance companies need to grasp the latest trends of the industry, policies and regulations, market trends and competitors in a timely and accurate manner in order to make scientific decisions and maintain competitive advantages.

[0003] Traditional information processing methods, especially the collection and organization of news information, rely on manual operations, such as manual screening, reading and summarizing news highlights to form summaries. Although this way of generating news summaries can ensure the pertinence of information to a certain extent, its inherent limitations are obvious. It has the disadvantages of low processing efficiency, difficulty in coping with the influx of massive information, and is limited by personal knowledge structure and judgment ability, so it is impossible to guarantee the accuracy of the generated news summaries. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a data generation method, device, computer equipment and storage medium based on artificial intelligence to solve the technical problems that the existing news summary generation method has low efficiency and cannot guarantee the accuracy of the generated news summary.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a data generation method based on artificial intelligence, which adopts the following technical solutions:

[0006] Get news data based on the preset crawling module;

[0007] Input the news data into a preset target large language model, and call a preset function corresponding to the target large language model; wherein the function includes a classification function, a topic screening function, a deduplication function, and a summary function;

[0008] Performing news classification processing on the news data based on the classification function to obtain corresponding news classification data;

[0009] Filtering the news data based on the topic filtering function and the news classification data to obtain corresponding first news data;

[0010] Performing deduplication screening processing on the first news data based on the deduplication function to obtain corresponding second news data;

[0011] Performing summary generation processing on the second news data based on the summary function to obtain corresponding initial news summary data;

[0012] The initial news summary data is optimized to obtain corresponding target news summary data.

[0013] Furthermore, the step of performing news classification processing on the news data based on the classification function to obtain corresponding news classification data specifically includes:

[0014] Based on the classification function, preprocessing the news data to obtain corresponding first designated news data;

[0015] Calling pre-built classification models;

[0016] Classify the specified news data based on the classification model to obtain corresponding classification results;

[0017] News classification data corresponding to the news data is generated based on the classification result.

[0018] Furthermore, the step of filtering the news data based on the topic filtering function and the news classification data to obtain the corresponding first news data specifically includes:

[0019] Based on the subject screening function, calling the preset filtering rules;

[0020] Filtering the news data based on the filtering rule and the news classification data to obtain second designated news data that meets the filtering condition;

[0021] Eliminate the second designated news data from the news data to obtain corresponding third news data;

[0022] The first news data is generated based on the third news data.

[0023] Furthermore, the step of performing summary generation processing on the second news data based on the summary function to obtain corresponding initial news summary data specifically includes:

[0024] Based on the summary function, performing sentence importance evaluation on the second news data to obtain corresponding sentence importance information;

[0025] Perform keyword extraction on the second news data to obtain corresponding keyword data;

[0026] Performing summary generation processing based on the sentence importance information and the keyword data to obtain corresponding summary data;

[0027] The original news summary data is generated based on the summary data.

[0028] Furthermore, the step of optimizing the initial news summary data to obtain corresponding target news summary data specifically includes:

[0029] Performing redundant word removal processing on the initial news summary data to obtain corresponding first data;

[0030] Performing sentence structure adjustment processing on the first data to obtain corresponding second data;

[0031] Performing length control processing on the second data to obtain corresponding third data;

[0032] The target news summary data is generated based on the third data.

[0033] Furthermore, after the step of optimizing the initial news summary data to obtain the corresponding target news summary data, the method further includes:

[0034] Get the preset specified report template;

[0035] Based on the specified report template, acquiring corresponding news-related data from the news data and the target news summary data;

[0036] Performing data filling processing on the designated report template based on the news-related data to obtain a corresponding initial report;

[0037] Performing visual adjustment processing on the initial report to obtain a corresponding target report;

[0038] The target report is exported.

[0039] Furthermore, after the step of performing visual adjustment processing on the initial report to obtain the corresponding target report, the method further includes:

[0040] Get the preset verification strategy;

[0041] Performing verification processing on the target report based on the verification strategy to obtain a corresponding verification result;

[0042] Performing corresponding adjustment processing on the target report based on the verification result to obtain an adjusted target report;

[0043] The adjusted target report is stored.

[0044] In order to solve the above technical problems, the embodiment of the present application further provides a data generation device based on artificial intelligence, which adopts the following technical solution:

[0045] A first acquisition module, used for acquiring news data based on a preset crawling module;

[0046] A processing module, used for inputting the news data into a preset target large language model, and calling a preset function corresponding to the target large language model; wherein the function includes a classification function, a topic screening function, a deduplication function and a summary function;

[0047] A classification module, used for performing news classification processing on the news data based on the classification function to obtain corresponding news classification data;

[0048] A screening module, used for screening the news data based on the subject screening function and the news classification data to obtain corresponding first news data;

[0049] a deduplication module, configured to perform deduplication screening processing on the first news data based on the deduplication function to obtain corresponding second news data;

[0050] A generating module, configured to perform summary generation processing on the second news data based on the summary function to obtain corresponding initial news summary data;

[0051] The optimization module is used to optimize the initial news summary data to obtain corresponding target news summary data.

[0052] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0053] Get news data based on the preset crawling module;

[0054] Input the news data into a preset target large language model, and call a preset function corresponding to the target large language model; wherein the function includes a classification function, a topic screening function, a deduplication function, and a summary function;

[0055] Performing news classification processing on the news data based on the classification function to obtain corresponding news classification data;

[0056] Filtering the news data based on the topic filtering function and the news classification data to obtain corresponding first news data;

[0057] Performing deduplication screening processing on the first news data based on the deduplication function to obtain corresponding second news data;

[0058] Performing summary generation processing on the second news data based on the summary function to obtain corresponding initial news summary data;

[0059] The initial news summary data is optimized to obtain corresponding target news summary data.

[0060] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0061] Get news data based on the preset crawling module;

[0062] Input the news data into a preset target large language model, and call a preset function corresponding to the target large language model; wherein the function includes a classification function, a topic screening function, a deduplication function, and a summary function;

[0063] Performing news classification processing on the news data based on the classification function to obtain corresponding news classification data;

[0064] Filtering the news data based on the topic filtering function and the news classification data to obtain corresponding first news data;

[0065] Performing deduplication screening processing on the first news data based on the deduplication function to obtain corresponding second news data;

[0066] Performing summary generation processing on the second news data based on the summary function to obtain corresponding initial news summary data;

[0067] The initial news summary data is optimized to obtain corresponding target news summary data.

[0068] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0069] The present application obtains news data based on a preset crawling module; then the news data is input into a preset target large language model, and a preset function corresponding to the target large language model is called; wherein the function function includes a classification function, a topic screening function, a deduplication function and a summary function; then the news data is classified based on the classification function to obtain corresponding news classification data; and the news data is screened based on the topic screening function and the news classification data to obtain corresponding first news data; subsequently, the first news data is deduplicated and screened based on the deduplication function to obtain corresponding second news data; further, the second news data is summarized based on the summary function to obtain corresponding initial news summary data; finally, the initial news summary data is optimized to obtain corresponding target news summary data. Compared with the existing manual processing method of manually collecting and organizing news summaries, the present application realizes the automatic and intelligent summary generation and processing of news data by introducing a crawling module and the intelligent analysis capability of a target large language model including classification functions, topic screening functions, deduplication functions and summary functions, which greatly reduces the manual burden, improves the processing efficiency of news summary generation, and ensures the accuracy of the generated target news summary data. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0071] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0072] Figure 2 A flowchart of an embodiment of the artificial intelligence-based data generation method according to the present application;

[0073] Figure 3 is a structural schematic diagram of an embodiment of an artificial intelligence-based data generation device according to the present application;

[0074] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0075] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0076] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0077] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0078] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0079] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0080] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to the laptop computer 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0081] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0082] It should be noted that the artificial intelligence-based data generation method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the artificial intelligence-based data generation device is generally set in the server / terminal device.

[0083] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0084] Continue to refer Figure 2 , showing a flow chart of an embodiment of the data generation method based on artificial intelligence according to the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. The data generation method based on artificial intelligence provided in the embodiment of the present application can be applied to any scenario where news summary data generation is required, and the data generation method based on artificial intelligence can be applied to products in these scenarios, for example, news summary data generation in the financial insurance field. The data generation method based on artificial intelligence includes the following steps:

[0085] Step S201, obtaining news data based on a preset crawling module.

[0086] In this embodiment, the electronic device (e.g. Figure 1The server / terminal device shown in the figure) can obtain news data through a wired connection or a wireless connection. It should be noted that the above-mentioned wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future. The news data obtained from the crawling module generally includes field data such as news title, content, release time, source, etc. And store these data in an appropriate data structure, such as a list, dictionary or database table.

[0087] The above crawling module is specifically a news data crawling module based on the Selenium framework. The technical implementation of the crawling module includes: using the Selenium framework to simulate the browser operation behavior of real users, supporting multiple interactive actions such as logging in, clicking, filling out forms, etc., to ensure that the required data can be accessed. Especially for those pages that rely on JavaScript to dynamically load content, Selenium can wait for the elements to be loaded before crawling.

[0088] The Selenium framework can effectively deal with this complex situation by simulating the browser behavior of real users. It can automatically handle interactive actions such as login verification, clicking links, scrolling pages, etc., ensuring that the required information can be successfully obtained even in a highly dynamic web environment. The specific implementation steps are as follows:

[0089] (1) Initialize the environment: Install Selenium and the corresponding WebDriver (such as ChromeDriver), and configure environment variables.

[0090] (2) Set up a target website list: Collect official websites, authoritative media, professional forums, etc. of the insurance industry to form a crawling list. For example: websites related to the China Insurance Association.

[0091] (3) Simulated login: For websites that require authentication, write a Selenium script to simulate the user login process.

[0092] (4) Crawling data: Start the crawler task at a predetermined time interval (such as early morning every half month), traverse the list of target websites, and capture news titles, release times, content summaries, and full-text links.

[0093] (5) Data storage: The crawled data is stored in a local database or cloud storage for subsequent processing.

[0094] Step S202, input the news data into a preset target large language model, and call a preset functional function corresponding to the target large language model; wherein the functional function includes a classification function, a topic screening function, a deduplication function and a summary function.

[0095] In this embodiment, the target large language model can adopt any large language model. By using the function_calling interface of the target large language model, four functional functions can be defined, namely news classification, topic selection, deduplication screening and news summary, which correspond to the classification function, topic screening function, deduplication function and summary function respectively.

[0096] Specifically, the logical implementation of the classification function includes: the function first pre-processes the news text to remove irrelevant words, and then inputs it into the trained classification model. The model will determine the classification result of the news based on the keywords and context information in the news content. The logical implementation of the topic screening function includes: this function receives the news text and filtering conditions as input, and checks the news one by one according to the set rules, such as whether the release time is within the specified range, whether it contains specific words, etc., so as to select the news that meets the conditions. The logical implementation of the deduplication function includes: the function first calculates the similarity between each two news in the news list. If the similarity exceeds a certain threshold (for example, 0.7), the two news are considered to be duplicates. On this basis, the news with the most detailed content is selected. The logical implementation of the summary function: this function uses NLP technology to analyze the structure of the news text, extract key words and phrases, and then generate a short and comprehensive news summary based on this information.

[0097] Among them, the above-mentioned target large language model can preferably adopt the Qwen large model. The Qwen large model has powerful natural language processing capabilities and can understand and generate high-quality text content. Through the function_calling interface, specific functional functions can be defined to implement functions such as news classification, topic selection, deduplication screening and news summarization. These functions are crucial for the automated processing of large amounts of text information, especially when faced with highly professional and terminology-intensive insurance news, where traditional rule-based methods are difficult to achieve ideal processing results. The Qwen model can learn and understand the semantics behind the text through deep learning technology, thereby providing more accurate and efficient text processing services.

[0098] Step S203: performing news classification processing on the news data based on the classification function to obtain corresponding news classification data.

[0099] In this embodiment, the specific implementation process of performing news classification processing on the news data based on the classification function to obtain corresponding news classification data will be further described in detail in subsequent specific embodiments of the present application, and will not be elaborated on here.

[0100] Step S204: filtering the news data based on the topic filtering function and the news classification data to obtain corresponding first news data.

[0101] In this embodiment, the specific implementation process of filtering the news data based on the topic filtering function and the news classification data to obtain the corresponding first news data will be further described in detail in subsequent specific embodiments of the present application and will not be elaborated on here.

[0102] Step S205: performing deduplication screening processing on the first news data based on the deduplication function to obtain corresponding second news data.

[0103] In this embodiment, the above-mentioned deduplication function can specifically use the remove_duplicates(news_list) function, where news_list is the news list data. The above-mentioned deduplication screening process can specifically include: based on the above-mentioned deduplication function, using the target large language model to convert the news text data into a vector representation. Then calculate the cosine similarity between the news text data. If the similarity exceeds a certain threshold (for example, 0.7), it is considered that the two news text data are duplicated. On this basis, the news with the most detailed content is selected to be retained.

[0104] Step S206: performing summary generation processing on the second news data based on the summary function to obtain corresponding initial news summary data.

[0105] In this embodiment, the specific implementation process of performing summary generation processing on the second news data based on the summary function to obtain the corresponding initial news summary data will be further described in detail in subsequent specific embodiments of the present application and will not be elaborated on here.

[0106] Step S207: Optimize the initial news summary data to obtain corresponding target news summary data.

[0107] In this embodiment, the specific implementation process of optimizing the initial news summary data to obtain the corresponding target news summary data will be further described in detail in subsequent specific embodiments of the present application, and will not be elaborated on here.

[0108] The present application obtains news data based on a preset crawling module; then the news data is input into a preset target large language model, and a preset function corresponding to the target large language model is called; wherein the function function includes a classification function, a topic screening function, a deduplication function and a summary function; then the news data is classified based on the classification function to obtain corresponding news classification data; and the news data is screened based on the topic screening function and the news classification data to obtain corresponding first news data; subsequently, the first news data is deduplicated and screened based on the deduplication function to obtain corresponding second news data; further, the second news data is summarized based on the summary function to obtain corresponding initial news summary data; finally, the initial news summary data is optimized to obtain corresponding target news summary data. Compared with the existing manual processing method of manually collecting and organizing news summaries, the present application realizes the automatic and intelligent summary generation and processing of news data by introducing a crawling module and the intelligent analysis capability of a target large language model including classification functions, topic screening functions, deduplication functions and summary functions, which greatly reduces the manual burden, improves the processing efficiency of news summary generation, and ensures the accuracy of the generated target news summary data.

[0109] In some optional implementations, step S203 includes the following steps:

[0110] Based on the classification function, the news data is preprocessed to obtain corresponding first designated news data.

[0111] In this embodiment, the classification function is specifically a classify_news(text) function, wherein text is news text data. The preprocessing may include removing irrelevant words, processing special characters, and the like.

[0112] Call a pre-built classification model.

[0113] In this embodiment, the above classification model is a machine learning model constructed by learning the one-to-one correspondence between news category keywords and news classification data. Among them, the category definition corresponding to the news classification includes three categories-"regulatory news", "technology regulatory penalties", and "digital financial news". The purpose of defining the three categories of "regulatory news", "technology regulatory penalties" and "digital financial news" is to carry out refined management of news in the financial industry, so that relevant stakeholders can quickly find the information they are concerned about according to the category. In addition, the keywords of each category may include: regulatory news involves supervision, approval, budget, etc.; technology regulatory penalties include penalties, fines, violations, risk management, information security, data protection, consumer rights, etc.; digital financial news includes digital finance, financial technology, Internet finance, blockchain, artificial intelligence, big data, cloud computing, mobile payment, online banking, virtual currency, encryption technology, digital currency, financial innovation, etc.

[0114] The designated news data is classified based on the classification model to obtain corresponding classification results.

[0115] In this embodiment, the designated news data may be input into a classification model, and the classification model will classify the designated news data according to keywords and contexts in the designated news data, thereby outputting corresponding classification results.

[0116] News classification data corresponding to the news data is generated based on the classification result.

[0117] In this embodiment, the classification result can be directly used as news classification data corresponding to the news data.

[0118] The present application pre-processes the news data based on the classification function to obtain the corresponding first designated news data; then calls the pre-built classification model; then classifies the designated news data based on the classification model to obtain the corresponding classification result; and subsequently generates news classification data corresponding to the news data based on the classification result. The present application pre-processes the news data based on the use of the classification function in the target large language model to obtain the corresponding first designated news data, and then classifies the designated news data based on the called classification model, so that the news classification processing of the news data can be completed quickly and accurately, the processing efficiency of the news classification is improved, and the data accuracy of the obtained news classification data is guaranteed.

[0119] In some optional implementations of this embodiment, step S204 includes the following steps:

[0120] Based on the subject screening function, the preset filtering rules are called.

[0121] In this embodiment, the topic screening function may be specifically a filter_news_by_criteria(news,criteria) function, where news is news list data and criteria is a filtering condition. Filtering rules may be pre-set according to actual processing requirements, for example, excluding specific types of advertising news.

[0122] The news data is screened based on the filtering rule and the news classification data to obtain second designated news data that meets the filtering condition.

[0123] In this embodiment, news data may be detected one by one according to the filtering rule, and the news type of the news data may be determined according to the news classification data, and then the second designated news data that meets the filtering condition corresponding to the filtering rule may be screened out from all news types.

[0124] The second designated news data is removed from the news data to obtain corresponding third news data.

[0125] In this embodiment, the corresponding third news data can be obtained by removing the second designated news data from the news data.

[0126] The first news data is generated based on the third news data.

[0127] In this embodiment, the third news data may be directly used as the first news data.

[0128] The present application uses the topic filtering function in the target large language model, and filters the news data based on the news classification data and the called filtering rules to obtain the second designated news data that meets the filtering conditions, and then removes the second designated news data from the news data to obtain the corresponding third news data; and subsequently generates the first news data based on the third news data. The present application uses the topic filtering function in the target large language model, and filters the news data based on the news classification data and the called filtering rules to obtain the second designated news data that meets the filtering conditions, and then removes the second designated news data from the news data, so that the news data can be quickly and accurately filtered, thereby improving the processing efficiency and accuracy of news filtering.

[0129] In some optional implementations, step S206 includes the following steps:

[0130] Based on the summary function, sentence importance evaluation is performed on the second news data to obtain corresponding sentence importance information.

[0131] In this embodiment, the above-mentioned summary function is specifically the summarize_news(text) function, where text is news text data. The corresponding sentence data can be obtained by performing word segmentation, part-of-speech tagging and sentence segmentation processing on the second news data, and then the importance of each sentence is evaluated using TF-IDF, TextRank or other algorithms to obtain the corresponding sentence importance information. The importance can be based on factors such as keywords in the sentence, sentence position, sentence length, etc.

[0132] Keyword extraction is performed on the second news data to obtain corresponding keyword data.

[0133] In this embodiment, keywords in the second news data can be extracted by using RAKE, TF-IDF or other methods. These keywords will help determine the key information in the summary.

[0134] A summary generation process is performed based on the sentence importance information and the keyword data to obtain corresponding summary data.

[0135] In this embodiment, the most important sentences or sentence fragments can be selected based on the obtained sentence importance information and keyword data. The selected sentences or sentence fragments are then reorganized to form coherent and concise summary data. The summary data can be further edited to ensure its accuracy and fluency.

[0136] The original news summary data is generated based on the summary data.

[0137] In this embodiment, the summary data may be directly used as the original news summary data.

[0138] The present application performs sentence importance evaluation on the second news data based on the summary function to obtain corresponding sentence importance information; then performs keyword extraction on the second news data to obtain corresponding keyword data; then performs summary generation processing based on the sentence importance information and the keyword data to obtain corresponding summary data; and subsequently generates the initial news summary data based on the summary data. The present application performs sentence importance evaluation on the second news data based on the use of a summary function to obtain corresponding sentence importance information, and performs keyword extraction on the second news data to obtain keyword data, and then performs summary generation processing based on the sentence importance information and the keyword data, so that the corresponding initial news summary data can be automatically and accurately generated, thereby improving the processing efficiency of summary generation and ensuring the accuracy of the obtained initial news summary data.

[0139] In some optional implementations, step S207 includes the following steps:

[0140] Perform redundancy word removal processing on the initial news summary data to obtain corresponding first data.

[0141] In this embodiment, the above redundancy word removal processing includes synonym replacement, duplicate word deletion, and useless word removal, etc. Among them, synonym replacement includes: checking for synonyms in the summary and attempting to replace them with more common or concise words. Duplicate word deletion includes: deleting duplicate words in the summary. Useless word removal includes: removing words that occur frequently but have little significance to the meaning of the summary, such as "de", "le", etc.

[0142] Perform sentence structure adjustment processing on the first data to obtain corresponding second data.

[0143] In this embodiment, the sentence structure adjustment processing includes sentence merging, sentence splitting, and sentence restructuring, etc. Among them, sentence merging includes: if there are multiple short sentences in the summary and their logic is tight, attempt to merge them into a long sentence. Sentence splitting includes: if the sentence in the summary is too long and contains multiple information points, split it into multiple short sentences to improve readability. Sentence restructuring includes: rearranging the sentences in the summary according to the logical order and importance of the information.

[0144] Perform length control processing on the second data to obtain corresponding third data.

[0145] In this embodiment, the length of the summary can be adjusted according to actual needs to ensure it is both concise and comprehensive. Among them, if the summary is too long, secondary information can be further deleted or the sentence structure can be simplified.

[0146] Generate the target news summary data based on the third data.

[0147] In this embodiment, the third data can be directly used as the target news summary data. Among them, a grammar checking tool or library can also be used to check for grammar errors in the summary. And a spelling checking tool can be used to correct spelling mistakes in the summary. In addition, transition words or phrases (such as "moreover", "however", etc.) can also be used to connect sentences to improve the fluency of the summary.

[0148] The present application performs redundant word removal processing on the initial news summary data to obtain corresponding first data; then performs sentence structure adjustment processing on the first data to obtain corresponding second data; then performs length control processing on the second data to obtain corresponding third data; and subsequently generates the target news summary data based on the third data. The present application performs redundant word removal processing, sentence structure adjustment processing, and length control processing on the initial news summary data, thereby effectively improving the quality and readability of the generated target news summary data.

[0149] In some optional implementations of this embodiment, after step S207, the electronic device may further perform the following steps:

[0150] Get the preset specified report template.

[0151] In this embodiment, the above-mentioned designated report template may specifically be a monthly report template that is pre-built according to actual monthly report building requirements and contains information types such as date, category, title, summary, and original text link.

[0152] Based on the specified report template, corresponding news-related data is acquired from the news data and the target news summary data.

[0153] In this embodiment, according to the information type specified in the above-mentioned specified report template, all news-related data matching the information type can be acquired from the news data and the target news summary data.

[0154] The designated report template is filled with data based on the news-related data to obtain a corresponding initial report.

[0155] In this embodiment, the data filling process of the specified report template may include: first, set up a directory to contain Markdown files, define a sequential list of file names, and then initialize a dictionary to save the Markdown file contents in sequence. Then traverse all the files in the directory and check whether the file name contains predefined keywords. If so, read the file content and add it to the dictionary entry of the corresponding keyword. Finally, merge the contents of the Markdown files in the defined order, and write the merged contents to a new Markdown file named "AI Industry News.md" containing the current date, and add a title to the beginning of the file. Finally, output a message indicating that the Markdown files have been merged in sequence.

[0156] Visual adjustment processing is performed on the initial report to obtain a corresponding target report.

[0157] In this embodiment, the generated initial report can be output according to the specified font size, size, and spacing, so as to obtain a target report that meets the visual display conditions.

[0158] The target report is exported.

[0159] In this embodiment, the generated target report can be exported to a specified format to facilitate reading and sharing. The specified format can be Word format.

[0160] This application obtains a preset designated report template; then based on the designated report template, obtains corresponding news-related data from the news data and the target news summary data; then based on the news-related data, performs data filling processing on the designated report template to obtain a corresponding initial report; subsequently, performs visual adjustment processing on the initial report to obtain a corresponding target report; and finally, exports the target report. This application obtains corresponding news-related data from news data and target news summary data based on the use of a designated report template; then based on the news-related data, performs data filling processing on the designated report template to obtain an initial report; and then intelligently performs visual adjustment processing on the initial report to obtain a target report, thereby automatically and quickly generating the required target report, effectively improving the generation efficiency and generation intelligence of the target report.

[0161] In some optional implementations of this embodiment, after the step of performing visual adjustment processing on the initial report to obtain a corresponding target report, the electronic device may further perform the following steps:

[0162] Get the preset authentication policy.

[0163] In this embodiment, the above verification strategy includes content verification, format verification and compliance verification. Among them, content verification includes: checking whether the data in the report is accurate, including key indicators such as sales, number of users, conversion rate, etc., to ensure that they are consistent with the original data source. Confirm whether the report covers all the content that needs to be reported, such as project progress, team dynamics, market trends, etc. And check whether the information in the report is logically clear, whether the various parts support each other, and there is no contradiction.

[0164] Format verification includes: Checking whether the report layout is neat, whether the font, size, color, etc. are uniform, and whether it meets the company's visual specifications. Verifying whether the charts and pictures are clear and accurate and whether they help to understand the report content. Confirming whether the header and footer contain the correct company information, date and page number.

[0165] Compliance verification includes: Checking whether sensitive information is leaked in the report, such as customer privacy, internal data, etc. Ensuring that the charts, pictures and other materials used have legal copyright or usage rights.

[0166] The target report is verified based on the verification strategy to obtain a corresponding verification result.

[0167] In this embodiment, the target report may be verified according to the verification contents including content verification, format verification and compliance verification in the above verification strategy, so as to obtain a corresponding verification result.

[0168] Based on the verification result, corresponding adjustment processing is performed on the target report to obtain an adjusted target report.

[0169] In this embodiment, the above adjustment process may include: correcting data errors, information omissions or logical inconsistencies found in the report. Adjusting the report layout to ensure that the monthly report is neat and beautiful. Updating charts and pictures to ensure that they accurately reflect the content of the report.

[0170] The adjusted target report is stored.

[0171] In this embodiment, there is no specific limitation on the storage method of the adjusted target report, for example, local database storage, blockchain storage, cloud server storage, etc. can be used.

[0172] The present application obtains a preset verification strategy; then verifies the target report based on the verification strategy to obtain a corresponding verification result; then performs corresponding adjustment processing on the target report based on the verification result to obtain an adjusted target report; and subsequently stores the adjusted target report. After the initial report is visually adjusted to obtain the target report, the present application will also intelligently verify the target report based on the use of the verification strategy to obtain a verification result, and then perform corresponding adjustment processing on the target report based on the verification result, effectively ensuring that the content of the target report finally generated is accurate and the format meets the requirements. In addition, the adjusted target report will also be intelligently stored, thereby improving the data security of the adjusted target report.

[0173] In some optional implementations, the user information obtained is subject to the user's consent and complies with relevant laws and policies.

[0174] In addition, the non-Company software tools or components that appear in the embodiments of the present application are merely examples and do not represent actual use.

[0175] In addition, compared with traditional manual processing methods, this application has achieved significant improvements in the efficiency and accuracy of news summary generation. By introducing Selenium crawler technology and the intelligent analysis capabilities of the target large language model, the automatic and intelligent processing of news data is realized, which greatly reduces the manual burden and ensures the timeliness and accuracy of the generated news summaries. In addition, by removing duplicate news, information redundancy is avoided and the user experience is improved. It can also bring the following business value:

[0176] 1. Improve work efficiency: Automated processing reduces the workload of manual screening and sorting of news, allowing employees to focus on higher-value tasks such as strategic planning and customer service.

[0177] 2. Enhance decision-making support: timely obtain and accurately interpret industry trends, provide strong data support for management, and help formulate more scientific and reasonable business strategies.

[0178] 3. Reduce compliance risks: Improve employees’ awareness of news facts in the insurance field, thereby reducing the risk of penalties for violations.

[0179] 4. Optimize user experience: By removing duplicate information, we ensure that the news content provided is more refined and valuable, thereby improving the user's reading experience and thereby enhancing customer satisfaction and loyalty.

[0180] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0181] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned target news summary data, the above-mentioned target news summary data can also be stored in a node of a blockchain.

[0182] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0183] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0184] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0185] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0186] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0187] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a data generation device based on artificial intelligence, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0188] like Figure 3As shown, the artificial intelligence-based data generation device 300 described in this embodiment includes: a first acquisition module 301, a processing module 302, a classification module 303, a screening module 304, a deduplication module 305, a generation module 306 and an optimization module 307. Among them:

[0189] A first acquisition module 301 is used to acquire news data based on a preset crawling module;

[0190] The processing module 302 is used to input the news data into a preset target large language model and call a preset function corresponding to the target large language model; wherein the function includes a classification function, a topic screening function, a deduplication function and a summary function;

[0191] A classification module 303, configured to perform news classification processing on the news data based on the classification function to obtain corresponding news classification data;

[0192] A screening module 304, configured to screen the news data based on the topic screening function and the news classification data to obtain corresponding first news data;

[0193] A deduplication module 305, configured to perform deduplication screening processing on the first news data based on the deduplication function to obtain corresponding second news data;

[0194] A generating module 306, configured to perform summary generation processing on the second news data based on the summary function to obtain corresponding initial news summary data;

[0195] The optimization module 307 is used to optimize the initial news summary data to obtain corresponding target news summary data.

[0196] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the artificial intelligence-based data generation method in the aforementioned implementation mode, and will not be repeated here.

[0197] In some optional implementations of this embodiment, the classification module 303 includes:

[0198] A preprocessing submodule, used for preprocessing the news data based on the classification function to obtain corresponding first designated news data;

[0199] The first calling submodule is used to call the pre-built classification model;

[0200] A classification submodule, used for classifying the specified news data based on the classification model to obtain corresponding classification results;

[0201] The first generating submodule is used to generate news classification data corresponding to the news data based on the classification result.

[0202] In some optional implementations of this embodiment, the screening module 304 includes:

[0203] A second calling submodule is used to call a preset filtering rule based on the subject screening function;

[0204] A screening submodule, configured to screen the news data based on the filtering rule and the news classification data to obtain second designated news data that meets the filtering condition;

[0205] A removal submodule, used for removing the second designated news data from the news data to obtain corresponding third news data;

[0206] The second generating submodule is used to generate the first news data based on the third news data.

[0207] In some optional implementations of this embodiment, the generating module 306 includes:

[0208] An evaluation submodule, configured to perform sentence importance evaluation on the second news data based on the summary function to obtain corresponding sentence importance information;

[0209] An extraction submodule, used for extracting keywords from the second news data to obtain corresponding keyword data;

[0210] A third generation submodule is used to perform summary generation processing based on the sentence importance information and the keyword data to obtain corresponding summary data;

[0211] The fourth generating submodule is used to generate the original news summary data based on the summary data.

[0212] In some optional implementations of this embodiment, the optimization module 307 includes:

[0213] A first processing submodule, configured to perform redundant word removal processing on the initial news summary data to obtain corresponding first data;

[0214] A second processing submodule, configured to perform sentence structure adjustment processing on the first data to obtain corresponding second data;

[0215] A third processing submodule, configured to perform length control processing on the second data to obtain corresponding third data;

[0216] The fifth generating submodule is used to generate the target news summary data based on the third data.

[0217] In some optional implementations of this embodiment, the artificial intelligence-based data generation device further includes:

[0218] The second acquisition module is used to acquire a preset specified report template;

[0219] A third acquisition module, configured to acquire corresponding news-related data from the news data and the target news summary data based on the specified report template;

[0220] A filling module, used for performing data filling processing on the specified report template based on the news-related data to obtain a corresponding initial report;

[0221] A first adjustment module, used for performing visual adjustment processing on the initial report to obtain a corresponding target report;

[0222] The export module is used to export the target report.

[0223] In some optional implementations of this embodiment, the artificial intelligence-based data generation device further includes:

[0224] The fourth acquisition module is used to obtain a preset verification strategy;

[0225] A verification module, used to perform verification processing on the target report based on the verification strategy to obtain a corresponding verification result;

[0226] A second adjustment module, configured to perform corresponding adjustment processing on the target report based on the verification result to obtain an adjusted target report;

[0227] A storage module is used to store the adjusted target report.

[0228] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0229] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0230] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0231] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of a data generation method based on artificial intelligence, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0232] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the artificial intelligence-based data generation method.

[0233] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0234] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the artificial intelligence-based data generation method as described above.

[0235] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0236] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A data generation method based on artificial intelligence, characterized in that: The steps include: Get news data based on the preset crawling module; Input the news data into a preset target large language model, and call a preset function corresponding to the target large language model; wherein the function includes a classification function, a topic screening function, a deduplication function, and a summary function; Performing news classification processing on the news data based on the classification function to obtain corresponding news classification data; Filtering the news data based on the topic filtering function and the news classification data to obtain corresponding first news data; Performing deduplication screening processing on the first news data based on the deduplication function to obtain corresponding second news data; Performing summary generation processing on the second news data based on the summary function to obtain corresponding initial news summary data; The initial news summary data is optimized to obtain corresponding target news summary data.

2. The artificial intelligence-based data generation method according to claim 1, characterized in that: The step of performing news classification processing on the news data based on the classification function to obtain corresponding news classification data specifically includes: Based on the classification function, preprocessing the news data to obtain corresponding first designated news data; Calling pre-built classification models; Classify the specified news data based on the classification model to obtain corresponding classification results; News classification data corresponding to the news data is generated based on the classification result.

3. The artificial intelligence-based data generation method according to claim 1, characterized in that: The step of filtering the news data based on the topic filtering function and the news classification data to obtain the corresponding first news data specifically includes: Based on the subject screening function, calling the preset filtering rules; Filtering the news data based on the filtering rule and the news classification data to obtain second designated news data that meets the filtering condition; Eliminate the second designated news data from the news data to obtain corresponding third news data; The first news data is generated based on the third news data.

4. The artificial intelligence-based data generation method according to claim 1, characterized in that: The step of performing summary generation processing on the second news data based on the summary function to obtain corresponding initial news summary data specifically includes: Based on the summary function, performing sentence importance evaluation on the second news data to obtain corresponding sentence importance information; Perform keyword extraction on the second news data to obtain corresponding keyword data; Performing summary generation processing based on the sentence importance information and the keyword data to obtain corresponding summary data; The original news summary data is generated based on the summary data.

5. The artificial intelligence-based data generation method according to claim 1, characterized in that: The step of optimizing the initial news summary data to obtain corresponding target news summary data specifically includes: Performing redundant word removal processing on the initial news summary data to obtain corresponding first data; Performing sentence structure adjustment processing on the first data to obtain corresponding second data; Performing length control processing on the second data to obtain corresponding third data; The target news summary data is generated based on the third data.

6. The artificial intelligence-based data generation method according to claim 1, characterized in that: After the step of optimizing the initial news summary data to obtain the corresponding target news summary data, the method further includes: Get the preset specified report template; Based on the specified report template, acquiring corresponding news-related data from the news data and the target news summary data; Performing data filling processing on the designated report template based on the news-related data to obtain a corresponding initial report; Performing visual adjustment processing on the initial report to obtain a corresponding target report; The target report is exported.

7. The artificial intelligence-based data generation method according to claim 6, characterized in that: After the step of performing visual adjustment processing on the initial report to obtain the corresponding target report, the method further includes: Get the preset verification strategy; Performing verification processing on the target report based on the verification strategy to obtain a corresponding verification result; Performing corresponding adjustment processing on the target report based on the verification result to obtain an adjusted target report; The adjusted target report is stored.

8. A data generation device based on artificial intelligence, characterized in that: include: A first acquisition module, used for acquiring news data based on a preset crawling module; A processing module, used for inputting the news data into a preset target large language model, and calling a preset function corresponding to the target large language model; wherein the function includes a classification function, a topic screening function, a deduplication function and a summary function; A classification module, used for performing news classification processing on the news data based on the classification function to obtain corresponding news classification data; A screening module, used for screening the news data based on the subject screening function and the news classification data to obtain corresponding first news data; a deduplication module, configured to perform deduplication screening processing on the first news data based on the deduplication function to obtain corresponding second news data; A generating module, configured to perform summary generation processing on the second news data based on the summary function to obtain corresponding initial news summary data; The optimization module is used to optimize the initial news summary data to obtain corresponding target news summary data.

9. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the artificial intelligence-based data generation method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the artificial intelligence-based data generation method according to any one of claims 1 to 7.

Images