Dynamic news information data filtering method and tool

Abstract

The invention provides a dynamic news information data filtering method and tool. The method comprises the steps of obtaining a dynamic news information data set obtained in a web crawler mode; preprocessing the dynamic news information data set to obtain a preliminarily filtered dynamic news information data set; wherein the preprocessing comprises empty judgment check; performing deep noise cleaning on the preliminarily filtered dynamic news information data set by using a filtering prompt word indication large model so as to identify and remove impurity noise information contained in the dynamic news information data set, and removing character contents in a preset range by using a regular expression; and comparing the news information data acquired through different information sources contained in the dynamic news information data set by adopting a comparison algorithm model so as to remove repeated information and obtain the dynamic news information data set after deep filtering. According to the invention, highly intelligent and automatic filtering processing of dynamic news information data can be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent data processing, and in particular to a method and tool for filtering dynamic news information data. Background Art

[0002] With the rapid development of information technology, the acquisition, processing and use of information has become a distinctive feature of modern society. Especially in the field of information security, the acquisition and analysis of dynamic news information plays an irreplaceable and important role in maintaining information security.

[0003] Traditionally, there are two main strategies for collecting dynamic news information datasets, namely manual collection and automatic crawling using crawler tools.

[0004] 1. Manual collection method: Manual collection can accurately filter information according to specific needs, reduce errors and interference from irrelevant data. In addition, in the face of complex and changing multiple sources of news information, manual collection can respond flexibly, adjust collection strategies, deeply understand the meaning behind each piece of information, and accurately collect various data. The data sets collected by this method are often of higher quality. However, this strategy relies on a large amount of human resources to collect a large amount of dynamic news information data from multiple websites, so the cost is relatively high.

[0005] 2. Automatic crawling method using crawler tools: Crawler tools can quickly and massively crawl web page data to obtain large-scale data sets, and once developed, the operating cost is relatively low and does not require a lot of manpower investment. However, for different websites, crawler strategies need to be adjusted, and some websites may also prohibit crawler access. Therefore, crawlers are not stable, and when problems arise, professional technicians are required to develop and maintain them, and the technical requirements are also high. Automatically crawled data often contains a lot of useless or duplicate information, such as advertisements on web pages, and are also prone to errors in missing key content. Due to the uneven quality of data, the subsequent analysis and utilization of dynamic news information often becomes extremely complex and arduous.

[0006] In order to speed up the construction of dynamic news information datasets, improve the efficiency of news information work and the quality of the collected dynamic news information data, we urgently need to develop an efficient and intelligent filtering tool that can quickly remove incomplete and duplicate dynamic news information obtained by traditional crawler tools and reduce the noise information contained in the news information. Summary of the invention

[0007] In view of this, an embodiment of the present invention provides a method and tool for filtering dynamic news information data to eliminate or improve one or more defects existing in the prior art.

[0008] One aspect of the present invention provides a method for filtering dynamic news information data, the method comprising the following steps: obtaining a dynamic news information data set obtained by means of a web crawler; preprocessing the dynamic news information data set to obtain a dynamic news information data set after preliminary filtering; wherein the preprocessing includes a blank check; using a filtering prompt word indication large model to perform deep noise cleaning on the dynamic news information data set after preliminary filtering to identify and remove impurity noise information contained in the dynamic news information data set, and using a regular expression to remove character content within a preset range; using a comparison algorithm model to compare news information data obtained from different information sources contained in the dynamic news information data set to remove duplicate information and obtain a dynamic news information data set after deep filtering.

[0009] In some embodiments of the present invention, the empty check includes: performing an empty check on the dynamic news information data set to filter out data with missing key fields; wherein the key fields include multiple ones of document number, source, title and text.

[0010] In some embodiments of the present invention, the impurity noise information includes multiple types of advertisements, irrelevant links, garbled codes, HTML tags and special symbols.

[0011] In some embodiments of the present invention, the use of the filtering prompt word indication big model to perform deep noise cleaning on the dynamic news information data set after preliminary filtering to identify and eliminate the impurity noise information contained in the dynamic news information data set, and use regular expressions to eliminate character content within a preset range, including: extracting the main content item data from each news information data contained in the dynamic news information data set after preliminary filtering; using regular expressions to eliminate the character content within a preset range contained in the main content item data; inputting the main content item data processed by the regular expression into the big model, using the pre-acquired filtering prompt word indication big model to intelligently filter the input content to identify and eliminate the impurity noise information contained in the dynamic news information data set; using regular expressions to correct the format of the output after intelligent filtering by the big model; replacing the original main content item data in each news information data with the output after format correction.

[0012] In some embodiments of the present invention, the character content in the preset range includes multiple types of web page links, email addresses, emoticons, consecutive multiple line breaks, tabs, and space characters.

[0013] In some embodiments of the present invention, the use of a comparison algorithm model to compare news information data obtained through different information sources and contained in a dynamic news information data set includes: extracting title item data of each news information data contained in the dynamic news information data set; removing spaces before and after the title character string of the title item data, determining whether there is duplication between the news information data obtained through different information sources, and eliminating the duplicate news information data.

[0014] Corresponding to the above method, the present invention also provides a dynamic news information data filtering tool, including: a vacancy value data screening module, used to obtain a dynamic news information data set obtained by a web crawler, and pre-process the dynamic news information data set to obtain a dynamic news information data set after preliminary filtering; wherein the pre-processing includes a blank check; a text data noise cleaning module, used to use a filtering prompt word indication large model to perform deep noise cleaning on the dynamic news information data set after preliminary filtering, so as to identify and eliminate the impurity noise information contained in the dynamic news information data set, and use regular expressions to eliminate character content within a preset range; a duplicate news data processing module, used to use a comparison algorithm model to compare the news information data obtained from different information sources contained in the dynamic news information data set, so as to eliminate duplicate information and obtain a dynamic news information data set after deep filtering.

[0015] In some embodiments of the present invention, the dynamic news information dataset obtained by means of a web crawler is in JSON format.

[0016] Corresponding to the above method, the present invention also provides a device, including a processor, a memory, and a computer program / instructions stored in the memory, wherein the processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the device implements the steps of the method described in any one of the above embodiments.

[0017] Corresponding to the above method, the present invention also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the method described in any one of the above embodiments are implemented.

[0018] The dynamic news information data filtering method and tool proposed in the present invention can make use of the intelligent characteristics of the excellent language understanding ability of the large language model to achieve highly intelligent and automated dynamic news information data filtering processing, reduce human intervention, and quickly eliminate incomplete and repeated dynamic news information obtained by traditional crawler tools, which can greatly improve data processing efficiency and quality.

[0019] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0020] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of the present application, and do not constitute a limitation of the present invention. In the drawings:

[0022] Figure 1 The present invention is a flowchart of a method for filtering dynamic news information data in one embodiment of the present invention.

[0023] Figure 2 This is a flow chart of a method for filtering dynamic news information data in another embodiment of the present invention.

[0024] Figure 3 The present invention is a flowchart of text data noise cleaning in a dynamic news information data filtering method in one embodiment of the present invention.

[0025] Figure 4 It is a structural diagram of dynamic news information data filtering in one embodiment of the present invention.

[0026] Figure 5 The figure is a schematic diagram of the structure of the computer device included in the device. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0028] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0029] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0030] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0031] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0032] In order to overcome the problems existing in the prior art, the present invention proposes a dynamic news information filtering method and tool to improve the quality of dynamic news information data obtained by web crawling and other methods. The solution adopts advanced technologies such as regular expressions and large language models, which can quickly eliminate incomplete and repeated dynamic news information obtained by traditional crawler tools, reduce the noise information contained in the news information, and realize fully automated and intelligent filtering and processing of dynamic news information data.

[0033] Figure 1 This is a flow chart of a method for filtering dynamic news information data in one embodiment of the present invention. The method comprises the following steps:

[0034] Step S110: Acquire a dynamic news information dataset obtained by means of a web crawler.

[0035] Among them, the dynamic news information dataset obtained by web crawler is in JSON format.

[0036] Step S120: pre-processing the dynamic news information data set to obtain a dynamic news information data set after preliminary filtering; wherein the pre-processing includes a null check.

[0037] Among them, the empty check can filter out data that lacks key information. Key fields include document number, source, title, and text. The empty check is not limited to judging whether the value is empty, and other meaningless values ​​should also be filtered out.

[0038] Step S130: Use the filtering prompt word to instruct the large model to perform deep noise cleaning on the dynamic news information data set after preliminary filtering, so as to identify and remove the impurity noise information contained in the dynamic news information data set, and use regular expressions to remove the character content within a preset range.

[0039] The impurity noise information includes advertisements, irrelevant links, garbled codes, HTML tags and special symbols. The quality of the news information content can be improved by adopting step S130.

[0040] Step S140: using an efficient comparison algorithm model to compare the news information data obtained from different information sources contained in the dynamic news information data set, identifying and eliminating duplicate news information data, and obtaining a deep-filtered dynamic news information data set.

[0041] The dynamic news information data filtering method and tool proposed in the present invention can make use of the intelligent characteristics of the excellent language understanding ability of the large language model to realize highly intelligent and automated dynamic news information data filtering and processing, reduce human intervention, and quickly eliminate incomplete and repeated dynamic news information obtained by traditional crawler tools, which can greatly improve data processing efficiency and quality.

[0042] In some embodiments of the present invention, the empty check of step S120 includes: performing an empty check on the dynamic news information data set to filter out data with missing key fields; wherein the key fields include multiple ones of document number, source, title and text.

[0043] By adopting this embodiment of the invention, it is possible to identify data that cannot be applied to the subsequent steps of this solution through a blank check, thereby deleting news information data in the dynamic news information data set that does not meet the requirements of this solution.

[0044] In some embodiments of the present invention, in step S130, the filtering prompt word is used to instruct the big model to perform deep noise cleaning on the dynamic news information data set after preliminary filtering, so as to identify and remove the impurity noise information contained in the dynamic news information data set, and the character content within a preset range is removed by using a regular expression, including: (1) extracting the text content item data from each news information data contained in the dynamic news information data set after preliminary filtering; (2) using a regular expression to remove the character content within a preset range contained in the text content item data; (3) inputting the text content item data processed by the regular expression into the big model, and using the filtering prompt word obtained in advance to instruct the big model to perform intelligent filtering on the input content, so as to identify and remove the impurity noise information contained in the dynamic news information data set; (4) using a regular expression to correct the format of the output after the big model intelligent filtering; wherein the purpose of using the regular expression twice is to correct the format problem in the output result that may be caused by deep noise cleaning. (5) replacing the original text content item data in each news information data with the output after format correction.

[0045] The character content in the preset range includes multiple types of web links, email addresses, emoticons, consecutive line breaks, tabs, and space characters.

[0046] By adopting the embodiment of the invention, it is possible to combine the two regular expression filtering keywords and the pre-trained language model to perform deep noise cleaning on the various news information data contained in the dynamic news information data set and filter out meaningless data.

[0047] In some embodiments of the present invention, a comparison algorithm model is used in step S140 to compare news information data obtained from different information sources and included in a dynamic news information data set, including: (1) extracting title item data of each news information data included in the dynamic news information data set; (2) removing spaces before and after the title character string of the title item data, determining whether there is duplication between the news information data obtained from different information sources, and eliminating the duplicate news information data.

[0048] By adopting the embodiment of the invention, the comparison algorithm model can be used to compare, identify and eliminate duplicate news information data.

[0049] In other embodiments of the present invention, the dynamic news information data set obtained by web crawling or the like is in JSON format. JSON is a lightweight data exchange format that is easy for people to read and write, and is also easy for machines to parse and generate. A dynamic news information data item contains many contents such as date, URL, website, text content, title, number, language, author, etc. A large amount of dynamic news information data is often captured by automatic crawler programs. The automatically crawled data text often contains a large amount of useless or duplicate information, such as advertisements in web pages, and is also prone to errors in missing key content. Due to the uneven levels of data quality, the subsequent analysis and utilization of dynamic news information will often become extremely complicated and arduous.

[0050] In some embodiments of the present invention, step S120 further includes the following steps: (1) performing a key field empty check on each item of the data set, and determining whether the document number, source, title, text, etc. are meaningful in turn, and marking the key fields as meaningless if any. (2) removing the data containing the meaningless marks to obtain preliminary filtering result data.

[0051] In some embodiments of the present invention, step S130 further includes the steps of: (1) extracting the text content item data of each item of news information data. (2) using regular expressions, processing the web links, email addresses, emoticons, multiple consecutive line breaks, tabs and spaces that the data may contain in turn, to reduce impurities in the text content. (3) inputting the data into the large model, using appropriate filtering prompt words, using its ability to understand the meaning of the text, performing intelligent filtering, and saving the results. (4) filtering the results filtered by the large model again using regular expressions to reduce formatting and other issues in the output results of the large model. (5) replacing the filtered text content with the text content item in the original news information data.

[0052] In some embodiments of the present invention, step S140 further includes the following steps: (1) extracting the title item data of each news information data. (2) after removing the spaces before and after the title string, determining whether there is a prefix between multiple title strings, and if so, recording one of them as duplicate news information. (3) removing the data determined to be duplicate news information, and obtaining the final filtered dynamic news information data set.

[0053] Figure 2 This is a flow chart of a method for filtering dynamic news information data in another embodiment of the present invention. This is a method for filtering dynamic news information data in embodiment 1 of the present invention, comprising the following steps:

[0054] Step 1-Key field judgment check.

[0055] Input the dynamic news information dataset obtained by web crawling and other methods, perform a null check on the key fields, and filter out the data that lacks key information. Among them, the key fields include document number, source, title, text, etc. The null check is not limited to judging that the value is null, other meaningless values ​​should also be filtered out. Among them, the json format data here is the dynamic news information dataset obtained by input web crawling and other methods. After the key field check, the output data after preliminary filtering is also in json format.

[0056] Specifically, this step 1- can include the following 3 steps:

[0057] (1) Input the dynamic news information dataset obtained by web crawling and other methods.

[0058] After starting the backend server, you can send a filtering request through the Internet interface and attach a dynamic news information dataset in json format. After parsing the http request, you can get the dataset.

[0059] (2) Check the key fields of each item in the data set to see if they are empty. Check the document number, source, title, text, etc. to see if they are meaningful. If there are key fields that are meaningless, mark them as meaningless.

[0060] Write a script to check each item of data in the data set in turn. Check whether the document number of the data is a valid number, and check whether the key items such as source, title, and text are empty, empty string, string 'null', or other meaningless content, and mark them.

[0061] (3) Remove data containing meaningless tags to obtain preliminary filtering result data.

[0062] Create a new array to store the initially filtered news information data in json format. If the data is meaningful, add it to the new array to remove other meaningless data.

[0063] Step 2- Noise cleaning of the main text content.

[0064] Use regular expressions and large model processing algorithms to perform deep noise cleaning on the text content, automatically identify and remove impurity noise information such as advertisements, irrelevant links, garbled codes, etc., and improve the quality of news information content;

[0065] In order to reduce the noise in the text field of news data, which is of most concern to news information collectors, we need to extract the text separately. By using the ability of regular expressions in string replacement and the ability of large language models to understand text, we can extract the original text content.

[0066] Specifically, this step 2 may include the following 5 steps:

[0067] (1) Extract the text content item data of each news information data.

[0068] In the result data of the initial filtering, the main text content data of each item is extracted, and the main text content is filtered separately subsequently.

[0069] (2) Using regular expressions, the data may contain web links, email addresses, emoticons, and multiple consecutive line breaks, tabs, and spaces, to reduce impurities in the text content.

[0070] Using regular expressions, replace web link strings with features such as http with the string "URL", replace email addresses consisting of alphanumeric characters and the '@' symbol with the string "MAIL", delete various emoticon codes, and replace multiple line breaks, tabs, and spaces with a single one.

[0071] (3) Input the data into the large model, use appropriate filter prompts, take advantage of its ability to understand the meaning of the text, perform intelligent filtering, and save the results.

[0072] Further use the internal pre-trained large language model (which can be replaced by the open source large model llama3:8b, etc.) to further filter the impurities such as advertisements, videos, and other article links that may exist in the web pages. Use the prompt "You are an impurity filter for news web crawling information. Below I will give you a section of English web crawling results, and you need to remove the impurity information. Please output the body of the English article, and keep the line breaks in the body."

[0073] (4) The results after filtering the large model are filtered again using regular expressions to reduce formatting and other issues in the output results of the large model. Use the regular expression module in step (2) again.

[0074] (5) Replace the filtered text content with the text content item in the original news information data.

[0075] The text content data of each news information data in the initial filtering result data is replaced with the text result after filtering in the above steps, and other contents are kept unchanged, so that the data after text filtering is obtained.

[0076] Step 3 - Repeat the news data processing.

[0077] An efficient comparison algorithm is used to compare news information data with each other, identify and eliminate duplicate news information, and obtain the final filtered dynamic news information data set.

[0078] The large amount of dynamic news information crawled will inevitably contain duplicate information, which may be due to different reports on the same event by multiple data sources, or due to direct reprinting of an article or multiple crawling of data due to errors in crawler tools. A large amount of duplicate information is redundant and meaningless and will increase the subsequent workload. Therefore, it is very necessary to eliminate duplicate news information.

[0079] Specifically, this step 3 may include the following three steps:

[0080] (1) Extract the title item data of each news information data.

[0081] In view of the fact that the system does not want identical duplicate news, but requires multiple sources to verify the same news event, the system uses the title field to process duplicate news data. In the result data after the text is filtered, the title field data of each item is extracted, and the title field data is filtered separately later.

[0082] (2) After removing the spaces before and after the title character string, determine whether there is a prefix between multiple title character strings. If so, record one of them as duplicate news information.

[0083] First, the system removes the spaces before and after the title of each news data. Then, it searches for each title in turn to see if there is another title that is exactly the same as it or is a prefix of another title. If there is such a situation, it is added to the duplicate news data.

[0084] (3) Remove the data judged as duplicate news information to obtain the final filtered dynamic news information data set.

[0085] Create a new array to store the final deduplicated news information data in json format. If the title of the data is not in the duplicate data, add it to the new array. This can filter out duplicate news with the same title or a prefix of another title, and also avoid filtering out news with different descriptions of the same news event from multiple sources.

[0086] The dynamic news information filtering method proposed in the present invention can solve the quality problem of crawled news information data. It uses advanced technologies such as regular expressions and large language models, has a high degree of intelligence and automation, can accurately remove duplicates, and has good flexibility and scalability.

[0087] Figure 3 The present invention is a flowchart of text data noise cleaning in a dynamic news information data filtering method in one embodiment of the present invention.

[0088] like Figure 3 As shown in the figure, the text data in the news information data filtering method is processed in multiple steps, such as regular expressions and large models. In order to clearly show that the news information collectors are most concerned about the filtering of the news data text field, we will show the changes in its content in each step. The original text of a certain news information collected manually on the web page is shown in Table 1. The character length of the original text content is 1321, and the translated Chinese content is 387.

[0089] Table 1 Length of original text string on web page

[0090]

[0091] However, using a web crawler tool to crawl the webpage, the original text and Chinese content of the data are shown in Table 2. The original text string length is 3151, of which the impurity information accounts for 58%, which is more than the original text content. We can clearly see a large amount of other impurity information such as \t\n, which seriously affects the normal reading of people. At this time, the Chinese content string length is 1466, of which the impurity accounts for as high as 74%.

[0092] Table 2 Crawling results of crawler tools

[0093]

[0094]

[0095] The results of regular expression processing of the original content and Chinese content are shown in Table 3. The original content string length is 2573, and it can be clearly seen that the length has been reduced to a certain extent, but the impurity ratio is still 49% at this time, mainly because a large number of meaningless \n\t characters have been filtered out. In addition to the main text of the news, the result also contains impurity information such as the date, advertisement, and copyright notice on the web page. These impurities are difficult to identify and remove through simple string replacement, and manual identification and removal are also difficult. The situation of Chinese content is similar, with a string length of 925 and an impurity ratio of 58%.

[0096] Table 3 Regular expression processing results

[0097]

[0098]

[0099] After filtering with the large model, the final results are shown in Table 4, using the prompt "You are an impurity filter for news information. I will give you a piece of news text below. Please only extract the main text information, that is, the core content of the article, and remove all impurity information that is not related to the main text. These impurity information may include promotional information, social media tags, dates, author signatures, publishing platform logos that usually appear at the beginning of the text, and journal information, institution names, school logos, copyright instructions, references, etc. that usually appear at the end of the text, as well as video downloads and advertisements that may be interspersed in the text. These impurity information often contain shorter content and may be accompanied by more line breaks, but they do not constitute the core content of the article, so they need to be paid attention to and given priority. When extracting the main text, please ensure that the output text is in plain text format and does not contain Markdown, HTML, or any other formatting tags. At the same time, please retain the subheadings and paragraph structure of the original text. If the text is long, please divide it into appropriate sections to improve readability. Finally, please present the entire processed text content completely and accurately, and do not change the original text expression and language. "Borrowing the excellent semantic understanding ability of the large language model, the length of the original content string after processing is 1322, and the length of the Chinese content string is 383. The length of the processed result is almost the same as the original text result of the manually collected web page, with only slight differences in the representation of the connection points of a few sentences.

[0100] Table 4 Large model processing results

[0101]

[0102]

[0103] After the above processing, the filtered news information data, whether in original text or Chinese, almost completely restores the content of the web news itself, removes all impurities in the text data obtained by the crawler, and greatly reduces the difficulty of reading and analysis for news information collectors.

[0104] By adopting the embodiment of the present invention, the quality of news information data obtained by web crawling and other methods can be effectively improved. First, the key fields of the news information data are judged to be empty, and then the data content is processed using regular expressions and large models to reduce the impurity noise information therein, and then the data is compared with each other to eliminate duplicate news information data, and finally the news information data obtained after filtering is output. Through the above steps, incomplete and duplicate dynamic news information is eliminated, the noise information contained in the news information is reduced, the efficiency of news information work is greatly improved, and the high quality and integrity of the news information are ensured.

[0105] However, using a news information data web crawler tool to crawl the webpage, the original text content and Chinese content obtained are shown in Table 2. The original text string length is 3151, of which the impurity information accounts for 58%, which is more than the original text content. It can be clearly observed that a large amount of other impurity information such as \t\n will seriously affect the normal reading of news information collectors. At this time, the Chinese content string length is 1466, of which the impurity accounts for as high as 74%.

[0106] Accordingly, another aspect of the present invention provides a dynamic news information data filtering tool, comprising:

[0107] (1) A missing value data screening module is used to obtain a dynamic news information data set obtained by a web crawler, and pre-process the dynamic news information data set to obtain a dynamic news information data set after preliminary filtering; wherein the pre-processing includes a missing value check.

[0108] (2) A text data noise cleaning module is used to use a filtering prompt word instruction model to perform deep noise cleaning on the dynamic news information dataset after preliminary filtering, so as to identify and remove the impurity noise information contained in the dynamic news information dataset, and use regular expressions to remove character content within a preset range.

[0109] (3) A duplicate news data processing module is used to use a comparison algorithm model to compare the news information data obtained from different information sources contained in the dynamic news information data set, so as to eliminate duplicate information and obtain a deep filtered dynamic news information data set.

[0110] Figure 4 The following is a schematic diagram of the structure of a dynamic news information data filtering tool in an embodiment of the present invention. The dynamic news information data filtering tool includes the following modules:

[0111] Module 410: A missing value data screening module inputs a dynamic news information data set obtained by web crawling or other means, performs a blank check on key fields, and filters out data that lacks key information. Among them, key fields include document number, source, title, text, etc. The blank check is not limited to judging that the value is empty, and other meaningless values ​​should also be filtered out.

[0112] Among them, the missing value data screening module is specifically used to input the dynamic news information data set obtained by web crawling and other methods; each item of the data set is checked for key fields to be empty, and the document number, source, title, text, etc. are judged in turn whether they are meaningful. If there are key fields that are meaningless, they are marked; data with meaningless marks are removed to obtain preliminary filtering result data.

[0113] Module 420: A text data noise cleaning module uses regular expressions and large model processing algorithms to perform deep noise cleaning on the text content, automatically identifying and removing impurity noise information such as advertisements, irrelevant links, garbled codes, etc., to improve the quality of news information content.

[0114] Among them, the text data noise cleaning module is specifically used to extract the text content item data of each news information data; use regular expressions to process the web links, email addresses, emoticons, multiple consecutive line breaks, tabs and spaces that the data may contain in turn to reduce impurities in the text content; input the data into the large model, use appropriate filtering prompts, use its ability to understand the meaning of the text, perform intelligent filtering, and save the results; filter the results after filtering by the large model again using regular expressions to reduce formatting and other issues in the output results of the large model; replace the filtered text content with the text content item in the original news information data.

[0115] Module 430: Duplicate news data processing module, which uses an efficient comparison algorithm to compare news information data with each other, identify and eliminate duplicate news information, and obtain the final filtered dynamic news information data set.

[0116] Among them, the duplicate news data processing module is specifically used to extract the title item data of each news information data; after removing the spaces before and after the title string, it is determined whether there is a prefix between multiple title strings. If so, one of them is recorded as duplicate news information; the data judged to be duplicate news information is removed to obtain the final filtered dynamic news information data set.

[0117] Corresponding to the above method, the present invention also provides a device, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the device implements the steps of the method described above.

[0118] Figure 5 This is a schematic diagram of the computer equipment included in the device. Figure 5 The computer device 00 includes: a processor 01, a memory 02, and a computer program stored in the memory 02 and executable on the processor 01. When the processor 01 executes the computer program, the human factor data server access control method provided in the above method embodiment is implemented.

[0119] Among them, the processor 01 is connected to the memory 02, such as through a bus 03. The processor 01 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can implement or execute various exemplary logic blocks, modules and circuits described in combination with the contents disclosed in this application. The processor 01 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of DSP and microprocessors, etc. The bus 03 may include a path to transmit information between the above components. The bus 03 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 130 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The memory 02 is used to store a computer program corresponding to the human factor data server access control method of the above embodiment of the present application, and the computer program is controlled and executed by the processor 01. The processor 01 is used to execute the computer program stored in the memory 02 to implement the content shown in the above method embodiment.

[0120] Corresponding to the above method, the present invention also provides a computer-readable storage medium, on which a computer program / instruction is stored, and when the computer program / instruction is executed by a processor, the steps of the method described in any one of the above embodiments are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0121] Corresponding to the above method, the present invention also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any one of the above embodiments when executed by a processor.

[0122] Specifically, the advantages of the method proposed by the present invention are:

[0123] (1) High degree of intelligence and automation: This method integrates three modules: missing value data screening, text data noise cleaning, and duplicate news data processing, to achieve fully automated filtering and processing of news information data, and has intelligent characteristics with the help of the excellent semantic understanding ability of the large language model. This highly integrated processing method greatly reduces manual intervention, improves processing efficiency and the quality of news information data, and is suitable for processing large-scale data sets.

[0124] (2) This method can greatly improve the efficiency of news information acquisition, ensure the high quality and integrity of news information, quickly eliminate incomplete and repeated dynamic news information obtained by traditional crawler tools, and reduce the noise information contained in news information.

[0125] (2) High flexibility and scalability: This method was designed with flexibility and scalability in mind. Regular expressions and filter prompts of large models can be adjusted and optimized according to actual needs to meet the data processing needs of science and technology news information from different sources and types.

[0126] (3) Accurate deduplication: Using an efficient comparison algorithm, it can accurately identify and remove news data with highly similar titles, reduce duplicate news information generated by reprinting, and avoid filtering out different descriptions of the same news event from multiple sources, ensuring the effectiveness and diversity of the final data set.

[0127] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0128] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0129] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0130] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for filtering dynamic news information data, characterized in that: include: Obtain dynamic news information datasets obtained through web crawlers; Preprocessing the dynamic news information data set to obtain a preliminary filtered dynamic news information data set; wherein the preprocessing includes a null check; Use the filtering prompt words to instruct the large model to perform deep noise cleaning on the dynamic news information data set after preliminary filtering, so as to identify and remove the impurity noise information contained in the dynamic news information data set, and use regular expressions to remove the character content in a preset range; A comparative algorithm model is used to compare the news information data obtained from different information sources contained in the dynamic news information dataset to eliminate duplicate information and obtain a deeply filtered dynamic news information dataset.

2. The method according to claim 1, characterized in that The empty field check includes: performing an empty field check on the dynamic news information data set to filter out data with missing key fields; wherein the key fields include multiple types of document number, source, title and text.

3. The method according to claim 1, characterized in that The impurity noise information includes multiple types of advertisements, irrelevant links, garbled codes, HTML tags and special symbols.

4. The method according to claim 1, characterized in that: The filtering prompt word instructs the large model to perform deep noise cleaning on the dynamic news information data set after preliminary filtering to identify and remove the impurity noise information contained in the dynamic news information data set, and use regular expressions to remove character content within a preset range, including: Extracting text content item data from each item of news information data contained in the dynamic news information data set after preliminary filtering; Using a regular expression to remove character contents within a preset range contained in the text content item data; Input the text content item data processed by the regular expression into the large model, and use the pre-acquired filtering prompt words to instruct the large model to intelligently filter the input content to identify and remove the impurity noise information contained in the dynamic news information data set; Use regular expressions to correct the format of the output after intelligent filtering of large models; The text content item data in the original news information data is replaced with the output after the format is corrected.

5. The method according to claim 4, characterized in that The character content in the preset range includes multiple types of web links, email addresses, emoticons, consecutive multiple line breaks, tabs, and space characters.

6. The method according to claim 1, characterized in that The use of a comparison algorithm model to compare news information data obtained from different information sources contained in a dynamic news information data set includes: Extracting title item data of each news information data contained in the dynamic news information data set; The spaces before and after the title character string of the title item data are removed, and it is determined whether there is duplication between the news information data obtained from different information sources, and the duplicate news information data is eliminated.

7. A dynamic news information data filtering tool, characterized in that: include: A missing value data screening module is used to obtain a dynamic news information data set obtained by a web crawler, and pre-process the dynamic news information data set to obtain a dynamic news information data set after preliminary filtering; wherein the pre-processing includes a blank value check; The text data noise cleaning module is used to use the filtering prompt word instruction large model to perform deep noise cleaning on the dynamic news information data set after preliminary filtering, so as to identify and remove the impurity noise information contained in the dynamic news information data set, and use regular expressions to remove the character content within a preset range; The duplicate news data processing module is used to use a comparison algorithm model to compare the news information data obtained from different information sources contained in the dynamic news information data set, so as to eliminate duplicate information and obtain a deep-filtered dynamic news information data set.

8. The dynamic news information data filtering tool according to claim 7, characterized in that: The dynamic news information data set obtained by means of a web crawler is in JSON format.

9. A device comprising a processor, a memory and a computer program / instructions stored in the memory, characterized in that: The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the tool implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method as claimed in any one of claims 1 to 7 are implemented.

Images