Multi-source public news policy collection method and platform

Abstract

The invention discloses a multi-source public news policy collection method and platform, and the platform comprises a multi-source public news policy collection module, a multi-source public news policy formatting processing and storage module, and a multi-source public news policy automatic collection module. And the acquired formatted information can be retrieved. The method not only solves the problems of scattered public news policy information and non-uniform data formats, but also realizes cross-platform and cross-domain multi-source public news policy information interconnection and intercommunication, and the technical architecture can also be applied to information fusion platforms in other fields, promotes information fusion and development, and has wide application prospects. And the portability and the efficiency of users and technicians are improved.

Description

Technical Field

[0001] The present invention relates to the field of Internet information collection and data processing, and in particular to a method and platform for collecting multi-source public news policies. Background Art

[0002] In today's rapidly evolving information and news landscape, the sources of policy news are becoming increasingly diverse, encompassing a wide range of channels, including official government websites, major news and policy departments, and media platforms. This multi-source public information is extremely valuable to people from all walks of life, including policy researchers, corporate decision-makers, and the general public. It helps people stay informed about policy developments and make informed decisions and actions.

[0003] Currently, public policy news from multiple sources is scattered across multiple websites and platforms, each with its own unique data formats and update frequencies. This makes it difficult for users to access public policy news. For example, technical personnel may need to manually visit multiple websites to search and record policy and development information, a time-consuming and labor-intensive process that can easily lead to missing important information.

[0004] Chinese patent publication number CN103310020A proposes a "Standard Information Synchronization and Status Change Warning System and Method," which describes a modular approach to acquiring standard resources and establishing a standard resource processing module and indexing module to store standard information. However, this approach can only acquire a limited amount of website information and cannot integrate information from multiple websites with different structures. Summary of the Invention

[0005] To address the issues raised by the background art, the present invention provides a multi-source public news policy collection method and platform. The present invention aims to collect public news policies, categorize and integrate them for storage, enable retrieval of the collected news policies, and build a system platform to integrate news policies from various websites.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A multi-source public news policy collection platform collects public news policy information from multiple sources, formats and stores the collected information, and is capable of searching the collected formatted information. The platform includes the following modules:

[0008] Multi-source public news policy collection module: This module collects legal information from multi-source public news policy websites and has the function of checking previously stored news policies, obtaining news policies that are not covered by the multi-source public news policy collection platform during collection;

[0009] Multi-source public news policy formatting and storage module: This module stores the content collected by the multi-source public news policy collection module, formats the collected multi-source public news policies, and stores them in a server or local file;

[0010] Multi-source public news policy automatic collection module: The module has the function of automatically collecting legal information from multi-source public news policy websites. It can execute the multi-source public news policy collection module manually or automatically according to the set time to complete the automatic update of multi-source public news policies.

[0011] Furthermore, the platform stores public news policy information from multiple sources, and formats the information before fusing it and storing it in a server or a local file.

[0012] Furthermore, the multi-source public news policy collection module includes the following process steps:

[0013] 1) Read the news policy content stored in the previous Json file;

[0014] 2) Use Beautiful Soup to parse the HTML page structure and dynamically splice relative links;

[0015] 3) Match the latest news and policy information and check the content that needs to be collected;

[0016] 4) Collect all news items according to title, URL, date, source, document number, and content. If the collected news policy does not contain relevant content, fill in None.

[0017] Furthermore, the multi-source public news policy formatting processing storage module stores according to title, URL, date, source, document number, and content. Each group is an overall news policy. The multi-source public news policy is classified and stored in different Json files. The final Json file content is sorted by date in descending order and deduplicated, and saved to the server or local file.

[0018] Furthermore, the multi-source public news policy automatic collection module includes the following process steps:

[0019] 1) If the public news policy is stored in a local file, manually run the multi-source public news policy collection module and store the updated public news policy in a JSON file;

[0020] 2) If the public news policy is stored on the server, set up scheduled tasks to run the multi-source public news policy collection module regularly and store the updated public news policy in a JSON file;

[0021] 3) If the public news policy is stored in the server, you need to update the content at this time, manually run the multi-source public news policy collection module, and store the updated public news policy in a Json file.

[0022] Furthermore, the retrieval of the collected formatted information is a keyword retrieval of the collected formatted and stored public news policy titles, URLs, dates, sources, document numbers, and contents.

[0023] Furthermore, the multi-source public news policy collection module adopts multi-threaded concurrent requests and automatic retry mechanisms.

[0024] The present invention also provides a multi-source public news policy collection method, comprising the following steps:

[0025] Step 1: Establish a dedicated multi-source public news policy collection platform to interact with administrators, public news policy sources, user platforms and users;

[0026] Step 2: Establish different data collection mechanisms and use multi-threaded concurrent requests and automatic retry mechanisms to categorize the collected content by title, URL, date, source, document number, and content;

[0027] Step 3: The multi-source public news policy collection platform has a built-in formatting and storage module. The storage module stores different sources in different Json files, formats and stores the collected content according to title, URL, date, source, document number, and content. Each group is a piece of news policy information, and the final Json file content is sorted by date in descending order and deduplication is performed;

[0028] Step 4: Obtain public news policies from multiple sources;

[0029] Step 5: Utilize legal information collection techniques to obtain formatted information and update the information manually or automatically at a set time to collect multiple sources of public news policies; the process includes the following steps:

[0030] 1) First read the previous Json file content;

[0031] 2) Match the content of multi-source public news policies with the content of previous Json files;

[0032] 3) Add the acquired update information to a Json file, sort by date in descending order, and remove duplicates;

[0033] 4) Collect public news policies from multiple sources manually or automatically at set times during off-peak hours.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] The multi-source public news policy collection method and platform of the present invention can collect multi-source public news policies and search the collected news policies, which can break the information silos and realize the interconnection of news policies across platforms and fields. It provides comprehensive, accurate and real-time information support for users and technicians who need to obtain news policies. This technology can be applied to information fusion platforms in other fields, promote the information fusion and development of information platforms, improve the portability and efficiency of users and technicians, and contribute to the country's high-quality development strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is the overall flow chart of the multi-source public news policy collection platform.

[0037] Figure 2 This is the flow chart of the multi-source public news policy collection module.

[0038] Figure 3 Flowchart for automatic collection of multi-source public news policies. DETAILED DESCRIPTION

[0039] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0040] The specific implementation process is as follows:

[0041] 1. As Figure 1 As shown in the figure, the overall flow chart of the multi-source public news policy collection platform is as follows. The platform collects public news policy information from multiple sources, formats the collected information, and stores it. The collected formatted information can be retrieved. The platform includes the following modules:

[0042] Multi-source public news policy collection module: This module collects legal information from multi-source public news policy websites and has the function of checking previously stored news policies, obtaining news policies that are not covered by the multi-source public news policy collection platform during collection;

[0043] Multi-source public news policy formatting and storage module: This module stores the content collected by the multi-source public news policy collection module, formats the collected multi-source public news policies, and stores them in a server or local file;

[0044] Multi-source public news policy automatic collection module: The module has the function of automatically collecting legal information from multi-source public news policy websites. It can execute the multi-source public news policy collection module manually or automatically according to the set time to complete the automatic update of multi-source public news policies.

[0045] 2. The platform stores public news policy information from multiple sources, formats the information, and integrates and stores it in a server or local file.

[0046] 3. Figure 2 The multi-source public news policy collection module includes the following process steps:

[0047] 1) Read the news policy content stored in the previous Json file;

[0048] 2) Use Beautiful Soup to parse the HTML page structure and dynamically splice relative links;

[0049] 3) Match the latest news and policy information and check the content that needs to be collected;

[0050] 4) Collect all news items according to title, URL, date, source, document number, and content. If the collected news policy does not contain relevant content, fill in None.

[0051] 4. The multi-source public news policy formatting processing storage module stores according to title, URL, date, source, document number, and content. Each group is an overall news policy. The multi-source public news policies are classified and stored in different Json files. The final Json file content is sorted by date in descending order and deduplicated, and saved to the server or local file.

[0052] 5. If Figure 3 The multi-source public news policy collection platform shown and described is characterized in that the multi-source public news policy automatic collection module includes the following process steps:

[0053] 1) If the public news policy is stored in a local file, manually run the multi-source public news policy collection module and store the updated public news policy in a JSON file;

[0054] 2) If the public news policy is stored on the server, set up scheduled tasks to run the multi-source public news policy collection module regularly and store the updated public news policy in a JSON file;

[0055] 3) If the public news policy is stored in the server, you need to update the content at this time, manually run the multi-source public news policy collection module, and store the updated public news policy in a Json file.

[0056] 6. The multi-source public news policy collection platform is characterized in that the retrieval of the collected formatted information is a keyword search of the collected formatted and stored public news policy title, URL, date, source, document number, and content.

Claims

1. A multi-source public news policy collection platform, characterized by: The platform collects public news policy information from multiple sources, formats and stores the collected information, and is capable of searching the collected formatted information. The platform includes the following modules: Multi-source public news policy collection module: This module collects legal information from multi-source public news policy websites and has the function of checking previously stored news policies, obtaining news policies that are not covered by the multi-source public news policy collection platform during collection; Multi-source public news policy formatting and storage module: This module stores the content collected by the multi-source public news policy collection module, formats the collected multi-source public news policies, and stores them in a server or local file; Multi-source public news policy automatic collection module: The module has the function of automatically collecting legal information from multi-source public news policy websites. It can execute the multi-source public news policy collection module manually or automatically according to the set time to complete the automatic update of multi-source public news policies.

2. A multi-source public news policy collection platform according to claim 1, characterized in that: The platform stores multi-source public news policy information, and formats the information and then integrates and stores it in a server or a local file.

3. A multi-source public news policy collection platform according to claim 1, characterized in that: The multi-source public news policy collection module includes the following process steps: 1) Read the news policy content stored in the previous Json file; 2) Use Beautiful Soup to parse the HTML page structure and dynamically splice relative links; 3) Match the latest news and policy information and check the content that needs to be collected; 4) Collect all news items according to title, URL, date, source, document number, and content. If the collected news policy does not contain relevant content, fill in None.

4. A multi-source public news policy collection platform according to claim 1, characterized in that: The multi-source public news policy formatting processing storage module stores according to title, URL, date, source, document number, and content. Each group is an overall news policy. The multi-source public news policies are classified and stored in different Json files. The final Json file content is sorted by date in descending order and deduplicated, and saved to the server or local file.

5. A multi-source public news policy collection platform according to claim 1, characterized in that: The multi-source public news policy automatic collection module includes the following process steps: 1) If the public news policy is stored in a local file, manually run the multi-source public news policy collection module and store the updated public news policy in a JSON file; 2) If the public news policy is stored on the server, set up scheduled tasks to run the multi-source public news policy collection module regularly and store the updated public news policy in a JSON file; 3) If the public news policy is stored in the server, you need to update the content at this time, manually run the multi-source public news policy collection module, and store the updated public news policy in a Json file.

6. A multi-source public news policy collection platform according to claim 1, characterized in that: The retrieval of the collected formatted information is a keyword retrieval of the collected formatted and stored public news policy titles, URLs, dates, sources, document numbers, and contents.

7. A multi-source public news policy collection platform according to claim 1, characterized in that: The multi-source public news policy acquisition module adopts multi-threaded concurrent request and automatic retry mechanism.

8. A method for collecting multi-source public news policies, characterized in that: These include: Step 1: Establish a dedicated multi-source public news policy collection platform to interact with administrators, public news policy sources, user platforms and users; Step 2: Establish different data collection mechanisms and use multi-threaded concurrent requests and automatic retry mechanisms to categorize the collected content by title, URL, date, source, document number, and content; Step 3: The multi-source public news policy collection platform has a built-in formatting and storage module. The storage module stores different sources in different Json files, formats and stores the collected content according to title, URL, date, source, document number, and content. Each group is a piece of news policy information, and the final Json file content is sorted by date in descending order and deduplication is performed; Step 4: Obtain public news policies from multiple sources; Step 5: Utilize legal information collection techniques to obtain formatted information and update the information manually or automatically at a set time to collect multiple sources of public news policies; the process includes the following steps: 1) First read the previous Json file content; 2) Match the content of multi-source public news policies with the content of previous Json files; 3) Add the acquired update information to the Json file, sort by date in descending order, and remove duplicates; 4) Collect public news policies from multiple sources manually or automatically at set times during off-peak hours.

Images