Internet data real-time downloading system based on configuration foreground

Abstract

The invention discloses an internet data real-time downloading system and method based on a configuration foreground. The method comprises the steps that P1, an interface front-end framework based on Vue.js is built and achieved; p2, building and realizing an interface rear-end framework; step P3, performing parameter transmission on a back-end data downloading program based on the front-end configuration parameters; step P4, automatically downloading the Internet data; and P5, monitoring the integrity and timeliness of data downloading. The system comprises a foreground configuration module, a JSON parameter transmission module, a background downloading module, a task scheduling module and a log and monitoring module. According to the invention, automatic, flexible and efficient downloading of Internet data can be realized, and the problems of complex configuration, low downloading efficiency, poor usability and the like of an existing downloading tool are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of real-time download of Internet data, in particular to a real-time download system and method of Internet data based on configuration foreground. BACKGROUND

[0002] With the rapid development of the Internet, massive and diverse big data resources are growing explosively, and the demand for Internet data download of enterprises and individual users is increasing. There are rich data resources on various data centers, research institutions and public websites waiting to be discovered and collected.

[0003] Traditional data download methods usually require manual programming or command line operations, which lack flexibility and ease of use. Although existing download tools (such as lftp, wget, etc.) are powerful, they are complex to configure and difficult to meet the needs of non-technical users. In addition, the integration and unified management of different download tools also face challenges. Therefore, it is urgent to develop a system that can realize the integration of multiple download technologies through foreground configuration operation to improve the efficiency and ease of use of data download.

[0004] The Internet data download system is based on various data source channels on the Internet to carry out data demand collection, data download, data management, data display, data monitoring and other work. At the same time, the active discovery capability based on external data websites is built to continuously realize the systematic collection of various industry data and realize the active collection and unified management of various source disaster, environmental, social and economic data. SUMMARY

[0005] Therefore, the technical problem to be solved by the present application is to provide a real-time download system and method of Internet data based on configuration foreground combining multiple download technologies such as lftp, wget, etc. to realize automatic, flexible and efficient download of Internet data and solve the problems of complex configuration, low download efficiency and poor ease of use of existing download tools.

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] A real-time download method of Internet data based on configuration foreground, comprising the following steps:

[0008] Step P1: build and implement a Vue.js-based interface front-end framework;

[0009] Step P2: build and implement an interface back-end framework;

[0010] Step P3: based on the front-end configuration parameters, pass parameters to the back-end data download program;

[0011] Step P4: automatically download Internet data;

[0012] Step P5: Monitor the integrity and timeliness of data download.

[0013] The above-mentioned real-time download method of Internet data based on configuration front-end, step P1 includes the following steps:

[0014] Step P101: Use the Vue.js framework to build the interface front-end, and use the AES / ECB / PKCS5Padding algorithm to encrypt the user login to ensure user privacy and security;

[0015] Step P102: The user page is built in a text interface input manner to construct the front-end interaction module of demand name, demand website, demand frequency, file format and storage path;

[0016] Step P103: User page display, front-end uses HTTP request data interface, and uses virtual DOM technology to realize efficient update and rendering of existing data types and data types being downloaded, uses responsive system to automatically track data changes, realizes real-time view data synchronization of the page through embedding JavaScript expressions and instructions in HTML, and uses component technology to improve code reusability;

[0017] Step P104: Use Vue CLI to realize hot reloading, and use Vue Router to realize navigation and data transmission between pages.

[0018] The above-mentioned real-time download method of Internet data based on configuration front-end, in step P101, the implementation process of the encryption algorithm when logging in to the page is as follows:

[0019] Step P101-1, user information collection: the user inputs the username and password in the front-end login page;

[0020] Step P101-2, encryption preparation: the system preloads symmetric encryption keys in the front-end; use AES algorithm, adopt ECB mode and PKCS5Padding padding method to encrypt the password; before encryption, convert the plaintext password to UTF-8 encoding format, and the key length is 128 bits;

[0021] Step P101-3, encryption result processing: the encrypted ciphertext is processed by Base64 encoding;

[0022] Step P101-4, transmission and verification: the front-end submits the encrypted password and username to the back-end through POST request;

[0023] In step P102, the front-end interaction module includes a requirement name input box, a data source URL input box, a requirement frequency selection box, a file format selection box, and a storage path input box; the front-end interaction module uses the data binding mechanism in the Vue.js framework to realize real-time data synchronization and bidirectional binding, and all fields are designed in combination with a form verification mechanism, error prompts and guidance, responsive design, and data caching and recovery functions;

[0024] In step P103, when the HTTP request data interface, the front-end sends a GET request to the back-end data interface through the Axios client library to obtain existing data categories and categories of data being downloaded;

[0025] In step P104, the working process of using Vue CLI to realize hot reloading is as follows: starting the development service command, when the developer modifies the Vue component, JavaScript script or style sheet file locally, the Vue CLI automatically detects the changes and triggers hot reloading; the reloading mechanism is realized based on the HMR plug-in of Webpack, and the specific configuration is encapsulated and automatically integrated by Vue CLI;

[0026] The working process of using Vue Router to realize navigation and data transmission between pages is as follows: uniformly defining the path and component mapping relationship of each functional page to realize the page navigation of “home page”, “automated collection” and “data resource management”; through the built-in <router-link>The component or JavaScript control mode realizes the jump, and data is carried through a route parameter or a Query parameter during page jump.

[0027] The above-mentioned internet data real-time downloading method based on the configuration foreground, step P2 includes steps of:

[0028] Step P201: the front end transmits the download demand detailed information to the back end in an HTTP mode; the technologies used in the transmission between the front end and the back end include a firewall, Netty, Nginx and HTTP, and a RESTful style is used to ensure efficient and safe transmission of information;

[0029] Step P202: the back end uses Spring Boot and XXL-JOB distributed task scheduling center for linkage to manage scheduling and concurrent execution of multiple download tasks, and the submitted download data document detailed information is stored in a MySQL database; the data Spring Boot dynamically configures xxljob timing tasks by using the data of the MySQL database, and the python program is called in a timing mode to download data, and the number of concurrent executions and the number of retries are supported;

[0030] Step P203: the real-time download data table is filtered into data total categories and all data information tables by Spring Boot, and the data information is transmitted to a page.

[0031] In step P201 of the above-mentioned internet data real-time downloading method based on the configuration foreground, the front end initiates a request to transmit download demand information to the back end in an HTTP POST request mode by using Axios or Fetch API; the back end interface is designed in a RESTful style, and the interface transmission uniformly uses a JSON format; the back end service is built based on a Netty framework, is responsible for receiving a front end request, forwarding to a core business module, and returning a response result; and a firewall is configured in a data transmission process; Nginx is deployed as a reverse proxy server between the front end and the back end, is responsible for an SSL terminal, load balancing and request forwarding, configures an HTTPS protocol and deploys an SSL certificate, realizes transmission encryption and identity authentication; an interface access frequency threshold is set in Nginx or the back end service, the back end receives demand information, checks parameter content, generates a task record and returns a task ID and confirmation information if the verification is passed; the front end receives a response, automatically switches to a "task state" page and loads task details, and completes a closed-loop interaction;

[0032] The implementation process of step P202 is as follows:

[0033] Step P202-1, the backend task management service is developed based on the Spring Boot framework, and after receiving the request from the front end, the submitted download requirement parameters are format-verified and semantically parsed, and the related fields are written into the task table in the MySQL database;

[0034] Step P202-2, XXL-JOB is used as a scheduling center, and the Spring Boot server integrates the XXL-JOB executor to dynamically submit the tasks in the task table to the XXL-JOB console in a registration form;

[0035] Step P202-3, the Spring Boot executor receives the scheduling center invocation request, loads the corresponding task information from the database according to the input task ID or parameters, and calls the preset Python script in the backend to start the data download process;

[0036] Step P202-4, the maximum number of concurrent tasks is configured, and the thread pool or queue flow limiting method is used to achieve this; when a task fails to download, the retry_count field is used to perform a maximum of a specified number of retries, and the retry interval and strategy can be dynamically adjusted through configuration items; the running record, log information, execution time and result of each scheduling task are managed by the XXL-JOB platform, and visual monitoring and troubleshooting are performed through the Web console; the task status can be controlled bidirectionally through the database field and the XXL-JOB console;

[0037] In step P203, after the download task is completed, the Python script writes the download record of each data into the download_log table in the MySQL database; the Spring Boot control layer extracts all downloaded data from the download_log table by calling the Service layer business logic, and groups and counts them according to the category field to generate a "data total category" list, and further constructs a detailed table according to the subcategory and file field; the backend uses RESTful style to define the interface, and the front end requests the interface through Axios to obtain the classification result JSON data, and renders it into a structured display list of "data category + data detailed table" in the page.

[0038] The above-mentioned Internet data real-time download method based on the configuration front end includes the following steps:

[0039] Step P301: Convert the configuration information into JSON format parameters, and generate corresponding JSON configuration files for the related parameters configured on the graphical page, including data source address, download frequency, file type, storage path and retry strategy;

[0040] Step P302: The backend parses the JSON format parameter, selects a download tool according to the parsed parameter, passes the parameter into the tool field in the JSON parameter, and executes a corresponding data acquisition task;

[0041] Step P303: A Python program reads and parses a JSON file generated by the front-end page configuration, calculates a specific date according to a specific date identifier and a time interval parameter preset in the configuration file, and passes the value to a corresponding position in the JSON file to realize automatic assignment of the date parameter in the data acquisition task;

[0042] Step P304: The program automatically adds a corresponding prefix to each file name to be processed according to the user inputted prefix matching symbol preset by the front-end page, and the added prefix contains timestamp information; at the same time, the program records the original file name, file size and download time of each file to a database table.

[0043] The above-mentioned Internet data real-time download method based on the configuration front end includes the following steps:

[0044] Step P401: An automatic proxy pool is established to dynamically allocate and switch the used proxy IP;

[0045] Step P402: The proxy IP is used to request when the script configuration is executed, the proxy IP is switched, and the single IP ban caused by the timing request is reduced;

[0046] Step P403: A UA library containing multiple browsers and devices is established, the UA library contains browser type, version number and operating system information, different UAs are set to access target websites; a UA is randomly selected each time to simulate multiple device and browser environments;

[0047] Step P404: The download tool is selected according to the tool field in the JSON parameter, the download module matches the download information of the website, date and file name based on the selected download tool; the breakpoint resume technology is adopted in combination with the data verification mechanism with.tmp suffix to ensure the integrity of the downloaded content, and multiple protocols including FTP, SFTP and HTTP are supported; during the download process, the system records logs in real time and returns status information to the front end;

[0048] Step P405: The problem data is screened by the file name comparison and / or file size comparison method, and the automatic retry is realized through the internal loop instruction or external scheduling of the program;

[0049] Step P406: The storage path collected in step P101 is identified, and the downloaded data is stored in the specified position.

[0050] The above-mentioned Internet data real-time download method based on the configuration foreground, step P5 includes the following steps:

[0051] Step P501: The system uses the mapping relationship between the identification information embedded in the file name and the pre-defined delay parameter to build a mapping table or data structure, and realizes the corresponding management between the file name and the file delay time; based on this mapping relationship, the system uses a pre-set algorithm to accurately calculate the scheduled arrival time of each file;

[0052] Step P502: After each file download operation is completed, the downloaded file quantity and corresponding data quantity are automatically counted through event triggering or callback function, and these statistical information is written into the database in real time by using an efficient transaction processing mechanism; the database uses a specially designed table structure, including timestamp, file count and data volume fields;

[0053] Step P503: Use Spring Boot timing to push the download detailed record log to the DI interface to realize the record of the event, and record the detailed information of the data download to a table in MySQL, and judge whether the data is abnormal by frequently querying the real-time updated download record table to push the alarm and display the red floating on the page.

[0054] The above-mentioned Internet data real-time download method based on the configuration foreground, step P1 includes the following steps:

[0055] Step P101: Use Vue.js framework to build the interface front end, and use AES / ECB / PKCS5Padding algorithm to encrypt the page login to ensure user privacy and security; when logging in the page, the implementation process of the encryption algorithm is as follows:

[0056] Step P101-1, user information collection: the user inputs the username and password in the front-end login page;

[0057] Step P101-2, encryption preparation: the system pre-sets the symmetric encryption key in the front end; use AES algorithm, adopt ECB mode and PKCS5Padding padding method to encrypt the password; before encryption, convert the plaintext password to UTF-8 encoding format, and the key length is 128 bits;

[0058] Step P101-3, encryption result processing: the encrypted ciphertext is processed by Base64 encoding;

[0059] Step P101-4, transmission and verification: the front end submits the encrypted password and username to the back end through POST request;

[0060] Step P102: The user page constructs a front-end interaction module for entering the demand name, demand URL, demand frequency, file format, and storage path in a text interface; the front-end interaction module includes a demand name input box, a data source URL input box, a demand frequency selection box, a file format selection box, and a storage path input box; the front-end interaction module uses the data binding mechanism in the Vue.js framework to achieve real-time data synchronization and bidirectional binding, and all fields are designed in combination with a form verification mechanism, error prompts and guidance, responsive design, and data caching and recovery functions;

[0061] Step P103: The user page is displayed, the front-end uses an HTTP request data interface, and uses virtual DOM technology to efficiently update and render the existing data categories and the data categories being downloaded, uses a responsive system to automatically track data changes, realizes real-time view data synchronization by binding events and data through embedding JavaScript expressions and instructions in HTML, and uses componentization technology to improve code reusability; when the HTTP request data interface is used, the front-end sends a GET request to the back-end data interface through the Axios client library to obtain the existing data categories and the data categories being downloaded;

[0062] Step P104: Hot reloading is implemented using Vue CLI, and page navigation and data transmission are implemented using Vue Router; the working process of hot reloading using Vue CLI is as follows: start the development service command, when the developer modifies the.vue component, JavaScript script, or style sheet file locally, Vue CLI automatically detects the changes and triggers hot reloading; the reloading mechanism is implemented based on the HMR plugin of Webpack, and the specific configuration is encapsulated and automatically integrated by Vue CLI;

[0063] The working process of using Vue Router to implement page navigation and data transmission is as follows: the path and component mapping relationship of each functional page is uniformly defined to realize the page navigation of "home page", "automated collection", and "data resource management"; the built-in <router-link>The navigation can be implemented using components or JavaScript controls, with data carried through route parameters or query parameters during page transitions.

[0064] Step P2 includes the following steps:

[0065] Step P201: The frontend transmits the download request details to the backend via HTTP. Technologies used in this transmission include firewall, Netty, Nginx, and HTTP, employing a RESTful style to ensure efficient and secure data transmission. The frontend sends the download request information to the backend via an HTTP POST request using either Axios or the Fetch API. The backend interface is designed using a RESTful style, and JSON format is used for all data transmissions. The backend service, built on the Netty framework, is responsible for receiving frontend requests, forwarding them to core business modules, and returning response results. Firewall configuration is performed during data transmission. Nginx is deployed between the frontend and backend as a reverse proxy server, responsible for SSL termination, load balancing, and request forwarding. HTTPS is configured, and an SSL certificate is deployed to achieve encrypted transmission and authentication. An interface access frequency threshold is set in Nginx or the backend service. After receiving the request information, the backend verifies the parameters. If verification is successful, a task record is generated, and the task ID and confirmation information are returned. Upon receiving the response, the frontend automatically switches to the "Task Status" page and loads the task details, completing a closed-loop interaction.

[0066] Step P202: The backend uses Spring Boot and the XXL-JOB distributed task scheduling center to manage the scheduling and concurrent execution of multiple download tasks. Detailed information of the submitted download data documents is stored in a MySQL database. Spring Boot dynamically configures scheduled tasks using the data from the MySQL database to periodically call the Python program to download the data, supporting configuration of concurrency and retries. The implementation process of step P202 is as follows:

[0067] Step P202-1: The backend develops a task management service based on the Spring Boot framework. After receiving the frontend request, it performs format validation and semantic parsing on the submitted download request parameters and writes the relevant fields into the task table in the MySQL database.

[0068] Step P202-2: Use XXL-JOB as the scheduling center. Integrate the XXL-JOB executor into the Spring Boot server and dynamically submit tasks from the task table to the XXL-JOB console in the form of registration.

[0069] Step P202-3, the scheduling center call request is received by the Spring Boot executor, the corresponding task information is loaded from the database according to the incoming task ID or parameters, and the preset Python script of the backend is called to start the data download process;

[0070] Step P202-4, the maximum number of concurrent tasks is configured, and the thread pool or queue throttling method is used to achieve it; when a task fails to download, according to the retry_count field, a maximum of a specified number of retries is performed, and the retry interval and strategy can be dynamically adjusted through configuration items; the running record, log information, execution time and result of each scheduling task are managed by the XXL-JOB platform, and visual monitoring and troubleshooting are performed through the Web console; the task status can be controlled bidirectionally through the database field and the XXL-JOB console;

[0071] Step P203: filter the real-time download data table into data total categories and all data information tables through Spring Boot, and pass the data information to the page; after the download task is completed, the Python script writes the download record of each data into the download_log table in the MySQL database; the Spring Boot control layer extracts all downloaded data from the download_log table by calling the Service layer business logic, and groups and counts them according to the category field to generate a "data total category" list, and further constructs detailed tables according to the subcategory and file field; the backend uses RESTful style to define the interface, the front end requests the interface through Axios to obtain the classification result JSON data, and renders it into a structured display list of "data category + data detailed table" in the page;

[0072] Step P3 includes the following steps:

[0073] Step P301: convert the configuration information into JSON format parameters, and generate a corresponding JSON configuration file for the related parameters configured on the graphical page, including data source address, download frequency, file type, storage path and retry strategy;

[0074] Step P302: the backend parses the JSON format parameters, selects the download tool according to the parsed parameters, selects the download module (such as lftp, wget or a custom collector) according to the parsed parameters, passes the tool field in the JSON parameters, and executes the corresponding data acquisition task;

[0075] Step P303: The Python program reads and parses the JSON file generated by the front-end page configuration, calculates the specific date according to the preset specific date identifier and its time interval parameter in the configuration file, and passes the value to the corresponding position in the JSON file to realize the automatic assignment of the date parameter in the data collection task;

[0076] Step P304: The program automatically adds a corresponding prefix to each file name to be processed according to the user inputted prefix matching symbol in the front-end page, and the added prefix contains timestamp information; at the same time, the program records the original file name, file size and download time of each file to the database table; in order to compare the information of the file in the subsequent operation, thereby effectively avoiding repeated downloading.

[0077] Step P4 includes the following steps:

[0078] Step P401: Establish an automatic proxy pool to dynamically allocate and switch the used proxy IP;

[0079] Step P402: Use proxy IP to request when executing script configuration, switch proxy IP to reduce single IP ban caused by timing request;

[0080] Step P403: Establish a UA library containing multiple browsers and devices, the UA library contains browser type, version number and operating system information, set different UA to access target website; randomly select UA for each request to simulate multiple device and browser environment;

[0081] Step P404: Select the download tool according to the tool field in the JSON parameter, the download module matches the download information of the website, date and file name based on the selected download tool; adopt breakpoint resume technology combined with.tmp suffix data verification mechanism to ensure the integrity of the downloaded content, support multiple protocols including FTP, SFTP and HTTP; during the download process, the system records logs in real time and returns status information to the front end;

[0082] Step P405: During the process of downloading data, step P404 may fail due to network delay and other problems, therefore, file name comparison and / or file size comparison method is adopted to screen problem data, and automatic retry is realized through internal loop instruction or external scheduling of the program;

[0083] Step P406: Identify the storage path collected in step P101, and store the downloaded data to the specified location;

[0084] Step P5 includes the following steps:

[0085] Step P501: The system uses the mapping relationship between the identification information embedded in the file name and the predefined delay parameter to build a mapping table or data structure, which realizes the corresponding management between the file name and the file delay time; based on this mapping relationship, the system uses a preset algorithm to accurately calculate the scheduled arrival time of each file; thereby supporting subsequent scheduling and resource optimization allocation, realizing efficient management and predictive control of asynchronous data flow;

[0086] Step P502: After each file download operation is completed, the number of downloaded files and the corresponding data volume are automatically counted through event triggering or callback functions, and these statistical information is written into the database in real time using an efficient transaction processing mechanism; the database uses a specially designed table structure, including timestamp, file count and data volume fields, to provide accurate and real-time data support for subsequent query, statistical analysis and performance monitoring, thereby optimizing the management and resource allocation of the overall download task of the system;

[0087] Step P503: Use Spring Boot timing to push the download detailed record log to the DI interface to realize event recording, and record the detailed information of data download to a table in MySQL at regular intervals, and update the download record table in real time by frequent query to determine whether the data is abnormal and push the alarm and display the red floating on the page.

[0088] An internet data real-time download system based on a configuration front end, for implementing the above-mentioned internet data real-time download method based on a configuration front end, comprising: a front-end configuration module, a JSON parameter passing module, a background download module, a task scheduling module and a log and monitoring module.

[0089] The front-end configuration module is used to provide a graphical interface, and the user configures the parameters of the download task through the interface.

[0090] The JSON parameter passing module is used to convert the parameters configured by the front-end configuration module into JSON format and pass them to the background download module.

[0091] The background download module is used to integrate multiple download tools including lftp, wget and crawler.

[0092] The task scheduling module is used to manage the concurrent execution of multiple download tasks.

[0093] The log and monitoring module is used to realize real-time task monitoring and log recording functions.

[0094] The above internet data real-time downloading system based on the configuration foreground, the user configures the downloading task in the foreground interface by using the foreground configuration module; the foreground converts the configuration information into JSON format and transmits it to the background; the background analyzes the JSON parameters and calls the corresponding downloading tool to execute the downloading task; the system monitors the task state in real time and feeds back to the user.

[0095] The technical scheme of the present application achieves the following beneficial technical effects:

[0096] 1. The internet data real-time downloading system and method based on the configuration foreground, the downloading task is configured through the foreground interface, and the configuration information is transmitted to the background lftp, wget and other downloading programs by using JSON parameter transmission, and flexible, stable and efficient data downloading is realized through date recognition, file name screening and other technical methods. The system can support the integration of multiple downloading tools, provide real-time monitoring and log recording functions, and is suitable for various internet data downloading scenarios.

[0097] 2. The present application provides a user-friendly graphical foreground interface, which significantly reduces the complexity of user configuration of the downloading task, and users do not need to master complex command line operations or script writing skills, but only need to fill in the relevant parameters through the interface to complete the real-time downloading of internet data. Non-technical personnel can also quickly complete the data downloading task, expanding the application range and user group of the system. The integration of multiple downloading tools (such as lftp, wget, crawler, etc.) is supported, and users can flexibly select the appropriate downloading tool according to specific needs, and through the task scheduling module, efficient concurrent execution of multiple tasks is realized, which can fully utilize system resources, significantly improve the efficiency of data downloading, and ensure the stability and efficiency in large-scale data downloading scenarios. The present application starts error handling and automatic retry mechanism to ensure successful data downloading in high-delay network environment, ensuring the integrity and continuity of data transmission. BRIEF DESCRIPTION OF DRAWINGS

[0098] Figure 1 The internet data real-time downloading system architecture based on the configuration foreground in the embodiment of the present application;

[0099] Figure 2 The downloading task configuration flowchart of the internet data real-time downloading system based on the configuration foreground in the embodiment of the present application;

[0100] Figure 3 The interaction schematic diagram of JSON transmission and background downloading module of the internet data real-time downloading system based on the configuration foreground in the embodiment of the present application. DETAILED DESCRIPTION

[0101] As Figure 1 As shown, the embodiment based on the configuration of the foreground of the Internet data real-time download system includes: foreground configuration module, JSON parameter passing module, background download module, task scheduling module and log and monitoring module; wherein, the foreground configuration module is used to provide a graphical interface, and the user configures the parameters of the download task through the interface; the JSON parameter passing module is used to convert the parameters configured by the foreground configuration module into JSON format and pass them to the background download module; the background download module is used to integrate various download tools including lftp, wget and crawler; the task scheduling module is used to manage the concurrent execution of multiple download tasks; the log and monitoring module is used to realize real-time task monitoring and log recording functions.

[0102] The implementation method of the embodiment based on the configuration of the foreground of the Internet data real-time download system is as follows:

[0103] Step P1: build and implement the interface front-end framework based on Vue.js;

[0104] In the embodiment, step P1 specifically includes the following steps:

[0105] Step P101: use the Vue.js framework to build the interface front-end, and use the encryption algorithm for page login; in order to ensure the secure transmission of private information during user login, the embodiment integrates the AES (Advanced Encryption Standard) encryption algorithm in the front-end, and uses the ECB (Electronic Codebook) working mode and the PKCS5Padding padding method to encrypt the password information input by the user. The implementation process is as follows:

[0106] User input information collection: the user inputs the username and password in the front-end login page.

[0107] Preparation before encryption: the system presets a symmetric encryption key in the front-end. The password is encrypted using the AES algorithm, ECB mode and PKCS5Padding padding method. Before encryption, the plaintext password is converted to UTF-8 encoding format, and the key length is 128 bits, which meets the AES standard requirements.

[0108] Encrypted result processing: the encrypted ciphertext is processed through Base64 encoding to facilitate transmission through the HTTP protocol.

[0109] Transmission and verification: the front-end submits the encrypted password (and username) to the back-end through a POST request.

[0110] Step P102: The user page is built in a text interface input manner to construct the front-end interaction module; the front-end interaction module is designed in the front-end page to realize multiple input fields in the form of a text interface, specifically including but not limited to the following parameter interaction modules: demand name input box: the user can customize the name of the current configuration task through the input box, which is used for task management and identification. This field is implemented through <input type＝"text"> implementation and supports input length and character type checking to avoid errors in subsequent processes caused by illegal characters. Data source URL input box: the user can input the access address (URL) of the target data, and the system checks the URL format legality through a regular expression and can optionally provide a historical address drop-down prompt to improve configuration efficiency. Demand frequency selection box: set the data access frequency through a drop-down selection or a free text box, for example: daily, hourly, every 10 minutes, etc. The field supports enumeration values and custom time expression (cron expression) input, and displays format examples in the page prompt field. Storage path input box: the user inputs the storage path of the target data on the local server, for example: / nasC / oper / data / DID_20240429_001 / data / . The front end uses path format prompts and basic permission detection to ensure that the path is legal and writable.

[0111] Each of the above input modules uses the data binding mechanism (such as v-model) in the Vue.js framework to realize real-time data synchronization and bidirectional binding, ensuring that what the user sees is what he gets. In addition, to enhance the user interaction experience, all fields are combined with the following design features:

[0112] Form verification mechanism: use the built-in form verification features of component libraries such as Vuetify or Element-UI for format checking and non-empty verification; error prompt and guidance: when the user inputs is illegal or missing, the user is prompted to correct through a red border, icon prompt, and guidance statement; responsive design: the page layout is adaptive to multiple terminal resolutions, ensuring normal use on desktops and mobile devices; data caching and recovery function: the front end introduces the LocalStorage mechanism to automatically cache user input content, preventing information loss due to unexpected browser closure or refresh.

[0113] Step P103: User page display, the front end uses the HTTP request data interface, and uses the virtual DOM technology to realize efficient update rendering of the existing data categories and the data categories being downloaded, uses the responsive system to automatically track data changes, realizes binding events and data through embedding JavaScript expressions and instructions in HTML to realize real-time view data synchronization of the page, and uses the component technology to improve code reusability; When the HTTP request data interface, the front end sends a GET request to the backend data interface through an HTTP client library such as Axios to obtain the following two types of core data: existing data categories: a list of various data types that have been completed in the system. The current data types are captured or synchronized; the data categories being downloaded: the data capture tasks currently being executed, including task name, progress, start time, and other attributes.

[0114] Request format example:

[0115]

[0116] Step P104: Use Vue CLI to realize hot reloading, and use Vue Router to realize navigation and data transmission between pages; Based on the Vue.js front-end framework, further use the Vue official development tool chain Vue CLI and the routing management tool Vue Router to realize the hot reloading function in the development process and the navigation and data transmission mechanism between different function pages of the system, to improve the development efficiency and system interactivity.

[0117] Use Vue CLI to realize hot reloading: To improve the efficiency of front-end development and debugging, the embodiment uses Vue CLI (Vue.js official command line tool) to initialize and manage the front-end project. The development server based on Webpack built into Vue CLI can automatically monitor source code file changes and support the Hot Module Replacement (HMR) mechanism. Its main working process is as follows:

[0118] Start the development service command: npm run serve. When the developer modifies the.vue component, JavaScript script, or style sheet file (.css / .scss) locally, Vue CLI automatically detects the changes and triggers hot reloading; the hot reloading process only re-renders the modified component, without refreshing the entire page, preserving the current page state and input content, thereby greatly improving the debugging efficiency. This hot reloading mechanism is based on the HMR plugin of Webpack, and the specific configuration is encapsulated and automatically integrated by Vue CLI, and the developer does not need to perform additional operations.

[0119] Page navigation and data transmission are implemented by using Vue Router: In order to realize the non-refresh jump and modular view organization between front-end pages, the system integrates the official routing management library Vue Router provided by Vue, and adopts the "single-page application (SPA)” architecture to build the user interface. The working implementation includes:

[0120] (1) Route configuration method: The system uniformly defines the path and component mapping relationship of each functional page in router / index.js, for example:

[0121]

[0122] The above configuration can realize the navigation of the "home page”, "automated collection”, "data resource management” and other pages.

[0123] (2) Navigation control: In the page, the built-in <router-link>The component or JavaScript control approach implements the jump, for example: <router-link to=" / config"> jumps to the configuration page< / router-link>

[0124] (3) Data transmission between pages: Data can be carried through route parameters or query parameters when jumping between pages. For example, when jumping to the status page, the task ID is transmitted:

[0125] this.$router.push({path: ' / status', query: { taskId: '123'}});

[0126] The target page obtains the transmitted data through this.$route.query.taskId, thereby realizing cross-page state transmission and sharing. In addition, more complex cross-component data synchronization and global state maintenance can also be realized by combining the Vuex state management library.

[0127] Step P2: Build and implement the interface backend framework;

[0128] In this embodiment, step P2 specifically includes the following steps:

[0129] Step P201: The front end transmits the download demand information to the back end in the HTTP mode; on the basis of completing the front-end page configuration and interaction function, the mechanism design of transmitting the download demand information from the front end to the back end is further implemented, so as to ensure that the data transmission process is safe, stable, standardized and easy to expand. The core technologies involved in this step include: HTTP protocol, RESTful interface specification, firewall, Nginx reverse proxy, Netty high-performance network communication framework and the like. The specific implementation process includes:

[0130] (1) Demand information construction and HTTP transmission: When the user completes the demand configuration on the front-end interface (such as demand name, data URL, frequency, format, storage path, etc.), the system encapsulates all fields into a JSON format request body object and transmits it to the back-end interface through HTTP POST request. Request example:

[0131] The front end initiates the request through Axios or Fetch API and defines the resource path and verb semantics (such as POST for creating resources) based on the RESTful interface design style.

[0132] (2) RESTful interface design specification: The back-end interface is designed in RESTful style, with clear and hierarchical interface URL semantics, such as:

[0133] POST / api / task / create: Create a download task;

[0134] GET / api / task / status / {taskId}: Get task execution status;

[0135] DELETE / api / task / {taskId}: Delete a specified task.

[0136] The interface transmission uses JSON format to ensure cross-language and cross-platform compatibility.

[0137] (3) Network communication and high concurrency support: The back-end service is built based on the Netty framework, which is a high-performance, asynchronous event-driven network communication framework suitable for handling a large number of concurrent connections and long connection scenarios. In this system, Netty serves as the underlying communication component of microservices, responsible for receiving front-end requests, forwarding them to core business modules, and returning response results.

[0138] (4) Transmission security mechanism: To ensure data security and service availability during transmission, the system deploys the following security policies and gateway configurations: Firewall configuration: whitelist limit on exposed external ports, allowing only legitimate source IP access to prevent malicious scanning and attacks; Nginx as a reverse proxy server: deployed between the front-end and back-end, responsible for SSL termination, load balancing, and request forwarding. Configure HTTPS protocol and deploy SSL certificate to achieve transmission encryption and identity authentication; Request throttling and exception monitoring: set interface access frequency threshold in Nginx or back-end service to prevent malicious interface brushing. Abnormal requests are recorded and alarmed through the log system.

[0139] (5) Data consistency and response mechanism: After the backend receives the demand information, it checks the parameter content (such as URL legality, storage path writability, frequency expression analysis), and if the verification is passed, it generates a task record and returns the task ID and confirmation information. Response format example:

[0140]

[0141] After the front end receives the response, it automatically switches to the "task status" page and loads the task details, completing a closed-loop interaction.

[0142] Step P202: The backend uses Spring Boot and XXL-JOB distributed task scheduling center to manage the scheduling and concurrent execution of multiple download tasks. The submitted download data document details are stored in the MySQL database. The data Spring Boot dynamically configures xxljob timing tasks to call the python program for data download at regular intervals, supporting the configuration of the number of concurrent and retry times;

[0143] After the front-end transmits the download demand information to the back-end system (step P201), the back-end system based on the Spring Boot framework and the XXL-JOB distributed task scheduling platform realizes the unified management and automatic scheduling execution of multiple download tasks, and saves the task metadata information to the MySQL database to support task configuration persistence and parameter dynamic adjustment. The main implementation process and technical structure are as follows:

[0144] (1) Download task data persistence to MySQL: The backend develops a task management service based on the Spring Boot framework. After receiving the front-end request, it first performs format verification and semantic analysis on the submitted download demand parameters, and writes relevant fields into the task table in the database, including but not limited to:

[0145]

[0146]

[0147] (2) Dynamic configuration of timing tasks (Spring Boot + MySQL + XXL-JOB): To realize the timing scheduling and parameterized control of tasks, the system uses XXL-JOB as the scheduling center, and integrates XXL-JOB executor (Executor) on the Spring Boot service side to dynamically submit tasks to the XXL-JOB console in the form of registration. The system realizes scheduling configuration through the following process:

[0148] Spring Boot periodically queries the database for active download tasks; encapsulates task information into XxlJobInfo objects and registers tasks via the scheduling platform API; sets the scheduling frequency (e.g., CRON expression), execution parameters (e.g., task ID, file URL, number of retries); configures execution routing strategies (e.g., round-robin by machine, grouped by task) and failure handling strategies (retries, alarms); and automatically creates corresponding task execution records and log tracing links. Example scheduling registration process code snippet:

[0149] XxlJobInfo jobInfo=new XxlJobInfo();

[0150] jobInfo.setJobDesc("Scheduled download task: "+taskName);

[0151] jobInfo.setAuthor("system");

[0152] jobInfo.setScheduleConf("0 0***?"); / / Execute once per hour

[0153] jobInfo.setExecutorRouteStrategy("ROUND");

[0154] jobInfo.setExecutorHandler("downloadExecutor");

[0155] jobInfo.setExecutorParam(taskId);

[0156] jobInfo.setMisfireStrategy("DO_NOTHING");

[0157] xxlJobService.addJob(jobInfo);

[0158] (3) Task scheduling calls the Python download program: When the task is scheduled to execute, the Spring Boot executor receives the call request from the scheduling center, loads the corresponding task information from the database according to the passed task ID or parameters, and calls the pre-set Python script on the backend to start the data download process. After receiving the parameters, the Python script performs the following operations:

[0159] Download data from the specified URL; determine the parsing method according to the file format; save the file to the configured path; support automatic retry logic (re-execute after failure); multiple tasks are executed concurrently, controlled by the Python script itself or the scheduler thread pool. Python script startup method:

[0160] String command = "python3 download.py --url=" + url + "--path=" + savePath + "--retry=" + retryCount;

[0161] Runtime.getRuntime().exec(command);

[0162] (4) Concurrent control and retry mechanism support: the system supports configuring the maximum number of concurrent tasks (such as running a maximum of 5 tasks at the same time), and scheduling can be achieved through thread pool or queue throttling. When a task fails to download, according to the retry_count field, a maximum of a specified number of retries are performed, and the retry interval and strategy can be dynamically adjusted through configuration items to ensure stable operation of the task.

[0163] (5) Schedule task state management: the running record, log information, execution time, and results of each scheduling task are managed by the XXL-JOB platform, and can be visually monitored and troubleshooting through the Web console. Task status can be controlled bidirectionally through database fields and XXL-JOB console, supporting operations such as enabling, pausing, deleting, etc.

[0164] Step P203: Filter the real-time download data table into data categories and all data information tables through Spring Boot, and pass the data information to the page. After completing the scheduling and execution of the download task, further realize the visual management of the real-time download data results. The backend system realizes data extraction, classification, and formatting processing based on Spring Boot, divides the data into general categories and specific data information tables, and responds to front-end requests through HTTP interfaces, supporting real-time display and interaction on the page side. The implementation process includes:

[0165] (1) Real-time download data into database: after the download task is executed, the Python script writes the download record of each data into the download_log table in the MySQL database, which is used for subsequent query and display. The table structure example includes the following fields:

[0166]

[0167] Where: category represents the data belongs to the large category (such as satellite data, radar data, reanalysis data, etc.); sub_category represents the specific data type or site; file_name and file_size record file details; status indicates the download status, such as success, failure, retry, etc.

[0168] (2) Backend classification aggregation logic: Spring Boot control layer (Controller) extracts all downloaded data from the download_log table by calling the Service layer business logic, and groups statistics according to the category field to generate a "data total category" list, and further constructs detailed tables according to sub-category, file, etc. The core processing logic is as follows:

[0169]

[0170] Each category of data is encapsulated as a JSON object again, including sub-category name, file name, size, download time, etc. Form a structured table data. The data format returned to the front end is as follows:

[0171]

[0172]

[0173] (3) Backend interface and front-end interaction: the backend uses RESTful style to define the interface, for example:

[0174] GET / api / data / summary

[0175] The front end requests this interface through Axios and other methods to obtain the classification result JSON data, and renders it into a "data category + data detailed table" structured display list in the page. Users can click on a certain category of data to expand its specific download file records. In order to support real-time refresh, data update synchronization can be achieved through front-end polling or back-end WebSocket push mechanism.

[0176] Step P3: Based on the front-end configuration parameters, the parameters of the back-end data download program are passed in;

[0177] In this embodiment, step P3 specifically includes the following steps:

[0178] Step P301: Convert configuration information into JSON format parameters, generate corresponding JSON configuration files for the relevant parameters configured by the graphical page, including data source address, download frequency, file type, storage path and retry strategy; For example:

[0179]

[0180]

[0181] Step P302: The backend parses the JSON format parameter, selects a download tool according to the parsed parameter, transmits the parameter into the tool field in the JSON parameter, and executes a corresponding data acquisition task; the Python program dynamically reads and parses the JSON configuration file, selects a suitable download module (such as lftp, wget or a custom collector) according to the parameter in the configuration file, and the background download module parses the JSON parameter and calls a corresponding download tool to execute a download task. For example, if the tool is lftp, the command line tool is parsed and called to execute the lftp command;

[0182] Step P303: The Python program reads and parses the JSON file generated by the front-end page configuration, calculates the specific date according to the preset specific date identifier and the time interval parameter in the configuration file, and transmits the value to the corresponding position in the JSON file to realize the automatic assignment of the date parameter in the data acquisition task; for example:

[0183]

[0184] On March 17, 2025, it will be automatically identified as the following JSON string:

[0185]

[0186]

[0187] Step P304: The program automatically adds a corresponding prefix to each file name to be processed according to the user input preset prefix matching symbol on the front-end page, and the added prefix contains timestamp information; at the same time, the program records the original file name, file size and download time of each file to the database table for information comparison of the file in subsequent operations, thereby effectively avoiding repeated downloading. For example:

[0188]

[0189]

[0190] Z_NAFP_C_FENS_(18-bit timestamp)_P_ will be added in front of each file name.

[0191] Step P4: Automatically download Internet data;

[0192] In this embodiment, step P4 specifically includes the following steps:

[0193] Step P401: Establish an automated proxy pool to dynamically allocate and switch the used proxy IP; through the construction of the automated proxy pool, proxy IP is collected from multiple third-party proxy interfaces or internal proxy resources at regular intervals, and a concurrent detection mechanism is used to verify the availability and response speed of the IP in real time. Effective proxy IP is stored in the proxy pool according to the weight or polling strategy;

[0194] Step P402: Use proxy IP to request when executing script configuration, switch proxy IP to reduce the single IP ban caused by timed requests; when executing the download script, the system randomly or according to the strategy selects a proxy IP from the proxy pool before initiating each HTTP / FTP request, and configures it into the request parameter; if a timeout or error response is encountered during the request process, the system will automatically switch the proxy IP and re-initiate the request to avoid the single IP being banned due to frequent access;

[0195] Step P403: Establish a UA library containing multiple browsers and devices, the UA library contains browser type, version number and operating system information, and set different UA to access target website; randomly select UA to simulate multiple device and browser environments each time; the system pre-constructs a user agent (User-Agent) library containing multiple browser types, version numbers and operating system information. Before each request, an UA string is extracted from the library through a random algorithm and filled into the request header to simulate the access behavior of different devices and browsers;

[0196] Step P404: Select the download tool according to the tool field in the JSON parameter, and the download module matches the download information of the website, date and file name based on the selected download tool; adopt the breakpoint resume technology combined with the data verification mechanism with.tmp suffix to ensure the integrity of the downloaded content, support multiple protocols including FTP, SFTP and HTTP; During the download process, the system records the log in real time and returns the status information to the front end; that is, the system automatically selects the corresponding download tool (such as lftp, wget, etc.) according to the "tool" field in the input JSON configuration parameter, and parses the target website, date and file name and other key information to accurately match the download task. During the download process, the breakpoint resume technology is adopted: first write the data into a temporary file with ".tmp" suffix, and then rename it to the formal file after data integrity verification. At the same time, this module supports multiple protocols such as FTP, SFTP and HTTP, and records the log and status feedback to the front-end interface in real time during the download process;

[0197] Step P405: Screen the problem data by using the file name comparison and / or file size comparison method, and automatically retry by internal loop instruction or external scheduling; For the case of download failure caused by network delay or other abnormalities, the system checks the download result by comparing file name and file size, etc. after each download is completed. Once the data is found to be incomplete or abnormal, the system will mark the problem file and trigger internal loop retry or automatic retry download operation with the help of external scheduling module until the download task is successful or reaches the upper limit of the set retry number;

[0198] Step P406: Identify the storage path collected in step P101, and store the downloaded data to the specified location. The system has obtained the target storage path information in the initial collection stage (step P101), and then moves or copies the downloaded data from the temporary directory to the specified location by parsing the path information after the download is completed. For example: When the system receives a download task, it first selects the proxy IP "192.168.1.100" from the proxy pool, randomly selects the UA "Mozilla / 5.0 (Windows NT 10.0; Win64; x64) …", and calls the "wget" module according to the tool field in the JSON parameter. During the download process, the ".tmp" suffix file is generated to verify the data integrity; if the network fluctuation causes the download to be interrupted, the system automatically retries the download operation by comparing the file name and size. Finally, the file is saved to the predetermined storage path, and the download log and status information are transmitted back to the front-end interface in real time.

[0199] Step P5: Monitor the integrity and timeliness of data download

[0200] In this embodiment, step P5 specifically includes the following steps:

[0201] Step P501: The system uses the mapping relationship between the identification information embedded in the file name and the predefined delay parameter to build a mapping table or data structure, which realizes the corresponding management between the file name and the file delay time; Based on this mapping relationship, the system uses a preset algorithm to accurately calculate the scheduled arrival time of each file; The system parses the file name according to the predefined naming rule before the file is generated or downloaded, and extracts the delay identification information contained therein; At the same time, a hash table or dictionary is constructed in the memory, which maps various file name modes to the corresponding delay time parameters (which can be dynamically loaded through configuration file or database);

[0202] Step P502: After each file download operation is completed, the number of downloaded files and the corresponding data volume are automatically counted through event triggering or callback functions (trigger the statistics module through events or callback mechanisms, extract the file count and size of each file from the download module, and accumulate the total download data volume), and these statistical information is written into the database in real time using an efficient transaction processing mechanism; the database uses a specially designed table structure, including timestamp, file count, and data volume fields;

[0203] Step P503: Use Spring Boot timing to push download detailed record logs to the DI interface using HTTP messages to record events, and record detailed information of data download to a table in MySQL at regular intervals. By frequently querying the real-time update download record table, determine whether the data is abnormal and push an alarm, and display a red floating on the page.

[0204] Through Spring Boot, the download task execution log is periodically collected, analyzed and processed, and the download event data is pushed to the external DI (Data Integration) interface in real time by means of HTTP protocol to complete the task status tracking and event archiving. At the same time, combined with the high-frequency polling or timing query logic of the download log table, possible data abnormalities (such as download failure, repetition, timeout, etc.) are identified, and visual alarm prompts on the page are triggered. The implementation process includes:

[0205] (1) Download record logs are pushed to the DI interface at regular intervals: the system uses the @Scheduled timing task mechanism of Spring Boot to execute the download log collection and event reporting operation every 5 minutes: query the log table data_download_record in MySQL that records the download details; filter out the newly added or state changed records within the time window; construct a JSON message body in a unified format, including task ID, data source, time, status, etc. fields; push the JSON message to the predefined DI interface address through HTTP POST request. Timing task configuration example:

[0206]

[0207] (2) Write download details to the log archive table: in order to support post-operation analysis and data tracing, before each download record is pushed, the system will simultaneously insert the corresponding data record into an archive table di_push_record, saving the complete data structure. SQL table structure:

[0208]

[0209] Mark the push state when writing to avoid duplicate pushing and facilitate subsequent checking:

[0210] log.setPushedToDi(true);

[0211] downloadHistoryRepository.save(log);

[0212] (3) High-frequency polling detects abnormal download records: The system sets another timing task that executes every minute to monitor the data_download_record and di_push_record tables to identify the following possible abnormalities: the number of download failures exceeds the threshold; a task is not completed within a certain time; the size of the downloaded file is 0; the same task is repeatedly downloaded; it is not run at the scheduled time, etc.

[0213]

[0214] (4) Trigger alarm and page-side highlight prompt: Once an abnormal record is found, the system calls the internal alarm service, writes the abnormal task information into the alert_log table, and notifies the front-end system to obtain the latest abnormal event list through polling or WebSocket:

[0215] The alarm information includes: task name, abnormal description, first abnormal time, status, etc.; after receiving the alarm information, the page side marks the corresponding task bar as "red floating" state; users can view detailed error information, download logs, manually retry, etc. by clicking. The front-end page uses the responsive data binding mechanism of Vue to automatically display the task bar as red, flashing or highlighted, realizing instant alarm and visual reminder of the fault task.

[0216] Obviously, the above embodiments are only examples for clear illustration, and not limitation of the embodiments. For ordinary skilled persons in the art, other different forms of changes or variations can be made on the basis of the above description. Here, it is not necessary and impossible to enumerate all the embodiments. The obvious changes or variations derived therefrom are still within the protection scope of the claims of the present patent application.

Claims

1. A method for real-time download of internet data based on a configured foreground, characterized in that, Comprising the following steps: Step P1: build and implement a Vue.js-based interface front-end framework; Step P2: build and implement an interface back-end framework; Step P3: pass parameters to the back-end data download program based on front-end configuration parameters; Step P4: automatically download Internet data; Step P5: monitor the completeness and timeliness of data download.

2. The method for real-time download of internet data based on the configured foreground according to claim 1, characterized in that, Step P1 comprises the following steps: Step P101: use the Vue.js framework to build the interface front-end, and use an encryption algorithm for page login; Step P102: build the front-end interaction module in the user page in the form of a text interface; Step P103: the user page is displayed, the front-end utilizes HTTP request data interfaces, and utilizes virtual DOM technology to efficiently update and render existing data categories and data categories being downloaded, utilizes a responsive system to automatically track data changes, realizes real-time view data synchronization through binding events and data by embedding JavaScript expressions and directives in HTML, and utilizes componentization technology to improve code reusability; Step P104: use Vue CLI to realize hot reloading, and use Vue Router to realize navigation and data transmission between pages.

3. The method of claim 2, wherein the method further comprises: In step P101, the implementation process of the encryption algorithm when logging in to the page is as follows: Step P101-1, user information collection: the user inputs the username and password in the front-end login page; Step P101-2, preparation before encryption: the system preloads a symmetric encryption key in the front-end; the password is encrypted using the AES algorithm in ECB mode and the PKCS5Padding padding method; the plaintext password is converted to UTF-8 encoding format before encryption, and the key length is 128 bits; Step P101-3, encryption result processing: the encrypted ciphertext is processed through Base64 encoding; Step P101-4, transmission and verification: the front-end submits the encrypted password and username to the back-end through a POST request; In step P102, the front-end interaction module includes a requirement name input box, a data source URL input box, a requirement frequency selection box, a file format selection box, and a storage path input box; the front-end interaction module realizes real-time data synchronization and bidirectional binding using the data binding mechanism in the Vue.js framework, and all fields are designed in combination with form validation mechanisms, error prompts and guidance, responsive design, and data caching and recovery functions; In step P103, when the HTTP request data interface, the front-end sends a GET request to the back-end data interface through the Axios client library to obtain existing data categories and data categories being downloaded; In step P104, the working process of using Vue CLI to realize hot reloading is as follows: start the development service command, when the developer modifies the Vue component, JavaScript script, or style sheet file locally, Vue CLI automatically detects the changes and triggers hot reloading; the reloading mechanism is realized based on the HMR plugin of Webpack, and the specific configuration is encapsulated and automatically integrated by Vue CLI; The working process of realizing the navigation and data transmission between pages by using Vue Router is as follows: uniformly defining the path and component mapping relationship of each function page, realizing the page navigation of "home page", "automatic collection" and "data resource management"; realizing the data transmission between pages by using the built-in <router-link>The component or JavaScript control mode realizes the jump, and data is carried by a route parameter or a Query parameter during the page jump.

4. The method of claim 1, wherein the method further comprises: Step P2 comprises the following steps: Step P201: The front end transmits download requirement information to the back end in an HTTP mode; Step P202: The back end uses Spring Boot and an XXL-JOB distributed task scheduling center to link, manage the scheduling and concurrent execution of multiple download tasks, stores the submitted download data document details into a MySQL database, uses the data of the MySQL database to dynamically configure the xxljob timing task of Spring Boot, calls the python program for data download at a timing, and supports the configuration of the number of concurrent and the number of retries; Step P203: The real-time download data table is filtered into a data total category and all data information tables by Spring Boot, and the data information is transmitted to the page.

5. The method of claim 4, wherein the method further comprises: In step P201, the front end transmits download requirement information to the back end by using a request mode of HTTP POST and initiating a request by using Axios or Fetch API, a back end interface is designed in a RESTful style, and an interface transmission is uniformly in a JSON format; the back end service is built based on a Netty framework, is responsible for receiving a front end request, forwarding to a core business module, and returning a response result, and performs a firewall configuration in a data transmission process; Nginx is deployed as a reverse proxy server between the front end and the back end, is responsible for an SSL terminal, load balancing and request forwarding, configures an HTTPS protocol and deploys an SSL certificate, realizes transmission encryption and identity authentication, an interface access frequency threshold is set in Nginx or the back end service, the back end receives requirement information, verifies parameter content, generates a task record and returns a task ID and confirmation information if the verification is passed, and the front end receives a response, automatically switches to a "task state" page and loads task details, and completes a closed-loop interaction; The implementation process of step P202 is as follows: Step P202-1, the back end develops a task management service based on a Spring Boot framework, receives a front end request, performs format verification and semantic analysis on submitted download requirement parameters, and writes related fields into a task table in a MySQL database; Step P202-2, XXL-JOB is used as a scheduling center, a Spring Boot service end integrates an XXL-JOB executor, and tasks in the task table are dynamically submitted to an XXL-JOB console in a registration form; Step P202-3, the Spring Boot executor receives a scheduling center calling request, loads corresponding task information from the database according to the input task ID or parameter, and calls a preset Python script of the back end to start a data download process. Step P202-4, configure the maximum number of concurrent tasks, implement through thread pool or queue throttling; when a task fails to download, perform a maximum of a specified number of retries according to the retry_count field, and the retry interval can be dynamically adjusted through configuration items; the running record, log information, execution time, and results of each scheduled task are managed by the XXL-JOB platform, and visual monitoring and troubleshooting are performed through the Web console; the task status can be controlled bidirectionally through the database field and the XXL-JOB console; In step P203, after the download task is completed, the Python script writes the download record of each data into the download_log table in the MySQL database; the Spring Boot control layer extracts all downloaded data from the download_log table by calling the Service layer business logic, and groups and counts them according to the category field to generate a "data total category" list, and further constructs a detailed table according to the subcategory and file field; the backend uses RESTful style to define the interface, and the frontend requests the interface through Axios to obtain the classification result JSON data and renders it into a structured display list of "data category + data detailed table" in the page.

6. The method of claim 1, wherein the method further comprises: Step P3 includes the following steps: Step P301: Convert configuration information to JSON format parameters, generate corresponding JSON configuration files for related parameters configured on the graphical page, including data source address, download frequency, file type, storage path, and retry strategy; Step P302: The backend parses the JSON format parameters, selects the download tool according to the parsed parameters, and passes the tool field in the JSON parameters to execute the corresponding data acquisition task; Step P303: Use the Python program to read and parse the JSON file generated by the front-end configuration page, calculate the specific date according to the preset date identifier and its time interval parameter in the configuration file, and pass the value to the corresponding position in the JSON file to realize the automatic assignment of the date parameter in the data collection task; Step P304: The program automatically adds the corresponding prefix to each file name to be processed according to the user input preset prefix matcher on the front-end page, and the added prefix contains timestamp information; at the same time, the program records the original file name, file size, and download time of each file to the database table.

7. The method of claim 1, wherein the method further comprises: Step P4 includes the following steps: Step P401: Establish an automated proxy pool to dynamically allocate and switch the used proxy IP; Step P402: Use proxy IP to request when executing script configuration, switch proxy IP to reduce the risk of single IP ban due to timed requests; Step P403: Establish a UA library containing multiple browsers and devices, the UA library contains browser type, version number, and operating system information, set different UA to access target websites; randomly select UA for each request to simulate multiple device and browser environments; Step P404: Select a download tool according to the tool field in the JSON parameter, and the download module matches the download information of the website, date and file name based on the selected download tool; adopts the breakpoint resume technology combined with the data verification mechanism with.tmp suffix to ensure the integrity of the downloaded content, supports multiple protocols including FTP, SFTP and HTTP; During the download process, the system records logs in real time and returns status information to the front end; Step P405: Use the file name comparison and / or file size comparison method to screen problem data, and automatically retry through internal program loop instructions or external scheduling; Step P406: Identify the storage path collected in step P101, and store the downloaded data in the specified location.

8. The method of claim 1, wherein the method further comprises: Step P5 includes the following steps: Step P501: The system uses the mapping relationship between the identification information embedded in the file name and the predefined delay parameter to build a mapping table or data structure, which realizes the corresponding management between the file name and the file delay time; Based on this mapping relationship, the system uses a preset algorithm to accurately calculate the scheduled arrival time of each file; Step P502: After each file download operation is completed, the number of downloaded files and the corresponding data volume are automatically counted through event triggering or callback functions, and these statistical information is written into the database in real time using an efficient transaction processing mechanism; This database uses a specially designed table structure, which contains timestamp, file count and data volume fields; Step P503: Use Spring Boot timing to push the download detailed record logs to the DI interface to realize event recording, and record the detailed information of data download to a table in MySQL, and update the download record table in real time by frequent query to determine whether the data is abnormal and push the alarm and display the red floating on the page.

9. The method of claim 1, wherein the method further comprises: Step P1 includes the following steps: Step P101: Use the Vue.js framework to build the interface front end, and use algorithm encryption to ensure user privacy and security when logging in; The implementation process of the encryption algorithm when logging in is as follows: Step P101-1, user information collection: the user inputs the username and password in the front-end login page; Step P101-2, encryption preparation: the system preloads symmetric encryption keys in the front end; Use AES algorithm, adopt ECB mode and PKCS5Padding padding method to encrypt the password; Before encryption, convert the plaintext password to UTF-8 encoding format, and the key length is 128 bits; Step P101-3, encryption result processing: the encrypted ciphertext is processed through Base64 encoding; Step P101-4, transmission and verification: the front end submits the encrypted password and username to the back end through POST request; Step P102: the user page is constructed in a text interface input manner to build a front-end interaction module; the front-end interaction module includes a requirement name input box, a data source URL input box, a requirement frequency selection box, a file format selection box, and a storage path input box; the front-end interaction module uses the data binding mechanism in the Vue.js framework to realize real-time data synchronization and bidirectional binding, and all fields are designed in combination with a form verification mechanism, error prompts and guidance, responsive design, and data caching and recovery functions; Step P103: the user page is displayed, the front-end uses HTTP request data interfaces, and uses virtual DOM technology to efficiently update and render existing data categories and downloading data categories, uses a responsive system to automatically track data changes, realizes page real-time view data synchronization by binding events and data through embedding JavaScript expressions and instructions in HTML, and uses component technology to improve code reusability; When the HTTP request data interface is used, the front-end sends a GET request to the back-end data interface through the Axios client library to obtain existing data categories and downloading data categories; Step P104: hot reloading is realized by using Vue CLI, and navigation and data transmission between pages are realized by using Vue Router; the working process of realizing hot reloading by using Vue CLI is as follows: when the developer modifies a.vue component, a JavaScript script, or a style sheet file locally, the Vue CLI automatically detects the changes and triggers hot reloading; the reloading mechanism is realized based on the HMR plug-in of Webpack, and the specific configuration is encapsulated and automatically integrated by the Vue CLI; The working process of realizing the navigation and data transmission between pages by using Vue Router is as follows: uniformly defining the path and component mapping relationship of each function page, realizing the page navigation of "home page", "automatic collection" and "data resource management"; realizing the data transmission between pages by using the built-in <router-link>The component or JavaScript control mode realizes jump, and data is carried through route parameters or query parameters during page jump; Step P2 includes the following steps: Step P201: the front-end transmits the download requirement details to the back-end in an HTTP manner; the front-end initiates a request to transmit the download requirement information to the back-end by using the HTTP POST request mode, Axios or Fetch API, the back-end interface is designed in a RESTful style, and the interface transmission uniformly uses the JSON format; the back-end service is built based on the Netty framework, is responsible for receiving front-end requests, forwarding to the core business module, and returning response results; and firewall configuration is performed during data transmission; Nginx is deployed as a reverse proxy server between the front-end and the back-end, is responsible for SSL termination, load balancing and request forwarding, configures the HTTPS protocol and deploys the SSL certificate to realize transmission encryption and identity authentication; an interface access frequency threshold is set in Nginx or the back-end service, the back-end receives the requirement information, checks the parameter content, and if the verification is passed, generates a task record and returns a task ID and confirmation information; after receiving the response, the front-end automatically switches to the "task status" page and loads the task details, completing a closed-loop interaction; Step P202: The backend uses Spring Boot and XXL-JOB distributed task scheduling center for linkage to manage the scheduling and concurrent execution of multiple download tasks. The submitted download data document details are stored in the MySQL database. The data Spring Boot dynamically configures xxljob timing tasks using the MySQL database. The python program is called periodically to download data. The number of concurrent and retry times are supported. The implementation process of step P202 is as follows: Step P202-1, the backend develops a task management service based on the Spring Boot framework. After receiving the front-end request, the submitted download requirement parameters are format checked and semantically parsed. The relevant fields are written into the task table in the MySQL database. Step P202-2, use XXL-JOB as the scheduling center. The Spring Boot server integrates the XXL-JOB executor to dynamically submit the tasks in the task table to the XXL-JOB console in the form of registration. Step P202-3, the Spring Boot executor receives the scheduling center call request. According to the task ID or parameters passed in, the corresponding task information is loaded from the database, and the Python script preset by the backend is called to start the data download process. Step P202-4, configure the maximum number of concurrent tasks through thread pool or queue flow limiting. When a task fails to download, according to the retry_count field, execute a maximum of a specified number of retries. The retry interval and strategy can be dynamically adjusted through configuration items. The running record, log information, execution time and result of each scheduling task are managed by the XXL-JOB platform. Visual monitoring and troubleshooting are performed through the Web console. The task status can be controlled bidirectionally through the database field and the XXL-JOB console. Step P203: Filter the real-time download data table into data total categories and all data information tables through Spring Boot. Pass the data information to the page. After the download task is completed, the Python script writes each data download record into the download_log table in the MySQL database. The Spring Boot control layer extracts all downloaded data from the download_log table by calling the Service layer business logic, groups and counts them according to the category field, generates a "data total category" list, and further constructs detailed tables according to the subcategory and file field. The backend uses RESTful style to define the interface. The front end requests the interface through Axios to obtain the classification result JSON data and renders it into a structured display list of "data category + data detailed table" in the page. Step P3 includes the following steps: Step P301: Convert the configuration information into JSON format parameters, and generate a corresponding JSON configuration file for the relevant parameters configured by the graphical page, including data source address, download frequency, file type, storage path, and retry strategy; Step P302: The backend parses the JSON format parameters, selects a download tool according to the parsed parameters, passes the parameters into the tool field in the JSON parameters, and executes the corresponding data acquisition task; Step P303: Use the Python program to read and parse the JSON file generated by the front-end configuration, calculate the specific date according to the pre-set specific date identifier and its time interval parameter in the configuration file, and pass the value to the corresponding position in the JSON file to realize the automatic assignment of the date parameter in the data acquisition task; Step P304: The program automatically adds a corresponding prefix to each file name to be processed according to the pre-set prefix matching symbol input by the user on the front-end page, and the added prefix contains timestamp information; at the same time, the program records the original file name, file size, and download time of each file to the database table; Step P4 includes the following steps: Step P401: Establish an automatic proxy pool to dynamically allocate and switch the used proxy IP; Step P402: Use proxy IP to request when executing script configuration, switch proxy IP to reduce the single IP ban caused by timed requests; Step P403: Establish a UA library containing multiple browsers and devices, the UA library contains browser type, version number, and operating system information, and set different UAs to access target websites; randomly select a UA for each request to simulate multiple device and browser environments; Step P404: Select a download tool according to the tool field in the JSON parameters, and the download module matches the download information of the website, date, and file name based on the selected download tool; use the breakpoint resume technology combined with the.tmp suffix data verification mechanism to ensure the integrity of the downloaded content, support multiple protocols including FTP, SFTP, and HTTP; During the download process, the system records logs in real time and returns status information to the front end; Step P405: Use the file name comparison and / or file size comparison method to screen problem data, and automatically retry through internal loop instructions or external scheduling; Step P406: Identify the storage path collected in step P101, and store the downloaded data in the specified location; Step P5 includes the following steps: Step P501: The system uses the mapping relationship between the identification information embedded in the file name and the pre-defined delay parameters to establish a mapping table or data structure, which realizes the corresponding management between the file name and the file delay time; based on this mapping relationship, the system uses a pre-set algorithm to accurately calculate the scheduled arrival time of each file; Step P502: After each file download operation is completed, the system automatically counts the number of downloaded files and the corresponding data volume through event triggering or callback functions, and uses an efficient transaction processing mechanism to write these statistical information into the database in real time; this database uses a specially designed table structure, which contains timestamp, file count, and data volume fields; Step P503: Use Spring Boot timing to push the download detailed record log to the DI interface to realize the record of the event, and record the detailed information of the data download to a table in MySQL, judge whether the data is abnormal by frequently querying the download record table updated in real time, push the alarm, and display the red floating on the page.

10. An Internet data real-time downloading system based on a configured foreground, characterized in that, The application is used for realizing the internet data real-time download method based on the configuration foreground, and includes a foreground configuration module, a JSON parameter transmission module, a background download module, a task scheduling module and a log and monitoring module. The foreground configuration module is used for providing a graphical interface, and a user configures parameters of a download task through the interface. The JSON parameter transmission module is used for converting the parameters configured by the foreground configuration module into a JSON format and transmitting the parameters to the background download module. The background download module is used for integrating multiple download tools including lftp, wget and a crawler. The task scheduling module is used for managing concurrent execution of multiple download tasks. The log and monitoring module is used for realizing real-time task monitoring and log recording functions.

Images