A method for automatically monitoring and reminding information of mining right auction and sale by public bidding

Abstract

The present application provides a kind of mining right auction information automatic monitoring and mail reminding method, comprising: S1. based on the target province website announcement information of edge node timing capture;S2. based on the mining right auction related announcement of edge node intelligent identification;S3. based on the filtered and de-duplicated announcement of edge node that has been handled;S4. based on the new announcement of edge node automatic sending mail.The present application reduces the cloud computing power consumption through edge node, through the dynamic polling mechanism based on surface fitting, according to website load (response time, request success rate) intelligent adjustment crawling interval, balance the server stress and information timeliness.Both avoid the IP ban risk caused by fixed high frequency crawling, and ensure the quick capture of information in low load period, replace the inefficient work of artificial repeated "brush website", significantly reduce manpower input.

Description

Technical Field

[0001] This invention relates to the field of automatic information monitoring technology, and in particular to a method for automatic monitoring and email reminders of mining rights bidding and auction information. Background Technology

[0002] In the field of monitoring information on mining rights bidding and auction, the traditional method mainly relies on manual, periodic access to the websites of natural resources departments in various provinces to manually search, filter, and track relevant announcements. This model has significant limitations:

[0003] First, it is inefficient and lacks timeliness. Manually refreshing multiple provincial websites daily is not only time-consuming but also fails to cover information updates during off-peak hours such as early mornings and holidays, resulting in delays in discovering some crucial announcements.

[0004] Secondly, information filtering is difficult. The website announcements contain a large amount of non-bidding and auction information (such as reclamation plans, permit issuance, geological hazard assessments, etc.), which requires manual review of each one. This is not only time-consuming, but also prone to errors due to subjective judgment, leading to the omission of valid information or misjudgment of invalid information.

[0005] Third, it is difficult to adapt to dynamic changes in the website. The website structures of different provinces vary greatly, and page layouts are often adjusted due to redesigns, requiring manual adaptation to the new structure repeatedly. At the same time, fixed-frequency manual access cannot match the fluctuations in website load (such as slow server response during peak hours and fast information updates during off-peak hours). Either the excessive frequency of access will trigger anti-scraping mechanisms, or the long intervals will cause important information to be missed.

[0006] Fourth, there is poor regional and policy adaptability. The wording used in provincial announcements has significant regional characteristics (for example, some provinces often use "listed for sale" while others often use "public transfer"), and policy adjustments may lead to changes in wording. Traditional keyword matching methods are difficult to adapt to these differences, resulting in low identification accuracy. Summary of the Invention

[0007] This invention provides a method for automatically monitoring and sending email reminders about mining rights bidding information, which reduces cloud computing power consumption through edge nodes.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A method for automatically monitoring and sending email reminders about mining rights bidding and auction information includes:

[0010] S1. Periodically crawl website announcements of target provinces based on edge nodes; deploy edge nodes in the target provinces, collect load characteristic data of the natural resources department websites of the target provinces, perform surface fitting processing to obtain dynamic polling intervals; based on the dynamic polling intervals, collect announcement information of the target provinces, perform parsing processing to obtain a list of announcements of each province containing titles, publication times, and links;

[0011] S2. Intelligent identification of mining rights bidding and auction related announcements based on edge nodes; The announcement list of each province is filtered by edge nodes based on two-level regular expression matching rules, and combined with the rule weights optimized by surface fitting to obtain the effective announcement list;

[0012] S3. Filtering and deduplication of processed announcements based on edge nodes; compare the list of valid announcements with the historical announcement data stored locally on the edge nodes to obtain a new list of announcements;

[0013] S4. Automatically send new announcement emails based on edge nodes; the new announcement list is generated by the edge nodes according to preset rules, and the email content is sent to the designated recipient through an encrypted transmission channel to complete the reminder.

[0014] In this specification, load characteristic data is collected in S1, and surface fitting processing is performed to obtain the dynamic polling interval, which includes:

[0015] For 30 consecutive days, server response times and request success rates for the target province were collected daily for 24 hours. Outlier removal and standardization were performed to obtain a load feature dataset. Based on a bivariate cubic polynomial, a surface fitting was performed on the load feature dataset to construct a response time prediction model and a request success rate prediction model. Combining the output results of the two prediction models, the dynamic polling interval for each province at different time periods was calculated and constrained by a preset objective function.

[0016] In this specification, the response time prediction model of the bivariate cubic polynomial is: the coefficients are solved by the least squares method to minimize the sum of the squares of the predicted response time and the actual response time; the request success rate prediction model is constructed using the same method to minimize the sum of the squares of the predicted success rate and the actual success rate; the objective function combines the response time weight, success rate weight and the base interval to calculate the polling interval, and adjusts the result through constraints of 10 to 60 seconds.

[0017] In this specification, S1, based on a dynamic polling interval, collects and parses announcement information, including:

[0018] Requests are sent to the target province websites sequentially at dynamic polling intervals, using province-specific request headers and embedding random delays. The returned HTML source code is cleaned, comments and escape characters are removed, and parsing rules are customized according to the webpage structure of each province to extract the title, publication time and links and standardize them to form a list of announcements for each province.

[0019] In this specification, the filtering process based on two-level regular expression matching rules in S2 includes:

[0020] The first level consists of exclusion rules, which include 17 categories of non-bidding / auction information and are precisely filtered using regular expressions. The second level consists of core rules, which include key actions, mining rights types, and fault-tolerant matching, and identify valid announcements using regular expressions. Announcements that are not excluded and meet the core rules are added to the list of valid announcements.

[0021] In this specification, the rule weights in S2 that combine surface fitting optimization include:

[0022] The matching accuracy of each rule in each province over the past 12 months was collected. A surface fitting was performed based on a bivariate quadratic polynomial to obtain a rule accuracy prediction model. The rule priority was adjusted according to the prediction accuracy: rules with high accuracy ≥ 0.9 were excluded and executed first, and core rules with high accuracy ≥ 0.85 had their confidence weight increased. Finally, the validity of the announcement was determined by the total confidence threshold.

[0023] In this specification, the historical announcement data stored locally in S3 includes:

[0024] The system primarily uses CSV files for storage, containing fields such as title, publication time, link, processing time, province code, and mineral type, and is stored in quarterly partitions. A Redis distributed hash table is also built as an in-memory cache, with the title hash value as the key, the existence flag as the value, and an expiration time of 365 days to ensure fast retrieval of historical data.

[0025] In this specification, the comparison process between S3 and locally stored historical announcement data includes:

[0026] Load historical title hashes from Redis or CSV into an in-memory set; for each announcement in the valid announcement list, calculate the title hash as the primary identifier and the title string as the secondary identifier; if the primary identifier is not in the historical set, it is determined to be a new announcement; if the primary identifier exists, verify whether it is a hash collision through the secondary identifier, and finally obtain a new announcement list and update it synchronously to CSV and Redis.

[0027] In this specification, S4 sends the following via an encrypted transmission channel:

[0028] The main channel uses the SMTP_SSL protocol, establishes an encrypted connection pool using corporate email, and enables DKIM signing; backup channels include WeChat and SMS; recipient preferences are verified before sending to ensure accurate message delivery.

[0029] In this manual, S4 constructs email content according to preset rules to complete the reminder, including:

[0030] Email titles are prioritized, and the body includes the announcement title, release time, mineral type, estimated area, countdown to deadline, and link. Dynamic retry is initiated when sending fails. After three failures, the email is added to the pending sending queue and an alarm is triggered. At the same time, the sending log is recorded and a weekly report is generated.

[0031] In summary, the present invention has at least the following beneficial effects:

[0032] This invention effectively solves the pain points of the traditional manual monitoring mode through innovative designs such as dynamic polling optimization, intelligent recognition upgrade, deduplication mechanism enhancement and multi-channel reminders. By using edge nodes as the main body for executing some steps, it can reduce the consumption of cloud computing power.

[0033] The specific effects are as follows:

[0034] 1. Advantages of edge computing:

[0035] Reduce transmission latency: Edge nodes process provincial data locally, reducing cross-regional data transmission and improving the real-time performance of crawling, comparison, and sending;

[0036] Reduce the pressure on the center: Edge nodes undertake some computing and storage tasks, avoiding excessive load on the central system due to processing data from multiple provinces;

[0037] Improved reliability: The failure of a single edge node only affects the corresponding province and does not affect the overall system, thus enhancing the stability of distributed processing.

[0038] 2. Improve monitoring efficiency and real-time performance

[0039] By employing a dynamic polling mechanism based on surface fitting, the crawling interval is intelligently adjusted according to website load (response time, request success rate), balancing server pressure and information timeliness. This avoids the risk of IP blocking caused by fixed high-frequency crawling while ensuring rapid capture of information during low-load periods, replacing the inefficient work of manually repeating "website crawling" and significantly reducing manpower input.

[0040] 3. Improve the accuracy of information identification

[0041] This system employs a two-level regular expression matching approach combined with surface fitting optimization to optimize rule weights, accurately filtering 17 categories of non-bidding / auction information. It also dynamically adapts to regional differences in terminology and policy changes across provinces. Through logic such as prioritizing high-accuracy rules and using confidence-weighted judgment, it significantly reduces interference from invalid information, minimizing misjudgments and missed judgments.

[0042] 4. Ensure the uniqueness and completeness of information.

[0043] By combining a distributed hash table with CSV storage for deduplication, the system enables rapid comparison of historical announcements and avoids duplicate reminders. It also supports partitioned data storage and scheduled backups to ensure the traceability of historical information and solves the problems of easy confusion and loss of manual records.

[0044] 5. Enhance the reliability of information delivery

[0045] Multi-channel notifications based on SMTP_SSL encrypted transmission (primarily email, supplemented by WeChat and SMS) support customized receiving preferences by province and mineral type, and a dynamic retry mechanism is used to handle sending failure scenarios, ensuring that important announcements can be delivered to relevant personnel in a timely manner in various network environments, avoiding the loss of opportunities due to information delays.

[0046] 6. Enhance system adaptability and scalability

[0047] The solution is capable of sensing and alerting to changes in website structure, and can quickly adapt to temporary adjustments through backup parsing rules. At the same time, models such as dynamic polling and weight optimization can be continuously iterated as data accumulates, supporting expansion to more provinces and providing stable support for strategic layouts across regions and mineral types. Attached Figure Description

[0048] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a schematic diagram of the automatic monitoring and email reminder method for mining rights bidding information involved in this invention.

[0050] Figure 2 This is a schematic diagram of the sample email body involved in this invention.

[0051] Figure 3 This is a schematic diagram of the announcement deduplication dual-identifier verification mechanism involved in this invention.

[0052] Figure 4This is a schematic diagram of the multi-channel transmission system architecture involved in this invention. Detailed Implementation

[0053] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the embodiments of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0054] The following disclosure provides many different implementations or examples for carrying out different structures of the embodiments of the present invention. To simplify the disclosure of the embodiments of the present invention, specific examples of components and arrangements are described below. Of course, these are merely examples and are not intended to limit the embodiments of the present invention. Furthermore, reference numerals and / or reference letters may be repeated in different examples of the embodiments of the present invention; such repetition is for simplification and clarity and does not in itself indicate a relationship between the various implementations and / or arrangements discussed.

[0055] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0056] like Figure 1 As shown, this embodiment provides a method for automatic monitoring and email notification of mining rights bidding information, including:

[0057] S1. Periodically crawl website announcements of target provinces based on edge nodes; deploy edge nodes in the target provinces, collect load characteristic data of the natural resources department websites of the target provinces, perform surface fitting processing to obtain dynamic polling intervals; based on the dynamic polling intervals, collect announcement information of the target provinces, perform parsing processing to obtain a list of announcements of each province containing titles, publication times, and links;

[0058] S2. Intelligent identification of mining rights bidding and auction related announcements based on edge nodes; The announcement list of each province is filtered by edge nodes based on two-level regular expression matching rules, and combined with the rule weights optimized by surface fitting to obtain the effective announcement list;

[0059] S3. Filtering and deduplication of processed announcements based on edge nodes; compare the list of valid announcements with the historical announcement data stored locally on the edge nodes to obtain a new list of announcements;

[0060] S4. Automatically send new announcement emails based on edge nodes; the new announcement list is generated by the edge nodes according to preset rules, and the email content is sent to the designated recipient through an encrypted transmission channel to complete the reminder.

[0061] By using edge nodes as the main entities for executing some steps, the consumption of cloud computing power can be reduced.

[0062] In some embodiments, the process of collecting load characteristic data in S1, performing surface fitting processing, and obtaining the dynamic polling interval includes:

[0063] For 30 consecutive days, server response times and request success rates for the target province were collected daily for 24 hours. Outlier removal and standardization were performed to obtain a load feature dataset. Based on a bivariate cubic polynomial, a surface fitting was performed on the load feature dataset to construct a response time prediction model and a request success rate prediction model. Combining the output results of the two prediction models, the dynamic polling interval for each province at different time periods was calculated and constrained by a preset objective function.

[0064] In some embodiments, the response time prediction model of the bivariate cubic polynomial is: the coefficients are solved by the least squares method to minimize the sum of squares of the predicted response time and the actual response time; the request success rate prediction model is constructed using the same method to minimize the sum of squares of the predicted success rate and the actual success rate; the objective function combines the response time weight, the success rate weight and the base interval to calculate the polling interval, and adjusts the result by the constraint condition of 10 to 60 seconds.

[0065] In some embodiments, S1, based on a dynamic polling interval, collects and parses announcement information, including:

[0066] Requests are sent to the target province websites sequentially at dynamic polling intervals, using province-specific request headers (including Host, Referer, dynamic User-Agent, and Cookie) and embedding random delays. The returned HTML source code is cleaned, comments and escape characters are removed, and parsing rules are customized according to the webpage structure (table or list layout) of each province to extract the title, publication time, and links and standardize them to form a list of announcements for each province.

[0067] In some embodiments, the filtering process based on two-level regular expression matching rules in S2 includes:

[0068] The first level consists of exclusion rules, which include 17 categories of non-bidding / auctioning information (such as reclamation plans, license issuance, etc.), and are precisely filtered using regular expressions. The second level consists of core rules, which include key actions (such as "bidding / auctioning" and "transfer / assignment"), mining right types (such as "mining rights" and "exploration rights"), and fault-tolerant matching (supporting space and symbol fault tolerance), and identify valid announcements using regular expressions. Announcements that are not excluded and meet the core rules are added to the list of valid announcements.

[0069] In some embodiments, the rule weights in S2 that combine surface fitting optimization include:

[0070] The matching accuracy of each rule in each province over the past 12 months was collected. A surface fitting was performed based on a bivariate quadratic polynomial to obtain a rule accuracy prediction model. The rule priority was adjusted according to the prediction accuracy: rules with high accuracy (≥0.9) were excluded and executed first, and core rules with high accuracy (≥0.85) had their confidence weight increased. Finally, the validity of the announcement was determined by the total confidence threshold.

[0071] In some embodiments, the historical announcement data stored locally in S3 includes:

[0072] The system primarily uses CSV files for storage, containing fields such as title, publication time, link, processing time, province code, and mineral type, and is stored in quarterly partitions. A Redis distributed hash table is also built as an in-memory cache, with the title hash value as the key, the existence flag as the value, and an expiration time of 365 days to ensure fast retrieval of historical data.

[0073] In some embodiments, the comparison process in S3 with locally stored historical announcement data includes:

[0074] Load historical title hashes from Redis or CSV into an in-memory set; for each announcement in the valid announcement list, calculate the title hash as the primary identifier and the title string as the secondary identifier; if the primary identifier is not in the historical set, it is determined to be a new announcement; if the primary identifier exists, verify whether it is a hash collision through the secondary identifier, and finally obtain a new announcement list and update it synchronously to CSV and Redis.

[0075] In some embodiments, sending via the encrypted transmission channel in S4 includes:

[0076] The main channel uses the SMTP_SSL protocol (port 465), establishes an encrypted connection pool using corporate email, and enables DKIM signing; backup channels include WeChat (for important announcements) and SMS (for email delivery failure scenarios); before sending, the recipient's preference settings are verified (filtered by province and mineral type) to ensure accurate information delivery.

[0077] In some embodiments, S4 constructs email content according to preset rules to complete the reminder, including:

[0078] Email titles are prioritized (urgent, important, normal), and the body includes the announcement title, release time, mineral type, estimated area, countdown to deadline, and link. If sending fails, dynamic retry is initiated (first attempt after 5 minutes, second attempt after 30 minutes and switching to a backup server). If three attempts fail, the email is added to the pending queue and an alarm is triggered. At the same time, sending logs are recorded and weekly reports are generated.

[0079] The technical concept of this invention is as follows:

[0080] Taking the websites of the natural resources departments of six provinces (including provinces, autonomous regions, municipalities, and special administrative regions) A1, B2, C3, D4, E5, and F6 as examples, the technical solution of this invention is further explained as follows:

[0081] S1: Periodically scrape website announcements from target provinces

[0082] Objective: To achieve efficient monitoring of the websites of the natural resources departments of six provinces (A1, B2, C3, D4, E5, and F6) through a crawling strategy optimized by dynamic polling and surface fitting. While accurately capturing announcement information, this aims to minimize the load on the target servers, resolving the contradiction between slow server response under high load and information delay under low load caused by traditional fixed-interval crawling, and providing real-time and highly complete raw data for subsequent stages.

[0083] Edge computing adaptation points: Deploy collection modules at edge nodes (such as regional edge servers) close to the target province to localize load monitoring and crawling scheduling, reducing cross-regional data transmission between the central server and the target website.

[0084] S1.1 Optimization of Dynamic Polling Mechanism Based on Surface Fitting

[0085] Traditional fixed-interval crawling strategies cannot adapt to the load fluctuations of target websites (such as a surge in traffic from 10:00 to 15:00 on weekdays and a sharp drop in traffic in the early morning), which can easily lead to IP blocking during high-load periods or untimely information crawling during low-load periods. By constructing a three-dimensional relationship of "time-province-load" through a surface fitting model, the polling interval can be intelligently adjusted to dynamically match the crawling frequency with the server load.

[0086] S1.1.1 Load Characteristic Data Acquisition and Preprocessing

[0087] To build a fitting model, the system needs to collect and clean the load characteristic data of the target website:

[0088] Data collection scope: 30 consecutive days, covering 24 hours a day (each hour is a time node), 6 provinces, forming 24×6=144 basic sample units.

[0089] Data collection content:

[0090] Response time z1(x,y): Send 10 requests to province y within one hour, record the time taken from sending to receiving a complete response (in seconds) for each request, and take the average value as the response time for that period (e.g., the response times for the 10 requests from 10:00 to 11:00 for B2 are 2.1, 2.3, ..., 2.5 seconds, with an average of 2.3 seconds).

[0091] Request success rate z2(x,y): 20 requests are sent to province y within one hour, and the percentage of times the return status code is 200 (normal) is counted (e.g., 19 out of 20 requests are successful, the success rate is 95%).

[0092] Data preprocessing:

[0093] Outlier handling: Remove extreme values ​​with response time > 10 seconds or success rate < 50% (considered as network fluctuations or server failures, not reflecting normal load) and replace them with the historical average for that period in that province.

[0094] Standardization: Scale the response time to the [0,10] range and keep the success rate in the [0,100] range to avoid the impact of differences in units on model fitting.

[0095] S1.1.2 Construction of a Bivariate Cubic Polynomial Surface Fitting Model

[0096] A bivariate cubic polynomial is used to fit the load characteristics to a surface, enabling load status prediction for any time and any province. The model construction process is as follows:

[0097] 1. Response time fitting model

[0098] Model expression:

[0099] Where: x is the time variable (hours, 1≤x≤24), and y is the province variable (codes 1-6, corresponding to A1 to F6); The predicted response time (in seconds); a0 to a9 are 10 coefficients to be solved, determined by the least squares method.

[0100] Solution process:

[0101] Construct the objective function: (Minimize the sum of squares of the actual response time and the predicted value).

[0102] The coefficients are solved using Python's scipy.optimize.curve_fit function, with initial values ​​set to a0~a9=0.1, iteration precision 1e-6, ensuring fitting error <5%.

[0103] 2. Request success rate fitting model

[0104] Model expression:

[0105] in: The predicted request success rate (in %) is given; b0 to b9 are coefficients to be solved, solved using the same method as the response time model, with the objective function being...

[0106] S1.1.3 Calculation and Constraints of Optimal Polling Interval

[0107] Based on the prediction results of the fitted model, a comprehensive optimization objective function is constructed to calculate the optimal crawling interval for each province at different time periods, balancing real-time performance and server load.

[0108] Objective function:

[0109] Where: t opt (x,y) represents the optimal polling interval, k1=5 (response time weight): the longer the response time, the higher the server load, and the interval needs to be increased to reduce the pressure; k2=0.1 (success rate weight): the higher the success rate, the more stable the server status, and the interval can be reduced to improve real-time performance; t0=10 (base interval): ensures the minimum crawling frequency and avoids the interval being too small due to weight offsetting.

[0110] Constraints:

[0111] Minimum interval: t opt ≥10 seconds (to avoid triggering anti-crawling measures due to high-frequency requests);

[0112] Maximum interval: t opt ≤60 seconds (ensure information delay does not exceed 1 minute).

[0113] If the calculation result is outside the range, the boundary value will be automatically taken (e.g., if the calculated result is 8 seconds, it will be adjusted to 10 seconds).

[0114] Instance computation and scheduling logic:

[0115] B2(y=2)10:00(x=10): Then t opt =5×2.3-0.1×92+10=11.5-9.2+10=12.3 seconds (take 12 seconds).

[0116] E5(y=5) 3:00 AM (x=3): Then t opt =5×0.7-0.1×99+10=3.5-9.9+10=3.6 seconds (adjusted to 10 seconds due to constraints).

[0117] S1.1.4 Dynamic Polling Execution and Model Update

[0118] By integrating the optimal interval into the crawling scheduling system, fully automated adaptive crawling can be achieved:

[0119] Scheduling matrix generation: Every day at 1:00 AM, the system automatically loads the latest data from the past 30 days, retrains the surface fitting model, and generates a polling interval matrix T = [t] for the current day's 24-hour period across 6 provinces. opt (x,y)] 24×6 The data is stored in config / dynamic_interval.csv.

[0120] Real-time adjustment execution: The main scanning function scan_provinces() reads the interval parameter of the current time period (e.g., 10:00-11:00) from matrix T every hour and executes according to the following logic:

[0121] 1. Scan sequentially according to province code order (A1→B2→…→F6);

[0122] 2. After scanning each province, based on t in matrix T opt Wait for the corresponding duration (e.g., B2 waits for 12 seconds, E5 waits for 10 seconds);

[0123] 3. If a scan of a certain province fails (e.g., returns 403), temporarily extend its interval by 50% (e.g., 12 seconds → 18 seconds) to avoid concentrated retries increasing the load.

[0124] Model iterative optimization: weekly Sunday analysis of the model prediction error over the past 7 days ( and If the average error is greater than 10%, the training data window is expanded to 60 days, the model coefficients are refitted, and the prediction accuracy is improved.

[0125] S1.2 Anti-crawling Mechanism Bypass

[0126] By combining a load-aware strategy with dynamic polling, the anti-crawling mechanism can be further optimized to reduce the probability of being identified as a crawler by the target website.

[0127] Dynamic generation and adaptation of request headers:

[0128] Basic parameter library: 100 sets of User-Agent (covering different versions of browsers such as Chrome, Firefox, and Edge) and 20 sets of Referer (the URLs of different subpages of the website of that province) are preset for each province and stored in config / headers_pool / .

[0129] Dynamic selection logic: During low load periods (t) opt <20 seconds), every 3 requests, a new set of User-Agent and Referer will be randomly generated; during high-load periods (t opt (≥20 seconds), change every 5 requests to reduce the crawler's "frequent changes" characteristic.

[0130] Intelligent Cookie Management: Maintain the session through the requests.Session() object. Update the Cookie every hour during low load periods (extracted from the response of the website's home page), and update it every 2 hours during high load periods, simulating the session cycle of real users.

[0131] Optimization of Request Sending Strategy:

[0132] Random Delay Embedding: Call time.sleep(random.uniform(delay_min, delay_max)) to achieve randomness in the request interval. During low load periods, delay_min = 0.5 seconds and delay_max = 1 second. During high load periods, delay_min = 1 second and delay_max = 2 seconds, simulating the natural pauses during human browsing.

[0133] URL Parameter Encoding: Strictly encode special characters (such as spaces and Chinese characters) in the request URL (e.g., "Zhaopaigua" → %E6%8B%9B%E6%89%B9%E6%8C%82) to avoid being recognized as an abnormal request due to improper encoding.

[0134] Distributed Requests: If the system is deployed on multiple servers (such as nodes in Chengdu and Kunming of the group), the crawling tasks are assigned according to the geographical division of provinces (e.g., the Chengdu node is responsible for B2 and F6, and the Kunming node is responsible for D4 and C3), and the IP segments of the corresponding regions are used (dynamically switched through VPN) to reduce the request frequency of a single IP.

[0135] S1.3 Web Page Parsing and Field Extraction

[0136] For the different structures of websites in 6 provinces (e.g., A1 uses a table layout and F6 uses a list layout), customize precise parsing logic to ensure the complete extraction of core fields (title, release time, link):

[0137] Parsing Toolchain and Preprocessing:

[0138] Parsing Engine: Mainly use BeautifulSoup4 (in conjunction with the lxml parser), and use pyquery as an auxiliary (to handle pages with JavaScript dynamic loading, such as some announcement pages of B2).

[0139] HTML Preprocessing: Clean the obtained original HTML source code, remove comment tags (<!---->), extra spaces, and escape characters (e.g., ; → space) to avoid interfering with parsing.

[0140] Parsing Rules for Some Provinces:

[0141] A1:

[0142] Announcement list container:

[0143]

[0144]

[0145]

[0146] <ul class＝"announcement-list">, each announcement corresponds to <liclass="ann-item">;

[0147] Title and Link: Located at <aclass="ann-link"href="https: / / zrzyt.gansu.gov.cn / gggs / 20XX07.html">F6 Copper Mine Mining Right Listing Announcement, extraction logic:

[0148] for li in soup.select('ul.announcement-list li.ann-item'): title = li.select_one('a.ann-link').text.strip() link = li.select_one('a.ann-link')['href'] # already an absolute path time = li.select_one('span.ann-time').text.strip() + '00:00' # complete hours and minutes.

[0149] Other provinces: The parsing rules for B2, C3, D4, and E5 are similar. For the detailed logic, see Appendix 1. The core difference lies in the tag class names (e.g., B2 uses class="notice-item", and C3 uses class="ann-item").

[0150] Field Standardization and Verification:

[0151] Title: Remove special characters at the beginning and end (such as 【】, [], ()), and uniformly convert to UTF-8 encoding (e.g., "採礦權” → "采矿权”).

[0152] Time: Parse various formats through the dateutil.parser library (e.g., "20XX年XX月XX日” → "20XX-XX-XX 00:00", "XX / XX / 20XX” → "20XX-XX-XX 00:00”). Those that cannot be parsed are marked as "unknown time" and trigger manual verification.

[0153] Link: Check if it is an absolute path (including http). If it is a relative path (e.g., / gggs / 20XX.html), then splice the domain name of the corresponding province (extracted from the Host parameter) to generate an absolute path.

[0154] Parsing Quality Monitoring: Statistically calculate the field extraction completeness rate (number of complete fields / total number of announcements) for each province every hour. If the completeness rate of a certain province is <80% for three consecutive hours, automatically trigger the exception warning mechanism of S1.4.

[0155] S1.4 Exception Handling and Warning

[0156] Establish a multi-level anomaly handling system to ensure that a failure in a single province does not affect the overall process, and that anomalies are traceable and can be responded to quickly:

[0157] Hierarchical anomaly definition and handling process:

[0158] Level 1 Exception (Minor): Single announcement parsing failure (e.g., title is empty). Handling logic: Log and skip this announcement. It does not affect other announcements. Log format: [Time] A1 Parsing failure: title is empty. Link: https: / / ...

[0159] Level 2 Anomaly (Moderate): Province scan fails 3 times consecutively (e.g., network timeout). Handling logic: Trigger retry(province, attempts=3, interval=10) to retry. If it still fails, pause the province scan for 30 minutes and try again. During this period, record detailed error information (status code, response content).

[0160] Level 3 Error (Severe): Webpage structure changed (parsing result is empty and no network error), processing logic:

[0161] Immediately invoke send_alert_email(province, level="urgent");

[0162] Save the current HTML source code to logs / error_pages / , with the following naming format: gansu_struct_change_20XX07101000.html;

[0163] Automatically switch to the province's alternative resolution rules (e.g., F6's alternative rules are based on...). <div class＝"old-list">Label, applicable to the temporary rollback version of the website).

[0164] Refined content of the warning email:

[0165] Recipient: Technical Maintenance Team (tech_team@zijinmining.com) + Business Leader (biz_head@zijinmining.com);

[0166] Subject: [Level 3 Warning] Failure to Parse the F6 Mining Right Announcement (Structural Change);

[0167] Body: Trigger Time: 20XX-XX-XX 10:00:00 Exception Description: Failed to extract valid information from the F6 announcement list page for 5 consecutive times, and the HTML structure has changed. Historical Normal Structure: ul.announcement-list li a.ann-link Current Detected Structure: div.new-announcement lia Suggested Action: 1. Check the source code of the attachment; 2. Update the CSS selector to div.new-announcement lia Backup Rule Status: Automatically enabled, current extraction success rate: 70%;

[0168] Attachment: HTML source code of the abnormal page + Trend chart of the parsing success rate in the past 7 days (for easy comparison of differences before and after structural changes).

[0169] S2: Intelligent Identification of Announcements Related to Mining Rights Auction and Listing (Optimized by Integrating Surface Fitting)

[0170] Objective: Dynamically optimize the weight parameters of regular matching rules through a surface fitting model to solve the recognition deviation caused by differences in provincial word usage habits (such as D4 commonly uses "listing for transfer", and F6 commonly uses "open transfer") and policy time changes (such as since 20XX, "auction, listing, and transfer" has been unified as "transfer"), and improve the recognition accuracy of announcements across provinces and time periods to over 99%.

[0171] S2.1 Dynamic Adjustment of Feature Weights Based on Surface Fitting

[0172] There are significant regional characteristics in the announcement word usage of different provinces (such as E5 commonly uses "mining right", and B2 commonly uses "mining industry right"), and the word usage in the same province may change with policy adjustments (such as after the tightening of ecological protection policies, the number of announcements related to "reclamation plan" increases). By constructing a three-dimensional surface model of "time - province - rule accuracy rate", dynamically adjust the matching weights of regular rules to make the recognition logic adapt to these differences.

[0173] S2.1.1 Rule Feature Data Collection and Annotation

[0174] To train the weighted fitting model, it is necessary to systematically collect and label the historical matching performance data of each regularization rule:

[0175] Rule system: Covers the 17 exclusion rules (R) in S2.2 11 -R 117 ) and 3 core rules (R 21 -R 23 There are 20 rules in total, and each rule is numbered k = 1...20.

[0176] Data collection dimensions:

[0177] Time x: Month (1-12, e.g., x=7 represents July);

[0178] Province y: 1-6 (same as province code in S1);

[0179] Accuracy w k (x,y): The actual matching accuracy of the k-th rule in province y and month x, calculated using the following formula:

[0180]

[0181] For example, R 21 The rule was triggered 100 times in month D47, and 92 of those times it correctly identified valid announcements, with an accuracy of w = 0.92.

[0182] Data annotation process:

[0183] The announcements crawled by S1 are manually labeled (valid / invalid) daily as the baseline truth value;

[0184] For each rule, count the number of times it is triggered and the number of times it is correct in each province and month;

[0185] At the end of each month, 20 rules × 12 months × 6 provinces = 1440 sample data are generated and stored in data / rule_performance.csv.

[0186] S2.1.2 Construction and Solution of Weighted Fitting Model

[0187] A surface fitting method is used to measure the accuracy of the rules using a bivariate quadratic polynomial. The model expression is as follows:

[0188]

[0189] in: c represents the prediction accuracy of rule k in province y and month x (values ​​range from 0 to 1); k0 ~c k5 The fitting coefficients for the k-th rule (each rule is solved separately);

[0190] Solution method: Minimize the prediction error using the least squares method. The coefficients are solved using scipy.optimize.curve_fit, with the initial value set to c. k0 ~c k5 =0.1.

[0191] S2.1.3 Application of Dynamic Weights in Matching Logic

[0192] The prediction accuracy obtained from the fitting The matching weights are converted into rules, and the execution priority and confidence calculation of the two-level matching logic are dynamically adjusted:

[0193] Application of weights for exclusion rule (R1):

[0194] Priority sorting: for High accuracy rules (such as R) 11 The "reclamation plan" exclusion rule is prioritized in the matching process (ranked in the top 5) to quickly filter out a large amount of irrelevant information; Low accuracy rules (such as R) 117 The "administrative notice" exclusion rule will be postponed until the core rule is matched, and a second verification will be added (such as strengthening the exclusion if the department that issued the announcement is "office").

[0195] Example: R in July of E5 14 (Exclusion rules for license-related products) Prioritize and directly filter "E5 Province Mining License Renewal Announcement"; while R 117 (Administrative Notice) If, after matching the core rules, the title contains "mining rights" and the issuing department is "Mining Management Office", then it cannot be excluded (to avoid accidental deletion).

[0196] Application of weights in core rules (R2):

[0197] Confidence-weighted: Set a base confidence level b for each core rule. k =0.5, if (High accuracy) then weighted confidence Otherwise s k =b k The total confidence level is the sum of the weighted confidence levels of each rule:

[0198] Judgment threshold: When S≥1.2, it is judged as a valid announcement.

[0199] Example: R in D47 months 21 ("Bidding, auction, and listing" matching rules) Then s1 = 0.5 + 0.2 × 0.92 = 0.684; R 22 ("Mining Rights Type" Rules) s2=0.5+0.2×0.88=0.676; R 23 (Fault Tolerance Rules) s3 = 0.5. The total confidence level S = 0.684 + 0.676 + 0.5 = 1.86 ≥ 1.2, so it is considered valid.

[0200] S2.2 Two-level regular expression matching

[0201] Based on dynamic weight adjustment, the expression of the two-level regular expression rule is optimized to enhance the ability to recognize fuzzy terms and novel expressions:

[0202] Level 1: Exclusion Rules (R1, Class 17)

[0203] Each rule targets a typical characteristic of a type of non-bidding / auctioning information, using regular expressions for precise filtering:

[0204] R 11 (Reclamation Plans): .*(Reclamation Plans | Ecological Restoration | Land Reclamation | Mine Restoration | Remediation | Soil Remediation).*(Covering variations such as "Mine Ecological Restoration Plan" and "Land Reclamation Acceptance");

[0205] R 14 (Licenses): *(Mining License | Exploration License | License Issuance | Renewal | Modification | Cancellation | Annual Inspection | Renewal). *(Covering "Mining License Renewal" and "Exploration License Cancellation", etc.);

[0206] (See Appendix 2 for complete rules. Each rule has been optimized through testing with 1000+ samples.)

[0207] Second level: Core rules (R2, 3 categories)

[0208] Dynamically identifies key combinations related to bidding and auction, supporting complex fault tolerance:

[0209] R 21 (Key Actions): *(Bidding, Auction, and Listing | Listing Announcement | Transfer Announcement | Public Transfer | Public Transfer | Mining Rights Transfer). *(Supports tolerance for spaces / symbols in "Bidding, Auction, and Listing" and "Listing-Transfer");

[0210] R 22 (Mining rights type): *(Mining rights|Exploration rights|Mineral rights|Mineral mining rights|Gold mining rights|Copper mining rights). *(Supports matching of traditional Chinese characters / similar words such as "mining rights" and "exploration rights");

[0211] R 23 (Fault Tolerance Enhancement): .*(Listing\s*transfer|transfer\s*announcement|(transfer|transfer)transfer|bid\s*auction\s*sale\s*listing\s*).*(Matches any space using \s*, (transfer|transfer) matches multiple forms).

[0212] Matching process:

[0213] 1. Input: The merge announcement list All_List output by S1;

[0214] 2. Exclude rule matching: Apply R1 rules sequentially, and skip announcements marked with is_excluded=True;

[0215] 3. Core rule matching: For announcements that have not been excluded, apply the R2 rule and calculate the total confidence S;

[0216] 4. Judgment: If S≥1.2, mark is_valid=True and add it to the valid list Valid_List.

[0217] S2.2 Recognition Result Verification and Feedback

[0218] Manual verification sampling: 50 identification results are randomly selected daily (25 valid + 25 invalid). Business personnel mark the results as misjudged (e.g., valid results are marked as invalid, or vice versa), and record the misjudgment rules and information such as province and time.

[0219] Model feedback iteration: Misclassified data is updated in real time to the rule feature dataset w. k (x,y) (If a valid announcement is R) 11 If an error is ruled out, the w value of that rule in the corresponding province and time period will be reduced. k The weighted fitting model is retrained weekly, and the coefficients c are adjusted. k0 ~c k5 This achieves closed-loop optimization of "data-model-recognition".

[0220] S3: Announcement filtering and deduplication have been processed.

[0221] Objective: To ensure no duplicate notifications occur in high-concurrency scenarios (e.g., a province issuing 50+ announcements per day) through a deduplication mechanism combining distributed hash tables and CSV backups, while also supporting fast historical data querying and statistical analysis. The announcement deduplication dual-identifier verification mechanism is as follows: Figure 3 As shown.

[0222] Edge computing adaptation points: Historical data (CSV and Redis) for the corresponding province is stored locally on the edge nodes. The comparison of new announcements is completed directly on the edge nodes, and only the final new announcement is synchronized to the central node, reducing the amount of data transmission.

[0223] S3.1 Local Storage and Distributed Hash Table Design

[0224] Main storage (CSV file):

[0225] Path and encoding: . / data / processed_announcements.csv, UTF-8-SIG encoding, supports Chinese characters without garbled characters;

[0226] Field expansion: In addition to the original 4 fields, province (province code) and category (mineral type, such as "gold mine") have been added to facilitate subsequent statistical analysis. Format: title,time,link,processed_time,province,category;

[0227] Partitioned storage: When the file size exceeds 100MB (approximately 100,000 records), it is automatically partitioned by quarter (e.g., 20XXQ1_processed.csv) to improve read speed.

[0228] Distributed hash table (memory cache):

[0229] Implementation: Use a Redis database to build a distributed hash table processed_hash, with the key being the hash value of title (hash(title)) and the value being 1 (marked as present);

[0230] Synchronization mechanism: Each time the CSV file is updated (a new record is added), it is synchronously written to Redis; when the system starts, all header hashes are loaded from the CSV into Redis to ensure that the data in memory and on disk are consistent.

[0231] Expiration policy: Set the key expiration time to 365 days (announcements from one year ago do not need to be deduplicated), automatically clean up old data, and free up memory.

[0232] S3.2 Deduplication and Comparison Process

[0233] 1. Loading historical data:

[0234] Call the load_history() function to read all hash values ​​from Redis into the memory collection History_Hash (if Redis is not running, it will be downgraded to loading from CSV);

[0235] Loading time optimization: Large files are read in chunks (10,000 rows per chunk) using the pandas library's read_csv function to avoid memory overflow caused by loading all at once.

[0236] 2. Comparison with new announcements:

[0237] For each announcement in the Valid_List output by S2, calculate the double identifier:

[0238] Main identifier: hash(title) (for quick comparison);

[0239] Secondary identifier: title string (for secondary verification in case of hash collision).

[0240] Comparison logic:

[0241] If hash(title) is not in History_Hash: it is considered a new announcement and added to New_List. Otherwise: if the title is completely consistent with the history (sub-identifier verification), it is considered a duplicate; otherwise, it is considered a hash collision (probability < 0.001%), added to New_List and recorded in the exception log.

[0242] 3. Data updates and backups:

[0243] Append to write: Call update_csv(New_List) to open the CSV file in append mode, write the 6 fields of the new announcement, and simultaneously update the processed_hash in Redis.

[0244] Scheduled backup: Automatically backs up CSV files to the . / backup / directory at 2 AM every day, retaining backups from the most recent 30 days (named YYYYMMDD_processed_backup.csv), and supports rolling back to data to a specified date using the restore_backup(date) function.

[0245] S4: New announcement emails are sent automatically.

[0246] Objective: To ensure that important announcements (such as large-scale gold mining rights transfers) reach key personnel within 5 minutes through multi-channel transmission and intelligent prioritization, while also supporting customized receiving preferences based on mineral type and province, thereby improving information delivery efficiency. The multi-channel transmission system architecture is as follows: Figure 4 As shown.

[0247] Edge computing adaptation points: Edge nodes handle email content construction and sending retries locally, and only synchronize the queue to be sent to the center when sending fails, thereby improving the sending response speed.

[0248] S4.1 Multi-channel Sending and Encrypted Transmission SMTP_SSL Main Channel:

[0249] Configuration optimization: sender email address mineral_monitor@zijinmining.com, SMTP server smtp.zijinmining.com (enterprise-specific, supports TLS 1.3 encryption), port 465, enable DKIM signature (email legitimacy verification) to reduce the probability of being marked as spam;

[0250] Connection pool management: Use smtplib.SMTP_SSL to create a long connection pool (5 connections) to avoid performance loss caused by frequent connection creation / closing. Connections will automatically reconnect after 30 minutes of inactivity.

[0251] Backup channels (WeChat / SMS):

[0252] WeChat Work: For announcements marked as "important" (such as those with titles containing "gold mine" or "lithium mine"), send application messages synchronously via the WeChat Work API, and @ relevant personnel (e.g., "@Manager Zhang, the announcement for the listing of D4 gold mine rights has been released").

[0253] SMS: When the email fails to send twice in a row, an SMS notification will be triggered (through the group's SMS gateway). The content is "[Mining Rights Reminder] An announcement for a certain copper mine in F6 has been released. View the link: https: / / ... (The SMS contains a short link and is valid for 24 hours)".

[0254] S4.2 Email Content Personalization and Prioritization

[0255] Personalized receiver preferences: Users can set preferences in config / receiver_preferences.json, such as in JSON format:

[0256] {

[0257] "li.san@zijinmining.com":{"provinces":[2,4],"minerals":["gold mines","lithium mines"]},

[0258] "zhang.si@zijinmining.com":{"provinces":[1,3],"minerals":["copper mines"]}

[0259] }

[0260] (Li San only accepts gold and lithium mine announcements for B2 and D4, while Zhang Si only accepts copper mine announcements for A1 and C3).

[0261] Email Priority Classification:

[0262] Level 1 (Urgent): Emails with the subject line containing "gold mine" or "lithium mine" and an area greater than 50 square kilometers, prefixed with "★Urgent★", sent to all core personnel (including senior executives), and triggering a WeChat notification.

[0263] Level 2 (Important): Titles containing "copper mine" or "lead-zinc mine" with the prefix "[Important]" should be sent to all members of the Mining Rights Investment Department.

[0264] Level 3 (Normal): Other mineral types, prefixed with "[Mining Rights Reminder]", sent to the regional manager.

[0265] Enhanced text:

[0266] Add mineral type and area estimation: Extract mineral type from title using NLP model (e.g., "Exploration right for a gold mine in F6 province" → "Gold mine"), and estimate mining area by combining historical data (e.g., "Large" → "50-100 square kilometers");

[0267] Countdown reminder: Calculate the remaining days based on the announcement deadline (e.g., "15 days left until the listing deadline") and highlight it;

[0268] Example text (e.g.) Figure 2 (As shown) HTML format:

[0269] <h2>

★Urgent★

[0270] <strong> Release time< / strong> Released at 10:00 AM on 20XX-XX-XX (outside of working hours)

[0271] <strong> minerals< / strong> Gold Mine | <strong> Estimated area< / strong> 80 square kilometers

[0272] <strong> Countdown to the deadline< / strong> ： <span style＝"color:red"> 15 days

[0273] <strong> View link< / strong> ： <a href＝"https: ..."> Click to access (valid for 30 days)

[0274] <em> This announcement matches your preference: D4 Province + Gold Mine< / em> .

[0275] S4.3 Intelligent Retry and Transmission Status Tracking

[0276] Dynamic retry strategy:

[0277] First failure: Retry after 5 minutes (applicable to temporary network fluctuations);

[0278] If the connection fails twice: retry after 30 minutes, and switch to a backup SMTP server (e.g., smtp.backup.zijinmining.com);

[0279] Three failures: Add to pending_emails queue (stored in . / data / pending / , JSON format) and trigger a technical alert.

[0280] Status tracking and statistics:

[0281] Sending logs: Record the send_time, status (success / failure), retry_count (number of retries), and receiver_feedback (e.g., read receipt) for each email, stored in . / logs / email_send_log.csv;

[0282] Weekly Report: Every Monday, a "Statistics Report on Sending Mining Rights Announcement Emails" is automatically generated, which includes metrics such as the number of emails sent to each province, average delivery time, failure rate, and user open rate. This report is then pushed to the technical and business managers via WeChat Work.

[0283] Surface fitting and end-to-end collaborative mechanism

[0284] Model linkage between S1 and S2:

[0285] The province code y and time x in the polling interval matrix T of S1 share the same spatiotemporal coordinate system with the weight fitting model of S2, ensuring the consistency of the "load-word" feature;

[0286] High-load period (t) of a certain province in S1 opt If the data capture time is ≥30 seconds, S2 will automatically increase the weight of the core rules for that province (e.g., +0.1) to compensate for incomplete information caused by possible capture delays.

[0287] Feedback loop between S2 and S4:

[0288] User feedback from S4 (such as marking "false positives") updates the rule accuracy of S2 in real time. k (x,y), retrain the weighted fitting model weekly;

[0289] For minerals (such as lithium) that are marked as "important" three times consecutively, S2 automatically increases the weight of core rules containing keywords related to that mineral (e.g., +0.15) and prioritizes their identification.

[0290] End-to-end anomaly coordination:

[0291] S1's webpage structure change alerts are sent through S4's emergency channel (WeChat + SMS) to ensure that technical personnel respond within 15 minutes;

[0292] A hash collision anomaly in S3 (probability <0.001%) triggers a manual confirmation email in S4, where business personnel determine whether it is a new announcement to avoid missed reports.

[0293] Appendix 1: CSS Selector Rules for Web Page Parsing in Each Province

[0294]

[0295]

[0296] Appendix 2: Exclusion Rules for 17 Categories of Non-Bidding / Auction Information (R1)

[0297] 1..*(Reclamation Plans | Ecological Restoration | Land Reclamation | Mine Restoration | Remediation | Soil Remediation | Geological Environment Remediation).*→Reclamation and Ecological Restoration Category;

[0298] 2..*(Review Results|Audit Results|Expert Review|Technical Review|Compliance Review).*→Review Category;

[0299] 3..*(Appraisal Report | Value Assessment | Mining Rights Appraisal | Asset Evaluation | Appraisal Disclosure).*→Appraisal Category;

[0300] 4..*(Reserve Verification | Reserve Report | Resource Reserves | Reserve Review | Reserve Filing).*→Reserve Verification Category;

[0301] 5..*(Mining License | Exploration License | License Issuance | Renewal | Modification | Cancellation | Annual Inspection | Renewal | Replacement).*→License Category;

[0302] 6..*(Geological Hazards|Hazard Assessment|Hazard Management|Prevention and Control Plans|Hazard Investigation).*→Geological Hazard Category;

[0303] 7..*(Development and Utilization Plan|Utilization Plan|Development Plan|Mining Plan).*→Development Plan Class;

[0304] 8..*(Acceptance Results|Acceptance Announcement|Completion Acceptance|Special Acceptance|Stage Acceptance).*→Acceptance Category;

[0305] 9..*(Qualification Review | Qualification Announcement | Qualification Certification | Qualification Renewal | Qualification Certificate Replacement).*→Qualification Category;

[0306] 10..*(Public Notice of Planning | Mineral Resources Planning | Master Plan | Special Plan | Zoning Adjustment).*→Planning Category;

[0307] 11..*(Prospecting right retention|Mining right cancellation|Mining right retention|Change of rights and functions).*→Change of rights and functions category;

[0308] 12..*(Information Change|Name Change|Address Change|Legal Representative Change|Entity Change).*→Information Change Category;

[0309] 13..*(Compensation Fee | Collection Notice | Payment Notice | Payment of Price | Revenue Transfer).*→Expense Category;

[0310] 14..*(Safety Facilities|Safety Review|Safety Acceptance|Safety Production|Safety Inspection).*→Safety Category;

[0311] 15..*(Exploration Plan|Exploration Filing|Exploration Report|Geological Exploration|Exploration Scheme).*→Exploration Category;

[0312] 16..*(Transfer Revenue|Revenue Assessment|Revenue Disclosure|Revenue Payment|Price Assessment).*→Revenue Category;

[0313] 17..*(Meeting Notices | Policy Interpretations | Service Guides | Public Information Guides | Work Updates).*→Administrative Notices.

[0314] The embodiments described above are for illustrative purposes only and are not intended to limit the invention. Therefore, any changes in numerical values ​​or substitutions of equivalent elements should still fall within the scope of this invention.

[0315] The above detailed description will enable those skilled in the art to understand that the present invention can indeed achieve the aforementioned objectives and has complied with the provisions of the Patent Law.

[0316] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention. The above descriptions are merely preferred embodiments of the invention and are not intended to limit the invention. It should be noted that any modifications, equivalent substitutions, and improvements made within the spirit and principles of the invention should be included within the scope of protection of the invention.

[0317] It should be noted that the above description of the process is for illustrative purposes only and does not limit the scope of this specification. Those skilled in the art can make various modifications and changes to the process under the guidance of this specification. However, these modifications and changes remain within the scope of this specification.

[0318] The basic concepts have been described above. Obviously, for those skilled in the art who have read this application, the above disclosure is merely illustrative and does not constitute a limitation of this application. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this application. Such modifications, improvements, and corrections are suggested in this application, and therefore, such modifications, improvements, and corrections still fall within the spirit and scope of the exemplary embodiments of this application.

[0319] Furthermore, this application uses specific terms to describe its embodiments. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic related to at least one embodiment of this application. Therefore, it should be emphasized and noted that "an embodiment," "one embodiment," or "an alternative embodiment" mentioned twice or more in different positions in this specification do not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of this application can be appropriately combined.

[0320] Furthermore, those skilled in the art will understand that aspects of this application can be described and illustrated through several patentable types or situations, including any new and useful combination of processes, machines, products, or substances, or any new and useful improvements thereof. Therefore, aspects of this application can be implemented entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or a combination of hardware and software. All of the above hardware or software can be referred to as a "unit," "module," or "system." Furthermore, aspects of this application can take the form of a computer program product embodied in one or more computer-readable media, wherein computer-readable program code is contained therein.

[0321] The computer program code required for the operation of each part of this application can be written in any one or more programming languages, including object-oriented programming languages ​​such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, and Python; general programming languages ​​such as C; Visual Basic, Fortran2103, Perl, COBOL2102, PHP, and ABAP; dynamic programming languages ​​such as Python, Ruby, and Groovy; or other programming languages. This program code can run entirely on the user's computer, or as a standalone software package on the user's computer, or partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any network, such as a local area network (LAN) or wide area network (WAN), or connected to an external computer (e.g., via the Internet), or in a cloud computing environment, or used as a service such as Software as a Service (SaaS).

[0322] Furthermore, unless expressly stated in the claims, the order of processing elements and sequences, the use of numbers and letters, or other names described in this application are not intended to limit the order of the processes and methods of this application. Although some currently considered useful embodiments of the invention have been discussed in the foregoing disclosure by way of various examples, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments; rather, the claims are intended to cover all modifications and equivalent combinations that conform to the substance and scope of the embodiments of this application. For example, although the implementation of the various components described above can be embodied in a hardware device, it can also be implemented as a purely software solution, such as an installation on an existing server or mobile device.

[0323] Similarly, it should be noted that, in order to simplify the description of the present application and thus aid in the understanding of one or more embodiments of the invention, the foregoing description of the embodiments of the present application sometimes combines multiple features into a single embodiment, drawing, or description thereof. However, this approach of the present application should not be construed as reflecting an intention that the claimed subject matter requires more features than expressly recited in each claim. Rather, the subject of the invention should possess fewer features than in any single embodiment described above. <table class＝"notice-table">Each announcement corresponds to one line. Title and Link: Located in <td class＝"title"><ahref=" / xxgk / 20XX.html"target="_blank"> Announcement of the transfer of exploration rights for a certain gold mine (A1) Extraction logic: for tr in soup.select('table.notice-table tr')[1:]:# Skip the header row title=tr.select_one('td.title a').text.strip() link='https: / / zrzyt.xizang.gov.cn'+tr.select_one('td.title a')['href']# Concatenate the domain name time=tr.select_one('td.time').text.strip()# Format: 20XX-XX-XX 09:30.F6: Announcement list container:

Claims

1. A mining right auction information automatic monitoring and email reminding method, characterized in that, include: S1. Periodically crawl website announcements of target provinces based on edge nodes; deploy edge nodes in the target provinces, collect load characteristic data of the natural resources department websites of the target provinces, perform surface fitting processing to obtain dynamic polling intervals; based on the dynamic polling intervals, collect announcement information of the target provinces, perform parsing processing to obtain a list of announcements of each province containing titles, publication times, and links; S2. Intelligent identification of mining rights bidding and auction related announcements based on edge nodes; The announcement list of each province is filtered by edge nodes based on two-level regular expression matching rules, and combined with the rule weights optimized by surface fitting to obtain the effective announcement list; S3. Processed announcement filtering and deduplication based on edge nodes; The list of valid announcements is compared with the historical announcement data stored locally on the edge nodes to obtain a new list of announcements. S4. Automatically send new announcement emails based on edge nodes; The new announcement list is generated by the edge nodes, which construct email content according to preset rules and send it to the designated recipient through an encrypted transmission channel to complete the reminder. Load characteristic data is collected in S1, and surface fitting processing is performed to obtain the dynamic polling interval, including: For 30 consecutive days, server response times and request success rates for the target province were collected daily for 24 hours. Outlier removal and standardization were performed to obtain a load feature dataset. The load feature dataset was fitted with a bivariate cubic polynomial to construct a response time prediction model and a request success rate prediction model. The outputs of the two prediction models were combined, and the dynamic polling interval for each province at different time periods was calculated and constrained by a preset objective function. The response time prediction model of the bivariate cubic polynomial is as follows: the coefficients are solved by the least squares method to minimize the sum of the squares of the predicted response time and the actual response time; the request success rate prediction model is constructed using the same method to minimize the sum of the squares of the predicted success rate and the actual success rate; the objective function combines the response time weight, success rate weight and the base interval to calculate the polling interval, and adjusts the result with constraints of 10 to 60 seconds. S2's filtering process based on two-level regular expression matching rules includes: The first level consists of exclusion rules, which include 17 categories of non-bidding and auction information and are precisely filtered using regular expressions. The second level consists of core rules, which include key actions, mining rights types, and fault-tolerant matching and identify valid announcements using regular expressions. Announcements that are not excluded and meet the core rules are added to the list of valid announcements. The rule weights in S2 that combine surface fitting optimization include: The matching accuracy of each rule in each province over the past 12 months was collected. A surface fitting was performed based on a bivariate quadratic polynomial to obtain a rule accuracy prediction model. The rule priority was adjusted according to the prediction accuracy: rules with high accuracy ≥ 0.9 were excluded and executed first, and core rules with high accuracy ≥ 0.85 had their confidence weight increased. Finally, the validity of the announcement was determined by the total confidence threshold.

2. The mining right auction information automatic monitoring and mail reminding method according to claim 1, characterized in that, S1 collects and processes announcement information based on dynamic polling intervals, including: Requests are sent to the target province websites sequentially at dynamic polling intervals, using province-specific request headers and embedding random delays. The returned HTML source code is cleaned, comments and escape characters are removed, and parsing rules are customized according to the webpage structure of each province to extract the title, publication time and links and standardize them to form a list of announcements for each province.

3. The method for automatic monitoring and mail reminding of mining right auction information according to claim 1, characterized in that, Historical announcement data stored locally in S3 includes: The system primarily uses CSV files for storage, containing fields such as title, publication time, link, processing time, province code, and mineral type, and is stored in quarterly partitions. A Redis distributed hash table is also built as an in-memory cache, with the title hash value as the key, the existence flag as the value, and an expiration time of 365 days to ensure fast retrieval of historical data.

4. The mining right auction information automatic monitoring and mail reminding method according to claim 3, characterized in that, The comparison process between S3 and locally stored historical announcement data includes: Load historical title hashes from Redis or CSV into an in-memory set; for each announcement in the valid announcement list, calculate the title hash as the primary identifier and the title string as the secondary identifier; if the primary identifier is not in the historical set, it is determined to be a new announcement; if the primary identifier exists, verify whether it is a hash collision through the secondary identifier, and finally obtain a new announcement list and update it synchronously to CSV and Redis.

5. The mining right auction information automatic monitoring and mail reminding method according to claim 1, characterized in that, S4 sends the following via an encrypted transmission channel: The main channel uses the SMTP_SSL protocol, establishes an encrypted connection pool using corporate email, and enables DKIM signing; backup channels include WeChat and SMS; recipient preferences are verified before sending to ensure accurate message delivery.

6. The mining right auction information automatic monitoring and mail reminding method according to claim 5, characterized in that, In S4, email content is constructed according to preset rules, and reminders are completed including: Email titles are prioritized, and the body includes the announcement title, release time, mineral type, estimated area, countdown to deadline, and link. Dynamic retry is initiated when sending fails. After three failures, the email is added to the pending sending queue and an alarm is triggered. At the same time, the sending log is recorded and a weekly report is generated.

Images