Data acquisition exception monitoring method based on multi-granularity state tracking, electronic device and computer readable storage medium

Abstract

The application discloses a data acquisition exception monitoring method based on multi-granularity state tracking, an electronic device and a computer readable storage medium, and comprises the following steps: S1, obtaining multi-granularity structured state logs generated in each execution stage of a data acquisition script running full link; S2, combining a multi-level time window to aggregate and count the multi-granularity structured state logs, and obtaining data acquisition script running statistical indicators corresponding to grouping dimensions; S3, performing multi-type exception determination on the data acquisition script to obtain an exception identification result; S4, respectively establishing an independent state record for each acquisition exception in the exception identification result; and S5, based on the result of the exception full life cycle state tracking, centrally pushing aggregated alarm notification and recovery notification according to a preset fixed time window. The application has the advantages of accurate fault positioning, good alarm noise reduction effect, high data acquisition quality and automated operation efficiency, high data exception discovery efficiency, and continuous tracking.

Description

Technical Field

[0001] This invention belongs to the field of Internet data acquisition and data quality monitoring technology, specifically relating to a data acquisition anomaly monitoring method based on multi-granularity state tracking, an electronic device, and a computer-readable storage medium. Background Technology

[0002] In existing network data acquisition services, a single acquisition task is not completed with a single request, but rather consists of multiple consecutive execution stages, including script startup, task acquisition, page access, proxy switching, result uploading, and field parsing. In actual operation and maintenance, it is common to encounter situations where the script process runs normally and server resource indicators are normal, but the acquisition task fails to execute effectively. Examples include failure to acquire the acquisition task, requests being restricted by the target site, empty upload results, or the target page being redesigned causing continuous parsing failures of core fields. Relying solely on process survival or machine resource monitoring is insufficient to pinpoint exactly where the problem occurs.

[0003] Currently, the mainstream monitoring methods are mainly divided into two categories: First, there is system-level monitoring, which mainly observes basic indicators such as server CPU, memory, disk, and network connectivity. It can only detect machine hardware failures and cannot determine whether the data collection script actually produces valid data. Second, process-level monitoring mainly determines whether the script is running and whether it exits normally. It can only detect script crashes and cannot identify business-related anomalies such as "the script is running but no valid data is collected".

[0004] The aforementioned existing technologies have the following obvious shortcomings in actual data collection scenarios: First, silent failures are difficult to detect in a timely manner: When the target website is redesigned, the collection account is restricted, or the proxy link fails, the script can still start and finish normally, but the amount of uploaded data is 0 or the core fields remain empty. Such anomalies do not manifest as process crashes, but directly affect the quality of downstream data, and usually take a long time to be discovered.

[0005] Second, there is a lack of records of continuous abnormal states: a single log entry can only record a single point of event, and it is impossible to directly determine whether the abnormality is the first occurrence, has lasted for several days, or has been recovered. Operation and maintenance personnel need to manually review logs from multiple days and compare the results of multiple operations in order to determine the severity of the fault and the priority of handling.

[0006] Third, alarms lack reproducible context: If alarm information does not carry key information such as script identifier, task sample, site information, or missing fields, technicians need to search the log system again to locate the problem, which lengthens the troubleshooting and recovery process. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a data acquisition anomaly monitoring method, electronic device, and computer-readable storage medium based on multi-granularity state tracking. It constructs a complete monitoring system from five dimensions: multi-granularity data entry throughout the acquisition process, time-window aggregation statistics, multi-type anomaly determination, anomaly lifecycle tracking, and time-segmented aggregation alarms. This solves the problems of ambiguous anomaly location, high alarm noise, and delayed detection of data quality anomalies in existing monitoring systems. It achieves accurate anomaly location, continuous state tracking, and low-noise alarms, thereby improving data acquisition quality and automated operation and maintenance efficiency.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A data acquisition anomaly monitoring method based on multi-granularity state tracking includes the following steps: S1. Obtain multi-granularity structured status logs generated at each execution stage of the entire data acquisition script execution chain. Each log entry of the multi-granularity structured status log carries a unified association identifier containing the unique identifier of the data acquisition script. S2. Based on the unified association identifier, the data collection script, country or site are used as grouping dimensions, and the multi-granularity structured status log is aggregated and statistically analyzed in combination with multi-level time windows to obtain the data collection script operation statistics indicators for the corresponding grouping dimensions. S3. Based on the preset judgment threshold and the running statistical indicators, perform multi-type anomaly judgment on the data acquisition script to obtain anomaly identification results; S4. Establish an independent status record for each collected anomaly in the anomaly identification results, and perform full life cycle status tracking of the anomaly based on the independent status record, and update the anomaly status information in real time. S5. Based on the results of the abnormal full life cycle status tracking, aggregated alarm notifications and recovery notifications are pushed centrally according to a preset fixed time window, and repeated notifications of the same abnormality are suppressed according to the independent status records.

[0009] In one embodiment, in step S1, the multi-granularity structured status log is output by a lightweight monitoring component deployed on the data acquisition terminal. The lightweight monitoring component supports automatic loading when the interpreter starts or explicit invocation by the data acquisition script. The unified association identifier also includes a timestamp and a task sample. The multi-granularity structured status log is linked together through the unified association identifier to form a traceable operation link from the start of the data collection script to the upload of the result. The multi-granularity structured status log contains at least five levels of status points: script startup granularity for recording the script entry running status, task acquisition granularity for recording task acquisition results, request access granularity for recording target site access results, proxy access granularity for recording proxy link running status, and result upload and field integrity granularity for recording data upload results and field integrity.

[0010] In one embodiment, the multi-level time window in step S2 includes an hourly window and a daily window. The hourly window is used for real-time anomaly detection and recovery status confirmation, and the daily window is used for full-scale operation data statistics and formal anomaly determination. The aggregated statistics of the data collection script operation include at least the following: number of startup records, number of tasks acquired, number of successful requests, number of failed requests, number of uploaded records, and the number of times each data field is missing.

[0011] In one embodiment, in step S3, the anomaly identification results obtained through multi-type anomaly determination correspond to at least three types of collection anomalies: startup anomaly, zero data anomaly, and field missing anomaly. The determination rules for startup anomaly, zero data anomaly, and field missing anomaly are as follows: Startup error: If there is a startup record but no task acquisition record, no request record, no upload record, and there is still no task acquisition, access request, or upload activity in the confirmation window looking back 180 minutes, it is determined to be a startup error. 0 data anomalies: There are task acquisition or request records, but no upload records, which is judged as 0 data anomalies; Field missing anomaly: If there are uploaded records within the daily window, and the number of missing records for the target data field is greater than or equal to the total number of uploaded records for that day, the target data field is determined to be completely missing anomaly.

[0012] In one embodiment, the independent status record in step S4 includes at least the following information fields: unique identifier of data acquisition script, name of data acquisition script, country or site, task sample, first discovery time, most recent occurrence time, recovery time, abnormal activity marker, cumulative number of times per hour, number of daily notifications, date of most recent daily notification and time of most recent notification; The flow rules for tracking the status of anomalies throughout their entire lifecycle are as follows: when an anomaly first hits the judgment rule, an independent status record is inserted and marked as active, the first discovery time and the most recent occurrence time are written, and the hourly cumulative count is set to 1; When the time window continues to match, update the most recent occurrence time and accumulate the number of occurrences by the hour; If the anomaly detection rule is no longer met in subsequent time windows, the system is marked as inactive and the recovery time is written.

[0013] In one embodiment, the preset fixed time window in step S5 includes a daily alarm window and a daily recovery window: Among them, 9:00 a.m. is the daily alarm window. Active abnormal records are filtered by country or site. Push aggregated alarm notifications are initiated for abnormal records that have lasted for 24 hours and have not been notified on the same day. Push aggregated alarm notifications are also initiated for abnormal records with zero data and abnormal records with missing fields that have lasted for 48 hours and have not been notified on the same day. If an anomaly is not resolved after the initial notification, a second alert will be sent to the daily alert window the following day. The daily recovery window opens at 2:00 PM daily. Abnormal records that have been converted to inactive status are filtered by country or site, and recovery notifications are pushed out uniformly. If the anomaly is resolved after the end of the daily recovery window, it will be postponed to the next daily recovery window.

[0014] In one embodiment, in step S5, alarm notification and recovery notification support a two-level routing mechanism, including country routing that matches the notification channel according to the country or site attribute corresponding to the data collection script, and subscriber routing that matches the person in charge of processing according to the unique identifier of the data collection script and the anomaly type. The alarm and recovery notifications should include the data collection script name, country or site, type of collection anomaly, first discovery time, cumulative number of times per hour, missing fields, and task sample information. By using the most recent daily notification date and time in the independent status record, the system restricts repeated push notifications for the same anomaly within the same time window.

[0015] In one embodiment, the method also includes an anomaly classification and closed-loop verification steps: for different types of collected anomalies, a corresponding investigation and processing path is matched. After the anomaly is repaired, the anomaly is automatically verified by the aggregated statistical results of the subsequent time window to determine whether the anomaly has been recovered. The independent status records corresponding to the anomaly's full lifecycle status tracking are updated synchronously to form a closed-loop process of monitoring, alarming, repairing and verification.

[0016] The present invention also discloses an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps: S1. Obtain multi-granularity structured status logs generated at each execution stage of the entire data acquisition script execution chain. Each log entry of the multi-granularity structured status log carries a unified association identifier containing the unique identifier of the data acquisition script. S2. Based on the unified association identifier, the data collection script, country or site are used as grouping dimensions, and the multi-granularity structured status log is aggregated and statistically analyzed in combination with multi-level time windows to obtain the data collection script operation statistics indicators for the corresponding grouping dimensions. S3. Based on the preset judgment threshold and the running statistical indicators, perform multi-type anomaly judgment on the data acquisition script to obtain anomaly identification results; S4. Establish an independent status record for each collected anomaly in the anomaly identification results, and perform full life cycle status tracking of the anomaly based on the independent status record, and update the anomaly status information in real time. S5. Based on the results of the abnormal full life cycle status tracking, aggregated alarm notifications and recovery notifications are pushed centrally according to a preset fixed time window, and repeated notifications of the same abnormality are suppressed according to the independent status records.

[0017] The present invention also discloses a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it performs the following steps: S1. Obtain multi-granularity structured status logs generated at each execution stage of the entire data acquisition script execution chain. Each log entry of the multi-granularity structured status log carries a unified association identifier containing the unique identifier of the data acquisition script. S2. Based on the unified association identifier, the data collection script, country or site are used as grouping dimensions, and the multi-granularity structured status log is aggregated and statistically analyzed in combination with multi-level time windows to obtain the data collection script operation statistics indicators for the corresponding grouping dimensions. S3. Based on the preset judgment threshold and the running statistical indicators, perform multi-type anomaly judgment on the data acquisition script to obtain anomaly identification results; S4. Establish an independent status record for each collected anomaly in the anomaly identification results, and perform full life cycle status tracking of the anomaly based on the independent status record, and update the anomaly status information in real time. S5. Based on the results of the abnormal full life cycle status tracking, aggregated alarm notifications and recovery notifications are pushed centrally according to a preset fixed time window, and repeated notifications of the same abnormality are suppressed according to the independent status records.

[0018] By adopting the above technical solution, the present invention has the following beneficial effects: (1) More accurate fault location: This invention uses multi-level granular status tracking points for data acquisition script startup, task acquisition, request access, proxy access, result upload, and field integrity to accurately locate the specific stage of the acquisition link where the anomaly occurs. Maintenance personnel do not need to check each link one by one and can start processing directly from the breakpoint, which greatly shortens the fault location time. (2) Quantifiable anomaly judgment: All anomaly rules in this invention are based on statistically significant numerical indicators, replacing the manual method of checking logs based on experience. The judgment standard is uniform and configurable, adapting to the monitoring sensitivity requirements of different services and reducing human judgment errors. (3) Status closed-loop traceability: This invention fully records the entire life cycle of an anomaly from its first appearance, continuous activity to recovery. The duration of the anomaly and the recovery status can be viewed directly, which is convenient for assessing the scope of the fault impact and the processing priority, without the need for manual comparison of logs from multiple time periods; (4) Significant alarm noise reduction effect: The present invention adopts the mechanism of detecting anomalies in hourly windows and pushing them in daily windows. Combined with the duration threshold, it can filter out invalid alarms caused by short-term network jitter and instantaneous fluctuations, while ensuring that continuous faults are not missed, and adapts to the daily processing rhythm of maintenance personnel. (5) Operation and maintenance process forms a closed loop: The alarm information of the present invention carries a complete context and task sample, and the operation and maintenance personnel can quickly verify the problem; after the anomaly is repaired, the system automatically verifies the recovery status and updates the record, forming a complete closed loop of "monitoring, alarm, repair and verification", and the level of automated operation and maintenance is high; In summary, this invention has the advantages of accurate fault location, good alarm noise reduction effect, high data acquisition quality and high efficiency of automated operation and maintenance, high efficiency of data anomaly detection, and continuous tracking. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the data acquisition anomaly monitoring method based on multi-granularity state tracking of the present invention. Figure 2 This is a schematic diagram of the architecture of the monitoring system used in this invention; Figure 3 This is a flowchart illustrating the multi-granularity anomaly detection logic of the present invention; Figure 4 This is a schematic diagram of the state transition of the abnormal life cycle of the present invention; Figure 5 This is a schematic diagram of the time-segmented aggregated alarm notification process of the present invention. Detailed Implementation

[0020] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0021] The components of the embodiments of the invention described and shown in the accompanying drawings can typically be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention.

[0022] Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0024] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0025] Example 1 This embodiment proposes a data acquisition anomaly monitoring method, electronic device, and computer-readable storage medium based on multi-granularity state tracking. This method not only checks whether the data acquisition script process exists, but also records several states that truly affect the output during the script's execution: whether it has entered the execution entry point, whether it has retrieved a task, whether it has initiated a request, whether the request was successful, whether the upload was completed, and whether empty fields appear in the upload result. The monitoring and alarm terminal then aggregates and judges based on the data acquisition script, country or site, time window, and anomaly type. In simpler terms, this invention further refines the assessment from "Is the data acquisition script still alive?" to "Where is the data acquisition script stuck? Has there been any output in this hour? How long has the anomaly lasted? Has it recovered?" The core technical problems solved by this invention are ambiguous anomaly localization, high alarm noise, and delayed detection of data quality anomalies.

[0026] This invention can be widely applied to the following specific scenarios: 1. Automated operation and maintenance of large-scale distributed web crawler clusters This system is suitable for maintaining a large number of data collection scripts from different sites, countries, or categories simultaneously. It can detect issues such as data collection scripts being started but not yet entering task processing, batch data collection scripts having zero data, and similar fields being collectively missing, and displays these issues centrally through a dashboard.

[0027] 2. Cross-category e-commerce data collection and competitor analysis This system is suitable for data collection scenarios with high requirements for fields such as product details, price, reviews, inventory, and images. When a page redesign results in empty fields such as price, review content, or image list, the system can identify the missing fields during the data collection phase, rather than waiting until downstream analysis. Addressing the issue of "0 data" or "missing core fields (such as price and Review_content)" caused by frequent e-commerce website redesigns, this invention provides precise anomaly location capabilities, ensuring the integrity of market intelligence data.

[0028] 3. Social media sentiment monitoring and data mining This invention is suitable for data collection scenarios involving significant fluctuations in account status, proxy quality, and target site access on platforms such as Twitter, YouTube, TikTok, and Reddit. Due to the ever-changing anti-scraping strategies of social media and frequent network fluctuations, the time-segmented aggregated alert mechanism of this invention (daily alert at 9:00 AM / recovery at 2:00 PM) effectively filters noise caused by momentary network jitter, helping operators focus on handling persistent account bans or IP restrictions. Through duration thresholds and daily aggregated notifications, short-term network jitter can be filtered out, allowing attention to be focused on persistently failing accounts, proxies, or sites.

[0029] 4. Data Quality Management (DQA) and Cleaning This invention is applicable to business systems with strict requirements on the structure of collected data. By monitoring "Missing Fields," it can detect parsing rule failures caused by minor adjustments to the webpage structure before data is entered into the database, ensuring the validity of core business fields (such as Helpful_count, Review_images, etc.) and reducing downstream data cleaning costs.

[0030] like Figures 1-5 As shown, in one embodiment of the present invention, the overall data acquisition anomaly monitoring method based on multi-granularity state tracking of the present invention comprises the following steps: Constructing a multi-granularity state acquisition system: This embodiment is based on an existing distributed crawler data collection system. After a lightweight monitoring component is installed on the data collection terminal, the lightweight monitoring component can be automatically injected and loaded when the Python interpreter starts, or it can be explicitly called by the data collection script through import. There is no need to make large-scale modifications to the existing data collection script, and the access cost is low.

[0031] During the execution of the data acquisition script, the lightweight monitoring component automatically outputs structured status logs at the following nodes: script startup, task acquisition, request access, proxy access, result upload, and field validation. These logs are written to a unified logging system or a time-series metrics database. Each structured status log carries a unified association identifier, specifically including a unique identifier for the data acquisition script, a timestamp, and a task sample. This unified association identifier connects status points at different granularities, forming a complete and traceable execution chain from script startup to result upload. Subsequent monitoring terminals do not need to replay business logic; they can complete anomaly detection simply by aggregating the status logs.

[0032] In this embodiment, the multi-granularity structured state log sets up a total of five levels of state points: Data acquisition script startup granularity: The corresponding status point is used to record whether the data acquisition script has entered the running entry point, carrying the unique identifier of the data acquisition script, the name of the data acquisition script, the path of the data acquisition script, and business platform information, which is used to identify the failure of the data acquisition script being loaded but not executing the subsequent acquisition link.

[0033] Task acquisition granularity: The corresponding status point is used to record whether the data acquisition script has obtained the task to be collected from the task queue or executor, and saves the task sample URL and task parameters for subsequent location of the specific target page, product or link to be collected.

[0034] Request access granularity: The corresponding status point is used to record the access behavior and execution result of the data collection script to the target site. The status result is divided into success and failure, and carries the request method, HTTP response status code and target host information.

[0035] Proxy access granularity: The corresponding status points are implemented in two forms: event-type and count-type. They are used to record the running status of the proxy link used in the collection process, carrying information such as proxy host, request method, and response status code, in order to distinguish between target site abnormalities and proxy link abnormalities.

[0036] Results upload and field integrity granularity: The corresponding status points are divided into two parts: upload status and empty field status. The upload status records the number of upload records generated by the data collection script, and the empty field status records the list of empty fields and the number of missing fields in the upload records, which is used to identify data output anomalies and field parsing anomalies.

[0037] Time-Window-Based State Aggregation and Anomaly Detection: The monitoring and alarm module aggregates and statistically analyzes multi-granularity structured status logs using two levels: hourly windows and daily windows. The hourly window has a default duration of 60 minutes and is used for real-time anomaly detection and recovery status confirmation; the daily window has a default duration of 1440 minutes and is used for full-scale data statistics and formal anomaly determination.

[0038] Taking an hourly window as an example, for each data collection script, the number of start records (denoted as I), the number of tasks acquired (denoted as T), the number of successful requests (denoted as R success), the number of failed requests (denoted as R failure), the number of uploaded records (denoted as U), and the number of missing records for each field (denoted as k) within the hour (denoted as M_k).

[0039] For example, if a data acquisition script has I=1, T=0, total requests=0, and U=0 for the current hour, and there are no tasks, requests, or upload records when looking back 180 minutes, it can be determined that the data acquisition script did not enter the execution chain after starting. If a data acquisition script has T=24, total requests=31, and U=0, it means that the data acquisition script has executed data acquisition and access, but has not produced any upload results. By comparing the above indicators, it is possible to directly determine whether the fault breakpoint is located at the data acquisition script entry point, task scheduling, network access, or result parsing stage.

[0040] The aggregated statistics are grouped by data collection script, country, or site, and the following operational statistics are calculated for each group: number of startup records, number of tasks acquired, number of successful requests, number of failed requests, number of uploaded records, and the number of missing data fields.

[0041] Based on the above operational statistics and preset judgment thresholds, the system performs multi-type anomaly judgment on the data acquisition script, identifying three types of acquisition anomalies. Aggregated statistics are triggered when the window ends; statistics are triggered once every 60 minutes for hourly windows and once every 24 hours for daily windows. The above operational statistics can be aggregated and calculated from the log retrieval system or directly read from the time-series indicator storage system. The specific judgment rules are as follows: Startup Anomaly Detection: When a startup record exists, but there are no task acquisition records, no request records, and no upload records, the system looks back 180 minutes into the confirmation window. If there are still no task acquisition, request access, or upload activities in this window, it is determined to be a startup anomaly. This type of anomaly usually corresponds to problems such as abnormal data collection script entry points, invalid scheduling configurations, and missing dependency libraries. For example, if a data collection script has 1 startup record in the current hour, but the number of task acquisitions, the total number of requests (including successful and failed requests), and the number of upload records are all 0, and there is still no collection activity when looking back 180 minutes, it can be determined to be a startup anomaly. Specifically, at 9:00 AM every day, the system queries the previous day's data by country. If any of the three types of anomalies has not been resolved within 24 hours, a unified alarm is sent; if it is not handled after the first notification, a second alarm is sent at 9:00 AM the following day.

[0042] Zero Data Anomaly Detection: When there are task acquisition or request records but no upload records, it is considered a zero data anomaly. This rule indicates that the data collection script has started processing the task or accessing the target site, but no valid upload result has been generated. Common reasons include target page redesign, account or IP restrictions, request timeout, and empty parsing results. The system synchronously retains the most recent task sample URLs for maintenance personnel to directly access and verify issues. That is, it is triggered when T>0 or the total number of requests>0 and U=0. Specifically, data is monitored daily at 14:00 by country. If the anomaly has been continuously recovered within 24 hours, a unified recovery notification will be sent; if it is recovered after 14:00, the notification will be sent at 14:00 the following day.

[0043] Field Missing Anomaly Detection: In this solution, field missing anomaly specifically refers to the situation where a field is completely missing. Within the daily window, when there are uploaded records and the number of missing values ​​for the target data field is greater than or equal to the total number of uploaded records that day, that is, when U24 > 0 within the daily window and M(k) ≥ U24 for field k, a field missing anomaly (complete missing state) is triggered. This state usually corresponds to page structure adjustments, CSS selector failure, field mapping errors, or the disappearance of a partial module of the target page. For example, if a data collection script uploads 1286 product records on a given day, where fields such as title, price, and rating all have valid values, but the product image field is missing 1286 times, it can be determined that it is not a case of individual products missing images, but rather that the page module or parsing rules corresponding to the image are completely invalid.

[0044] Establish an abnormal lifecycle state tracking mechanism: The system establishes an independent status record for each identified data collection anomaly in the status storage database, storing them in separate tables according to anomaly type. Each independent status record contains the following information fields: unique identifier of the data collection script, data collection script name, country or site, task sample, first discovery time, most recent occurrence time, recovery time, anomaly activity flag, cumulative number of occurrences per hour, number of daily notifications, most recent daily notification date, and most recent notification time. Records with zero data anomalies and field missing anomalies additionally store the task sample, while records with field missing anomalies additionally store the name of the missing field.

[0045] The three types of exceptions are implemented using a categorized, same-database storage approach: startup exception records do not require storing task samples and missing field information; zero-data exception records additionally store task sample information for problem reproduction; and field missing exception records additionally store task samples and missing field name information for locating the location where the parsing rule fails.

[0046] The above-mentioned state transitions are all triggered and updated in hourly windows. After the aggregation statistics and anomaly judgment of each hourly window are completed, the corresponding state record addition or update operation is uniformly executed to achieve near real-time anomaly state tracking at the hourly level.

[0047] The flow rules for tracking the entire lifecycle of anomalies are as follows: First hit: When a new anomaly is detected in the hourly window, insert an independent status record, set the anomaly activity flag to active, write the first discovery time and the most recent occurrence time, and set the hourly cumulative count to 1.

[0048] Continuous hits: If the anomaly is detected in subsequent hourly windows, the most recent occurrence time in the record is updated, and the cumulative number of hits per hour is incremented by 1.

[0049] Anomaly Recovery: If the anomaly is not detected within a consecutive hourly window, the anomaly activity flag is set to inactive and the recovery time is written.

[0050] Notification Updates: After each push of an alarm or recovery notification, update the daily notification count, the date of the most recent daily notification, and the time of the most recent notification.

[0051] Based on the above status records, a visual monitoring panel can be built. The panel is divided into an alarm area and a recovery area. The alarm area displays all active anomalies, while the recovery area displays anomalies that have been marked as recovered. The panel displays the following fields: data collection script name, unique identifier of the data collection script, country or site, alarm type, missing field, cumulative number of alarms per hour, first discovery time, most recent occurrence time, recovery time, and task sample link. The cumulative number of alarms per hour directly reflects the persistence of the anomaly, and the task sample supports one-click access to verify the issue.

[0052] Establish a time-segmented aggregated alarm and routing notification strategy: This embodiment decouples anomaly detection from notification sending: The first level is a multi-level time window at the time granularity level, where the hourly window is responsible for real-time anomaly detection, status accumulation, and recovery detection; the daily window is responsible for full data statistics. The second level consists of functional nodes within the daily window, with two fixed functional windows set up within the daily window to centrally push notifications, avoiding a large number of fragmented alarms caused by short-term fluctuations.

[0053] Set two fixed time windows each day to perform alarm push notifications and recovery checks respectively: The daily alarm window is set to 9:00 AM, and the execution logic is as follows: Iterate through all active anomaly records and calculate the duration of the anomaly. An anomaly is considered complete if its initial discovery time is earlier than or equal to 9:00 AM the previous day (lasting ≥ 24 hours); anomalies with zero data or missing fields are considered complete if their initial discovery time is earlier than or equal to 9:00 AM the previous two days (lasting ≥ 48 hours). Anomalies that meet the duration threshold and whose most recent daily notification date is not the current day are added to the daily aggregated notification list. For missing field anomalies, an additional condition of U24 > 0 and M(k) ≥ U24 must be met to prevent a small number of records from triggering alarms due to missing characters.

[0054] The daily window is set to resume at 14:00 daily, and the execution logic is as follows: Iterate through all inactive exception records. If an exception has been continuously recovered for 24 hours and no recovery notification has been pushed, send a unified recovery notification and update the exception status in the dashboard and collaboration tools. If an exception is recovered after 14:00 on the same day, the recovery notification will be pushed at 14:00 the following day.

[0055] Duplicate notification suppression mechanism: The system uses "unique identifier of data collection script + country or site + anomaly type + missing field name" as the unique identification of the same anomaly. The "recent daily notification date" field in the independent status record marks the anomalies that have been notified on the same day. The same anomaly is pushed only once in the same function window to avoid duplicate alarms and reduce alarm noise.

[0056] In this embodiment, alarm messages are pushed to the enterprise's instant messaging tool. Each alarm message includes at least: data collection script name, unique identifier of data collection script, country / site, alarm type, first discovery time, cumulative duration in hours, and task sample link; for field missing alarms, the names of the missing fields are listed in addition, such as "comment content, product main image, rating".

[0057] The notification supports a two-level routing mechanism: Country routing: Based on the country and language attributes of the site corresponding to the data collection script, it maps to the corresponding regional operation and maintenance notification group; Subscriber routing: Matches the corresponding responsible person or special maintenance group based on the unique identifier of the data collection script, the configuration of monitoring personnel, and the anomaly category.

[0058] For startup anomalies that require manual follow-up, the system can also synchronously write the anomaly information into the collaborative task table and automatically generate a work order to be processed; after the anomaly is recovered, the work order status is automatically updated to resolved, realizing the linkage between alarms and task flow.

[0059] Establish a closed-loop system for anomaly classification, handling, and maintenance: Based on the above monitoring system, maintenance personnel can handle anomalies according to standardized procedures, forming a complete closed-loop maintenance system. Receive alerts: Check aggregated alert messages at 9:00 AM daily, or view the list of active anomalies in the visual monitoring panel.

[0060] Qualitative investigation: A three-tiered handling mechanism is adopted, prioritizing and processing timelines based on anomaly type, duration, and field importance. Level 1 Anomaly (Highest Priority, to be handled within 24 hours): Startup anomaly, which directly affects the overall execution of the data collection task. Prioritize checking server status, data collection script dependencies, scheduling configuration, and execution entry point. Level 2 anomalies (medium priority, to be processed within 48 hours): 0 data anomalies, affecting single script data output. Prioritize clicking the task sample link to verify whether the target page is accessible, whether the page structure has been changed, and whether the account or IP is restricted. Level 3 Exceptions (Regular Priority, Iterative Processing): Field Missing Exceptions. First, determine if the missing field is a core business field. If it is a core field, immediately fix the parsing rules. If it is not a core field, it can be included in the regular optimization iteration.

[0061] Among them, 0 data anomalies: prioritize clicking the task sample link to verify whether the target page can be accessed normally, whether the page structure has been changed, and whether the account or IP is restricted; Startup error: Highest priority, first check server status, data collection script dependencies, scheduling configuration and execution entry point; Field missing anomaly: First determine whether the missing field is a core business field. If it is a core field, immediately fix the parsing rules. If it is not a core field, it can be included in the regular optimization iteration.

[0062] Repair Verification: After technicians repair the data collection script, agent, account, or parsing rules, no manual confirmation is required; the system will automatically recalculate the statistical indicators in subsequent hourly windows.

[0063] Recovery Confirmation: Maintenance personnel can view the recovery notification after the daily recovery window at 14:00 on the same day, or confirm in the monitoring panel whether the anomaly has been transferred from the alarm area to the recovery area.

[0064] After the anomaly is repaired, the system automatically verifies whether the anomaly has been recovered by aggregating statistical results in subsequent time windows, and synchronously updates the recovery time and activity flag in the independent status record, forming a complete closed loop of "monitoring, alarming, repairing, and verification".

[0065] This approach allows for more specific fault location: by using status points such as startup, task, request, proxy, upload, and empty fields, it is possible to determine whether the anomaly occurred at the data acquisition script entry point, task queue, network access, proxy link, result output, or field parsing stage. After seeing the alarm, troubleshooters do not need to guess whether it is a server, proxy, or parsing problem; they can start checking from the breakpoint according to the status link.

[0066] The system can calculate exception rules: it uses the number of startup records, the number of tasks acquired, the number of successful requests, the number of failed requests, the number of uploaded records, and the number of times fields are missing to make judgments, reducing the need for manual log sifting based on experience.

[0067] The status loop is clear: the first occurrence of the anomaly, the continuous hits, the recovery, the number of notifications and the time of the most recent notification are all recorded, which makes it easy to determine the processing priority and recovery status.

[0068] Alerts are better aligned with the processing rhythm: hourly window statistics tasks are responsible for detection and recovery, continuous anomalies are sent out in a concentrated manner at 9:00 am every day, and recovery is checked in a concentrated manner at 2:00 pm, reducing fragmented notifications.

[0069] This embodiment provides an electronic device, including a memory and a processor, which communicate with each other via a communication bus. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the steps of the data acquisition anomaly monitoring method based on multi-granularity state tracking described in the above embodiment.

[0070] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the above-described data acquisition anomaly monitoring method based on multi-granularity state tracking. The storage medium can be any medium capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory, a random access memory, a magnetic disk, or an optical disk.

[0071] This embodiment does not impose any limitation on the shape, material, structure, etc. of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention shall fall within the protection scope of the technical solution of the present invention.

Claims

1. A data acquisition anomaly monitoring method based on multi-granularity state tracking, characterized in that, Includes the following steps: S1. Obtain multi-granularity structured status logs generated at each execution stage of the entire data acquisition script execution chain. Each log entry of the multi-granularity structured status log carries a unified association identifier containing the unique identifier of the data acquisition script. S2. Based on the unified association identifier, the data collection script, country or site are used as grouping dimensions, and the multi-granularity structured status log is aggregated and statistically analyzed in combination with multi-level time windows to obtain the data collection script operation statistics indicators for the corresponding grouping dimensions. S3. Based on the preset judgment threshold and the running statistical indicators, perform multi-type anomaly judgment on the data acquisition script to obtain anomaly identification results; S4. Establish an independent status record for each collected anomaly in the anomaly identification results, and perform full life cycle status tracking of the anomaly based on the independent status record, and update the anomaly status information in real time. S5. Based on the results of the abnormal full life cycle status tracking, aggregated alarm notifications and recovery notifications are pushed centrally according to a preset fixed time window, and repeated notifications of the same abnormality are suppressed according to the independent status records. In step S1, the multi-granularity structured status log is output by a lightweight monitoring component deployed on the data acquisition terminal. The lightweight monitoring component supports automatic loading when the interpreter starts or explicit invocation by the data acquisition script. The unified association identifier also includes a timestamp and a task sample. The multi-granularity structured status log is linked together through the unified association identifier to form a traceable operation link from the start of the data collection script to the upload of the result. The multi-granularity structured status log contains at least five levels of status points: script startup granularity for recording the script entry running status, task acquisition granularity for recording task acquisition results, request access granularity for recording target site access results, proxy access granularity for recording proxy link running status, and result upload and field integrity granularity for recording data upload results and field integrity. In step S3, the anomaly identification results obtained through multi-type anomaly determination correspond to at least three types of collection anomalies: startup anomaly, zero data anomaly, and field missing anomaly. The specific determination rules for startup anomaly, zero data anomaly, and field missing anomaly are as follows: Startup error: If there is a startup record but no task acquisition record, no request record, no upload record, and there is still no task acquisition, access request, or upload activity in the confirmation window looking back 180 minutes, it is determined to be a startup error. 0 data anomalies: There are task acquisition or request records, but no upload records, which is judged as 0 data anomalies; Field missing anomaly: If there are uploaded records within the daily window, and the number of missing records for the target data field is greater than or equal to the total number of uploaded records for that day, the target data field is determined to be completely missing anomaly.

2. The data acquisition anomaly monitoring method based on multi-granularity state tracking according to claim 1, characterized in that: The multi-level time window in step S2 includes an hourly window and a daily window. The hourly window is used for real-time anomaly detection and recovery status confirmation. The daily window is used for full-scale operation data statistics and formal anomaly determination. The aggregated statistics of the data collection script operation include at least the following: number of startup records, number of tasks acquired, number of successful requests, number of failed requests, number of uploaded records, and the number of times each data field is missing.

3. The data acquisition anomaly monitoring method based on multi-granularity state tracking according to claim 1, characterized in that: The independent status record in step S4 includes at least the following information fields: unique identifier of data acquisition script, name of data acquisition script, country or site, task sample, first discovery time, most recent occurrence time, recovery time, abnormal activity marker, cumulative number of times per hour, number of daily notifications, date of most recent daily notification and time of most recent notification; The flow rules for tracking the status of anomalies throughout their entire lifecycle are as follows: when an anomaly first hits the judgment rule, an independent status record is inserted and marked as active, the first discovery time and the most recent occurrence time are written, and the hourly cumulative count is set to 1; When the time window continues to match, update the most recent occurrence time and accumulate the number of occurrences by the hour; If the anomaly detection rule is no longer met in subsequent time windows, the system is marked as inactive and the recovery time is written.

4. The data acquisition anomaly monitoring method based on multi-granularity state tracking according to claim 1, characterized in that: The preset fixed time window in step S5 includes a daily alarm window and a daily recovery window: Among them, 9:00 a.m. is the daily alarm window. Active abnormal records are filtered by country or site. Push aggregated alarm notifications are initiated for abnormal records that have lasted for 24 hours and have not been notified on the same day. Push aggregated alarm notifications are also initiated for abnormal records with zero data and abnormal records with missing fields that have lasted for 48 hours and have not been notified on the same day. If an anomaly is not resolved after the initial notification, a second alert will be sent to the daily alert window the following day. The daily recovery window opens at 2:00 PM daily. Abnormal records that have been converted to inactive status are filtered by country or site, and recovery notifications are pushed out uniformly. If the anomaly is resolved after the end of the daily recovery window, it will be postponed to the next daily recovery window.

5. The data acquisition anomaly monitoring method based on multi-granularity state tracking according to claim 1, characterized in that: In step S5, alarm notifications and recovery notifications support a two-level routing mechanism, including country routing that matches the notification channel based on the country or site attribute corresponding to the data collection script, and subscriber routing that matches the person in charge of processing based on the unique identifier of the data collection script and the anomaly type. The alarm and recovery notifications should include the data collection script name, country or site, type of collection anomaly, first discovery time, cumulative number of times per hour, missing fields, and task sample information. By using the most recent daily notification date and the most recent notification time in the independent status record, the same exception can be restricted from being pushed repeatedly within the same time window.

6. The data acquisition anomaly monitoring method based on multi-granularity state tracking according to claim 1, characterized in that: It also includes anomaly classification and closed-loop verification steps: matching the corresponding investigation and processing path for different types of collected anomalies, and automatically verifying whether the anomaly has been recovered through the aggregated statistical results of subsequent time windows after the anomaly is repaired, and synchronously updating the independent status records corresponding to the anomaly's full lifecycle status tracking, forming a closed loop of monitoring, alarming, repair and verification.

7. An electronic device, characterized in that: The method includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the data acquisition anomaly monitoring method based on multi-granularity state tracking as described in any one of claims 1-6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the data acquisition anomaly monitoring method based on multi-granularity state tracking as described in any one of claims 1-6.

Images