A standardized cleaning method and related apparatus for multi-source alarm data access.

Abstract

This application discloses a standardized cleaning method and related apparatus for multi-source alarm data access, relating to the field of IT operations and maintenance technology in the financial industry. The method obtains user-configured synchronization task parameters, including task execution plans, multi-protocol compatible processing classes, parsing and mapping rules, and initiates a task to receive multi-source alarm data. Dirty data is filtered through processing class matching, parsed into indexable objects, mapped to a unified structure data, and then formatted (including type conversion, null value filling, integrity and consistency checks) to complete standardized cleaning. This method achieves multi-protocol compatibility and unified access to multi-source data without the need for customized scripts, reducing technical barriers and maintenance costs. Through data filtering and standardized processing, it improves data quality and operational efficiency, facilitates rapid response to critical alarms, and ensures the continuity of core financial businesses.

Description

Technical Field

[0001] This application relates to the field of IT operations and maintenance technology in the financial industry, and in particular to a standardized cleaning method and related apparatus for multi-source alarm data access. Background Technology

[0002] In the financial sector, including funds and securities, IT systems must ensure the stable operation of core businesses such as trading, clearing, risk control, and custody. Their IT operations and maintenance face key challenges in alarm management: alarm sources are highly dispersed, involving multiple monitoring tools such as Zabbix and Prometheus, and employing more than ten different protocols including SNMP Trap, Syslog, Kafka messages, and custom APIs, making unified alarm data management difficult; in existing technologies, financial institutions often use custom-developed scripts to interface with alarm data, requiring developers to possess professional programming skills and business understanding, and the significant differences in data formats across systems result in extremely poor script reusability; simultaneously, script maintenance requires continuous manpower investment, execution history and logs are difficult to manage uniformly, and troubleshooting efficiency is low, ultimately leading to low operational efficiency, inability to quickly respond to critical alarms, and impact on business continuity. Summary of the Invention

[0003] The purpose of this application is to provide a standardized cleaning method and related apparatus for multi-source alarm data access, which can achieve multi-protocol compatibility and unified access of multi-source data.

[0004] To achieve the above objectives, this application provides the following solution:

[0005] Firstly, this application provides a standardized cleaning method for multi-source alarm data access, including the following steps:

[0006] Obtain the synchronization task parameters configured by the user; the synchronization task parameters include the synchronization task execution plan, several protocol processing classes adapted to the data source, the parsing rules of the source alarm data, and the standardized mapping rules.

[0007] Start the multi-source alarm data synchronization task to receive multi-source alarm data; multi-source alarm data includes processing class identifier, source alarm data, additional attributes and related information.

[0008] The multi-source alarm data is matched with the protocol processing class based on the processing class identifier. If the match fails, the corresponding multi-source alarm data is judged as dirty data and discarded.

[0009] For successfully matched multi-source alarm data, the data is parsed according to the corresponding parsing rules, converting the structured or unstructured source alarm data into indexable objects.

[0010] Based on standardized mapping rules, indexable objects are mapped to data with a unified structure.

[0011] The data with a unified structure is formatted to complete the standardization and cleaning of alarm data from multiple sources; the formatting process includes data type conversion, null value filling, integrity verification, and consistency verification.

[0012] Optionally, the synchronization task parameters configured by the user are obtained, specifically including the following steps:

[0013] Obtain the synchronous task execution plan input by the user and determine the capability type of the system being connected; capability types include support for sending data to external systems and support for API queries.

[0014] If the system being connected supports sending data out, select at least one of the following active push protocol processing classes: RESTAPI, WebHook, or SNMP.

[0015] If the system being integrated supports API queries, select at least one passive polling protocol processing class from RESTAPI, JDBC / ODBC, and LOG.

[0016] Set a unique identifier for the synchronization task to associate the protocol processing class with the multi-source alarm data synchronization task.

[0017] The user-input parsing rules are received through a visual field relationship configuration interface. These parsing rules define the field traversal method for structured data or the text extraction logic for unstructured data, in order to convert the source data into indexable objects.

[0018] The user inputs standardized mapping rules through a visual field relationship configuration interface. The standardized mapping rules are used to define the correspondence between the attribute fields of indexable objects and the preset standard alarm data fields, which include event_id, description, and last_change.

[0019] Optionally, the uniformly structured data is formatted, specifically including the following steps:

[0020] Convert integer field values ​​to character values, or convert boolean field values ​​to the numbers 0 / 1.

[0021] If the time field in the source alarm data only contains month, day, hour, and minute, then the current year and second digits are added to generate a time value in the format "YYYY-MM-DD HH:MM:SS".

[0022] Check the required fields in the unified structure data. If a required field is missing, the validation is deemed to have failed and an anomaly is marked. The required fields include the alarm object.

[0023] Verify the consistency between the alarm object name and the device IP address. If the suffix of the alarm object name does not match the suffix of the device IP address, the verification is deemed to have failed and an anomaly is marked.

[0024] Optionally, the method further includes the following steps:

[0025] After starting the multi-source alarm data synchronization task, the running status and performance indicators of the synchronization task are recorded in real time; the running status includes running, paused, and failed; the performance indicators include data receiving rate and parsing time.

[0026] Store the execution history of the synchronization task; the execution history includes the actual execution time, total time, and time for each step.

[0027] It receives user query requests, filters standardized and cleaned multi-source alarm data based on data attributes, and displays the filtering results; data attributes include alarm level, data source, and time range.

[0028] Receive log viewing requests from users and display detailed execution logs for the specified synchronization task or specified execution action.

[0029] Optionally, the synchronous task execution plan includes the execution start time, execution end time, and execution frequency; the execution frequency can be configured to be between 1 minute and 60 minutes.

[0030] Optionally, the method further includes the following steps:

[0031] Provides a start / stop control switch for multi-source alarm data synchronization tasks, and receives start / stop commands from users to control the start or stop of multi-source alarm data synchronization tasks;

[0032] By configuring the synchronization task parameters using parameters, the reuse of multi-source alarm data synchronization tasks can be supported, enabling a single multi-source alarm data synchronization task or a single protocol processing class to adapt to the alarm data access of multiple interfacing systems.

[0033] Secondly, this application provides a standardized cleaning system for multi-source alarm data access, including the following functional modules:

[0034] The user configuration acquisition module is used to obtain the synchronization task parameters configured by the user. The synchronization task parameters include the synchronization task execution plan, several protocol processing classes adapted to the data source, parsing rules for source alarm data, and standardized mapping rules.

[0035] The multi-source data receiving module is used to initiate a multi-source alarm data synchronization task and receive multi-source alarm data. The multi-source alarm data includes processing class identifier, source alarm data, additional attributes, and related information.

[0036] The processing class matching module is used to match the processing class identifier of multi-source alarm data with the protocol processing class. If the match fails, the corresponding multi-source alarm data is judged as dirty data and discarded.

[0037] The source data parsing module is used to parse the successfully matched multi-source alarm data according to the corresponding parsing rules, and convert the structured or unstructured source alarm data into indexable objects.

[0038] The mapping rule matching module is used to map indexable objects to uniform structured data based on standardized mapping rules.

[0039] The formatting module is used to format data with a unified structure, completing the standardization and cleaning of multi-source alarm data; the formatting process includes data type conversion, null value filling, integrity verification, and consistency verification.

[0040] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the standardized cleaning method for multi-source alarm data access described above.

[0041] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the standardized cleaning method for multi-source alarm data access described above.

[0042] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the standardized cleaning method for multi-source alarm data access described above.

[0043] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0044] This application provides a standardized cleaning method and related apparatus for multi-source alarm data access. The method includes: obtaining user-configured synchronization task parameters, wherein several protocol processing classes adapted to the data source can achieve multi-protocol compatibility without relying on custom development scripts, and the pre-configuration of standardized mapping rules lays the foundation for subsequent unification of multi-source data formats, avoiding access barriers caused by inconsistent protocol standards from the source and reducing the complexity of subsequent data processing; then receiving multi-source alarm data through a unified task entry point, eliminating the need for maintenance personnel to switch between multiple systems to view data, directly solving the problem of highly dispersed alarm sources and fragmented management; and finally, using a matching and filtering mechanism to quickly filter invalid and incompatible data, avoiding dirty data. This process takes into account the system's storage and computing resources, while preventing invalid data from interfering with subsequent standardized processing. Then, through parsing rules, it achieves unified processing of diverse data formats, solving the problem of high processing difficulty caused by large differences in data formats in traditional technologies. Next, based on standardized mapping rules, alarm data originally scattered across different systems and with varying formats is converted into standardized data with a consistent structure, avoiding the problems of low information acquisition efficiency and easy neglect of critical alarms due to format chaos. Finally, the unified structure data is formatted to complete the standardized cleaning of multi-source alarm data, ultimately shortening fault diagnosis time, improving operational response efficiency, and ensuring the continuity of core businesses such as trading, clearing, and risk control in the fund and securities industries. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart illustrating a standardized cleaning method for multi-source alarm data access, provided as an embodiment of this application.

[0047] Figure 2 This is a flowchart of step A1 in a standardized cleaning method for multi-source alarm data access provided in an embodiment of this application.

[0048] Figure 3 This is a flowchart of step A6 in a standardized cleaning method for multi-source alarm data access provided in an embodiment of this application.

[0049] Figure 4 This is a schematic diagram of the functional modules of a standardized cleaning system for multi-source alarm data access provided in an embodiment of this application.

[0050] Figure 5This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0053] This application provides a standardized cleaning method for multi-source alarm data access. In one exemplary embodiment, such as... Figure 1 As shown, it includes the following steps:

[0054] A1. Obtain the user-configured synchronization task parameters; these parameters include the synchronization task execution plan, several protocol processing classes adapted to the data source, parsing rules for source alarm data, and standardized mapping rules. The synchronization task execution plan includes the execution start time, execution end time, and execution frequency; the execution frequency supports execution at the second level or at a specified time point.

[0055] The protocol processing class is selected based on the capabilities of the interfaced system, supporting both active push (RESTAPI, WebHook, SNMP) and passive polling (RESTAPI, JDBC / ODBC, LOG) modes; parsing rules and mapping rules are configured through a visual interface to ensure low-code operation.

[0056] In this embodiment, as Figure 2 As shown, step A1 specifically includes the following steps:

[0057] A11. Obtain the synchronous task execution plan input by the user and determine the capability type of the interfaced system; the capability type includes supporting external data transmission and supporting API queries. If the interfaced system supports external data transmission, proceed to step A12; if the interfaced system supports API queries, proceed to step A13.

[0058] A12. Select at least one of the following active push-type protocol processing classes: RESTAPI, WebHook, and SNMP.

[0059] A13. Select at least one passive polling type protocol processing class from RESTAPI, JDBC / ODBC, and LOG.

[0060] A14. Set a unique identifier for the synchronization task to associate the protocol processing class with the multi-source alarm data synchronization task.

[0061] A15. The user-input parsing rules are received through a visual field relationship configuration interface. The parsing rules are used to define the field traversal method for structured data or the text extraction logic for unstructured data, so as to convert the source data into indexable objects.

[0062] A16. The user inputs standardized mapping rules through a visual field relationship configuration interface. The standardized mapping rules are used to define the correspondence between the attribute fields of indexable objects and the preset standard alarm data fields. The standard alarm data fields include event_id, description, and last_change.

[0063] A2. Initiate a multi-source alarm data synchronization task to receive multi-source alarm data. Multi-source alarm data includes a processing class identifier, source alarm data, additional attributes, and associated information. The format of the multi-source alarm data is not limited (JSON, file, string, etc.). The processing class identifier is used to match the protocol processing class. Source alarm data includes the alarm source, alarm time, alarm content, and alarm level. Additional attributes include device information, location, responsible person, and device type. Associated information includes system logs, change records, and network topology.

[0064] A3. Match the processing class identifier of the multi-source alarm data with the protocol processing class. If the match fails, the corresponding multi-source alarm data is judged as dirty data and discarded. If no corresponding protocol processing class exists for the processing class identifier, it is judged as dirty data and discarded to avoid invalid data consuming resources.

[0065] A4. For successfully matched multi-source alarm data, parse it according to the corresponding parsing rules, converting the structured or unstructured source alarm data into indexable objects. For example, parse the structured data [143,"CPUs utilization is too high","08011209"] into the indexable object {"alarm_id":143,"description":"CPUs utilization is too high","create_time":"08011209"}.

[0066] A5. Based on standardized mapping rules, indexable objects are mapped to unified structured data. For example, according to the mapping rules, alarm_id in the above indexable object is mapped to the standard field event_id, and create_time is mapped to last_change, resulting in unified structured data {"event_id":143, "description":"CPUs utilization is too high","last_change":"08011209"}.

[0067] A6. Format the data with a unified structure to complete the standardized cleaning of multi-source alarm data; the formatting process includes data type conversion, null value filling, integrity verification, and consistency verification.

[0068] In one exemplary embodiment, such as Figure 3 As shown, step A6 specifically includes the following steps:

[0069] A61. Convert integer field values ​​to character values, or convert boolean field values ​​to the numbers 0 / 1. For example, convert an integer value (such as alarm level "1") to the character value "1", or convert the boolean value "true" to the number "1".

[0070] A62. If the time field in the source alarm data only contains month, day, hour, and minute, then supplement the current year and seconds to generate a time value in the format "YYYY-MM-DD HH:MM:SS". For example, if the source alarm data time field "08011209" is missing the year and seconds, supplement it to "2025-08-01 12:09:00".

[0071] A63. Check the required fields in the unified structure data. If a required field is missing, the validation is deemed to have failed and an anomaly is marked. The required fields include the alarm object.

[0072] A64. Verify the consistency between the alarm object name and the device IP address. If the suffix of the alarm object name does not match the suffix of the device IP address, the verification is considered a failure and marked as abnormal. For example, if the alarm object name is "Unified Alarm Database Master-17.139" and the device IP address is "10.16.17.139", the suffixes should match for it to be considered normal; otherwise, it should be marked as abnormal.

[0073] In another exemplary embodiment of this application, the method further includes task management and data query functions: real-time recording of the running status (running / paused / failed) and performance indicators (data receiving rate, parsing time) of synchronization tasks; storage of execution history (execution time, total time) and detailed logs, supporting users to query historical records; filtering standardized data according to attributes such as alarm level, alarm source, and time range, facilitating maintenance personnel to quickly locate key information. Specifically, the method further includes the following steps:

[0074] B1. After starting the multi-source alarm data synchronization task, record the running status and performance indicators of the synchronization task in real time; the running status includes running, paused, and failed; the performance indicators include data receiving rate and parsing time.

[0075] B2. Store the execution history of the synchronization task; the execution history includes the actual execution time, total time, and time for each step.

[0076] B3. Receive user query requests, filter standardized and cleaned multi-source alarm data according to data attributes, and display the filtering results; data attributes include alarm level, data source, and time range.

[0077] B4. Receive user log viewing requests and display detailed execution logs for the specified synchronization task or specified execution action.

[0078] In another exemplary embodiment of this application, the method further includes the following steps:

[0079] C1 provides a start / stop control switch for the multi-source alarm data synchronization task and receives start / stop commands from users to control the start or stop of the multi-source alarm data synchronization task.

[0080] C2. By configuring synchronization task parameters parametrically, the reuse of multi-source alarm data synchronization tasks is supported, allowing a single multi-source alarm data synchronization task or a single protocol processing class to adapt to alarm data access from multiple interfacing systems. This parameterized configuration of synchronization task parameters enables a single synchronization task or a single protocol processing class to adapt to multiple interfacing systems, similar to multiple servers of the same type, eliminating the need for redundant development and reducing maintenance costs.

[0081] The following example uses the IT operations and maintenance scenario of a fund company to illustrate in detail the method of multi-source alarm data access and standardized cleaning. The specific steps are as follows:

[0082] Corresponding to step A1 above, configuring the synchronization task parameters includes the following three parts:

[0083] Synchronization task execution plan configuration: Users can set the execution start time of the synchronization task to "2025-08-01 00:00:00", the execution end time to "2025-08-31 23:59:59", and the execution frequency to "once every 1 minute" through the visual interface.

[0084] Protocol processing configuration: Connect to the fund company's Zabbix monitoring system (supports external data transmission) and Prometheus monitoring system (supports API queries). For Zabbix, select the "SNMP" proactive push type protocol processing class; for Prometheus, select the "RESTAPI" passive polling type protocol processing class, and set a unique identifier "Fund_Alarm_001" for this synchronization task.

[0085] Configuration of parsing and mapping rules: Configure the parsing rules for Zabbix alarm data (such as extracting the "alarm ID", "alarm content", and "alarm time" fields from the log) and the standardized mapping rules (map "alarm ID" to "event_id", map "alarm time" to "last_change", and supplement the time format as "YYYY-MM-DDHH:MM:SS").

[0086] Corresponding to step A2 above, the synchronization task is started and data is received. For example, if the synchronization task "Fund_Alarm_001" is started, the system will begin to receive SNMP alarm data pushed by Zabbix (such as {"class_id":"zabbix_wehhook","alarm_id":143,"content":"CPUs utilization is too high","time":"08011209","device":"Server_01"}) and Prometheus REST API query data (such as {"class_id":"prometheus_v1","alert_id":256,"info":"Memory utilization exceeds 90%","timestamp":"08011210","ip":"10.16.17.139"}) at 1-minute intervals.

[0087] Next, corresponding to step A3, is the protocol processing class matching. For Zabbix data with "class_id":"zabbix_wehhook", the matching is successful for the "zabbix_wehhook" protocol processing class associated with the synchronization task; for Prometheus data with "class_id":"prometheus_v1", the matching is successful for the associated "prometheus_v1" protocol processing class; if a piece of data with "class_id":"Syslog" is received, there is no corresponding protocol processing class, so it is judged as dirty data and discarded.

[0088] Corresponding to the source data parsing in step A4 above, this application parses the Zabbix data into an indexable object: {"alarm_id":143,"description":"CPUs utilization is too high","create_time":"08011209", "device":"Server_01","class_id":"zabbix_wehhook"}.

[0089] Parse Prometheus data into an indexable object: {"alert_id":256,"description":"Memory usage exceeds 90%","create_time":"08011210","device_ip":"10.16.17.139","class_id": "prometheus_v1"}.

[0090] Next, we perform field mapping corresponding to step A5 above. According to the mapping rules, we map the Zabbix indexable objects to uniform structured data: {"event_id":143,"description":"CPUs utilization is too high","last_change":"08011209","device":"Server_01"}.

[0091] Map Prometheus indexable objects to uniform structured data: {"event_id":256,"description":"Memory usage exceeds 90%","last_change":"08011210", "device_ip":"10.16.17.139"}.

[0092] This corresponds to step A6 in the previous section, where the data with the unified structure is formatted. Since there are no integer / Boolean fields to convert in this example, type conversion is skipped. Specifically, null value filling involves replacing "last_change":"08011209" in the Zabbix data with "2025-08-01 12:09:00", and replacing "last_change":"08011210" in the Prometheus data with "2025-08-01 12:10:00". Integrity verification specifically checks that both data sets contain the required "event_id", "description", and "last_change" fields; this verification passes. Consistency verification specifically checks that the device IP address in the Prometheus data is "10.16.17.139", and if the corresponding alarm object name is "Server_02-17.139", the suffixes match, and the verification passes.

[0093] In addition, this example also records the running status of the multi-source alarm data synchronization task "Fund_Alarm_001" as "running" in real time, with performance indicators of "data reception rate of 2 records / minute, parsing time of 0.5 seconds / record"; storage execution history: records "executed at 12:09:00 on 2025-08-01, taking 1.2 seconds"; operation and maintenance personnel can query the alarm data from "12:00 to 12:10 on 2025-08-01", filter the results with "high-risk alarm level", quickly locate CPU and memory alarms, and carry out fault handling.

[0094] In this example, the fund company achieved unified access and standardization of alarm data from Zabbix and Prometheus. Operations and maintenance personnel no longer need to switch between the two systems and can view and handle alarms through a single platform, improving processing efficiency and avoiding the development and maintenance of custom scripts, thus significantly reducing labor costs.

[0095] Furthermore, the method described in this application is not only applicable to the fund and securities industries, but can also be extended to other financial sectors with high requirements for IT operation and maintenance stability, such as banking and insurance, as well as industries such as power and transportation that require multi-source monitoring and alarm management, thus having a wide range of application scenarios.

[0096] Based on the specific embodiments provided in this application, the following technical effects are disclosed:

[0097] Solve the problem of scattered alarm sources: Through multi-protocol adaptation (JDBC / ODBC, RESTAPI, etc.) and unified task configuration, unified access of alarm data from multiple sources can be achieved, eliminating the need for maintenance personnel to switch between multiple monitoring systems and saving time and costs.

[0098] Reduce technical barriers and maintenance costs: Low-code visual configuration eliminates the need for custom development scripts, solving the problems of poor script reusability and difficult maintenance, and reducing manpower input.

[0099] Improve data processing reliability: By filtering dirty data through processing class matching and formatting verification, ensure data integrity and consistency, and avoid invalid data interfering with operation and maintenance decisions.

[0100] Enhanced operational convenience: Supports task start / stop control, historical record and log query, and data filtering, enabling operations and maintenance personnel to quickly locate critical alarms and improve response efficiency.

[0101] Based on the same inventive concept, this application also provides a system for implementing the standardized cleaning method for multi-source alarm data access described above. The solution provided by this system is similar to the implementation described in the above method. In an exemplary embodiment, such as... Figure 4As shown, a standardized cleaning system for multi-source alarm data access is provided, including the following functional modules:

[0102] The user configuration acquisition module is used to obtain the synchronization task parameters configured by the user. The synchronization task parameters include the synchronization task execution plan, several protocol processing classes adapted to the data source, parsing rules for source alarm data, and standardized mapping rules.

[0103] The multi-source data receiving module is used to initiate a multi-source alarm data synchronization task and receive multi-source alarm data. The multi-source alarm data includes processing class identifier, source alarm data, additional attributes, and related information.

[0104] The processing class matching module is used to match the processing class identifier of multi-source alarm data with the protocol processing class. If the match fails, the corresponding multi-source alarm data is judged as dirty data and discarded.

[0105] The source data parsing module is used to parse the successfully matched multi-source alarm data according to the corresponding parsing rules, and convert the structured or unstructured source alarm data into indexable objects.

[0106] The mapping rule matching module is used to map indexable objects to uniform structured data based on standardized mapping rules.

[0107] The formatting module is used to format data with a unified structure, completing the standardization and cleaning of multi-source alarm data; the formatting process includes data type conversion, null value filling, integrity verification, and consistency verification.

[0108] As an optional implementation, the system also includes a task management module and a data query module: the task management module is used to control the start and stop of synchronous tasks, record the running status and execution history; the data query module is used to filter standardized data according to attributes and display execution logs to improve the convenience of operation and maintenance.

[0109] In addition, as an alternative implementation, the user configuration acquisition module provides a web-based visual interface to receive user input of execution plans, protocol processing classes, and rule configurations, and stores the configuration parameters in a MySQL database to ensure parameter persistence.

[0110] The multi-source data receiving module is based on the Quartz scheduling framework and starts tasks according to the execution plan. It receives Zabbix push data through SNMP TrapListener and obtains Prometheus data by sending HTTP requests through REST Template. It supports concurrent reception with a maximum concurrency of 100 messages per second.

[0111] The processing class matching module queries the database for a list of protocol processing classes associated with the synchronization task, and matches the "class_id" of the received data with the list. The matching logic is implemented using a Hash Map, and the matching efficiency is O(1).

[0112] The source data parsing module uses the Jackson framework to parse JSON format data, uses regular expressions to extract unstructured log fields, and encapsulates the parsing results into Java Beans (indexable objects).

[0113] The mapping rule matching module reads the mapping rules stored in the database (such as "alarm_id→event_id") and uses reflection to assign values ​​to the standard fields of Java Beans, generating data with a unified structure.

[0114] The formatting module implements type conversion (such as Integer.toString()), null value filling (Simple Date Format for formatting time) and validation logic (custom validator) through utility classes, and stores the formatted data in Elasticsearch, supporting efficient querying.

[0115] Task Management Module: Monitors task running status through Spring Boot Admin, provides start / stop button interfaces, records task logs to Logback, and supports log export.

[0116] Data Query Module: Based on Elasticsearch's DSL query syntax, it provides a front-end query interface that supports filtering by multiple dimensions such as time, alarm level, and device. Query results are displayed in the form of tables and charts.

[0117] The system adopts a microservice architecture, with each module deployed independently and supporting horizontal scaling. When the alarm data volume increases to 1,000 records per minute, only two additional application servers are needed to meet the performance requirements, demonstrating good scalability.

[0118] certainly, Figure 4 The architecture shown is merely exemplary; it can be omitted as needed when implementing different functionalities. Figure 4 One or at least two components of the system shown.

[0119] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 5As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it can implement the standardized cleaning method for multi-source alarm data access provided in the previous embodiment.

[0120] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0121] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0122] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0123] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0124] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0125] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0126] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0127] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0128] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A standardized cleaning method for multi-source alarm data access, characterized in that, include: Obtain the synchronization task parameters configured by the user; the synchronization task parameters include the synchronization task execution plan, several protocol processing classes adapted to the data source, parsing rules for source alarm data, and standardized mapping rules; wherein, obtain the synchronization task execution plan input by the user and determine the capability type of the connected system; the capability type includes supporting external data transmission and supporting API query; if the connected system supports external data transmission, select at least one active push type protocol processing class among RESTAPI, WebHook, and SNMP; if the connected system supports API query, select at least one passive polling type protocol processing class among RESTAPI, JDBC / ODBC, and LOG; set a unique identifier for the multi-source alarm data synchronization task and associate the protocol processing class with the multi-source alarm data synchronization task; Initiate a multi-source alarm data synchronization task to receive multi-source alarm data; the multi-source alarm data includes processing class identifier, source alarm data, additional attributes, and related information; The processing class identifier of the multi-source alarm data is matched with the protocol processing class. If the match fails, the corresponding multi-source alarm data is judged as dirty data and discarded. The multi-source alarm data that is successfully matched is parsed according to the corresponding parsing rules, and the structured or unstructured source alarm data is converted into indexable objects. Based on the standardized mapping rules, the indexable objects are mapped to unified structured data; The unified structure data is formatted to complete the standardization and cleaning of the multi-source alarm data; the formatting process includes data type conversion, null value filling, integrity verification, and consistency verification.

2. The standardized cleaning method for multi-source alarm data access according to claim 1, characterized in that, Retrieving user-configured synchronization task parameters also includes: The system receives user-inputted parsing rules through a visual field relationship configuration interface. These parsing rules define the field traversal method for structured data or the text extraction logic for unstructured data, in order to convert the source data into indexable objects. The user-inputted standardized mapping rules are received through the visual field relationship configuration interface. The standardized mapping rules are used to define the correspondence between the attribute fields of indexable objects and the preset standard alarm data fields. The standard alarm data fields include event_id, description, and last_change. The attribute fields include alarm ID, alarm content, and alarm time. event_id corresponds to alarm ID, description corresponds to alarm content, and last_change corresponds to alarm time.

3. The standardized cleaning method for multi-source alarm data access according to claim 1, characterized in that, The formatting process for the unified structured data specifically includes: Convert integer field values ​​to character values, or convert boolean field values ​​to the numbers 0 / 1; If the time field in the source alarm data only contains month, day, hour, and minute, then add the current year and second to generate a time value in the format "YYYY-MM-DD HH:MM:SS". The required fields in the unified structure data are checked. If any required field is missing, the validation is deemed to have failed and an anomaly is marked. The required fields include alarm objects. Verify the consistency between the alarm object name and the device IP address. If the suffix of the alarm object name does not match the suffix of the device IP address, the verification is deemed to have failed and an anomaly is marked.

4. The standardized cleaning method for multi-source alarm data access according to claim 1, characterized in that, Also includes: After the multi-source alarm data synchronization task is started, the running status and performance indicators of the synchronization task are recorded in real time; the running status includes running, paused, and failed; the performance indicators include data receiving rate and parsing time. Store the execution history of the synchronization task; the execution history includes the actual execution time, total time, and time for each step; The system receives user query requests, filters standardized and cleaned multi-source alarm data based on data attributes, and displays the filtering results; the data attributes include alarm level, data source, and time range. Receive log viewing requests from users and display detailed execution logs for the specified synchronization task or specified execution action.

5. The standardized cleaning method for multi-source alarm data access according to claim 1, characterized in that, The synchronous task execution plan includes the execution start time, execution end time, and execution frequency; the execution frequency is configured to be an interval of 1 minute to 60 minutes.

6. The standardized cleaning method for multi-source alarm data access according to claim 1, characterized in that, The method further includes: Provides a start / stop control switch for the multi-source alarm data synchronization task, and receives start / stop commands from users to control the start or stop of the multi-source alarm data synchronization task; By configuring the parameters of the synchronization task in a parameterized manner, the reuse of multi-source alarm data synchronization tasks can be supported, enabling one multi-source alarm data synchronization task or one protocol processing class to adapt to the alarm data access of multiple docking systems.

7. A standardized cleaning system for multi-source alarm data access, characterized in that, The system is used to implement the standardized cleaning method for multi-source alarm data access as described in any one of claims 1-6, the system comprising: The user configuration acquisition module is used to acquire user-configured synchronization task parameters. These parameters include the synchronization task execution plan, several protocol processing classes adapted to the data source, parsing rules for source alarm data, and standardized mapping rules. Specifically, it acquires the user-input synchronization task execution plan and determines the capability type of the connected system. The capability type includes support for external data transmission and support for API queries. If the connected system supports external data transmission, it selects at least one proactive push protocol processing class from RESTAPI, WebHook, and SNMP. If the connected system supports API queries, it selects at least one passive polling protocol processing class from RESTAPI, JDBC / ODBC, and LOG. A unique identifier is set for the multi-source alarm data synchronization task, associating the protocol processing class with the multi-source alarm data synchronization task. The multi-source data receiving module is used to initiate a multi-source alarm data synchronization task and receive multi-source alarm data; the multi-source alarm data includes processing class identifier, source alarm data, additional attributes, and related information; The processing class matching module is used to match the processing class identifier of the multi-source alarm data with the protocol processing class. If the matching fails, the corresponding multi-source alarm data is judged as dirty data and discarded. The source data parsing module is used to parse the successfully matched multi-source alarm data according to the corresponding parsing rules, and convert the structured or unstructured source alarm data into indexable objects. The mapping rule matching module is used to map the indexable object into unified structure data based on the standardized mapping rules. The formatting module is used to format the unified structure data to complete the standardization and cleaning of the multi-source alarm data; the formatting process includes data type conversion, null value filling, integrity verification, and consistency verification.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the standardized cleaning method for multi-source alarm data access as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the standardized cleaning method for multi-source alarm data access as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the standardized cleaning method for multi-source alarm data access as described in any one of claims 1-6.

Images