Equipment data processing method and device

Abstract

The invention discloses an equipment data processing method and device. The method comprises the steps that to-be-processed equipment data are obtained, an ETL process and a task relation graph corresponding to the to-be-processed equipment data are determined, the ETL process comprises a plurality of ETL tasks, and the task relation graph comprises dependency relations among the tasks; when the ETL process is executed, a state signal generated by each ETL task is obtained, and the state signal is used for reflecting the execution state of the task; performing rationality verification on each state signal, and if any state signal is found to be abnormal, determining an abnormal ETL task corresponding to the abnormal state signal; performing data traceability analysis on the abnormal ETL task according to the task relation graph, and determining a specific anomaly source causing anomaly; and correcting the abnormal source, and re-executing the ETL process from the first ETL task corresponding to the abnormal source in the task relation graph. According to the method and the device, the technical problem that the ETL task generated for mass device data in a device management scene is difficult to efficiently schedule and process exceptions is solved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a device data processing method and apparatus. Background Art

[0002] Amid the rapid development of big data and IoT technologies, enterprises face the challenge of processing and analyzing massive amounts of device data. With the widespread adoption of big data platforms like Hadoop, the collection, transmission, and processing of device data have become critical. However, traditional device data processing relies primarily on fixed task scheduling and manual operations. This approach struggles with dynamically changing device data and complex ETL (Extract, Transform, Load) processes. Data processing efficiency and accuracy are impacted by the quality of device data transmission, the diversity of data structures, and the complexity of data processing logic. In particular, during the flow of device data, missing source data and inconsistent data standards often lead to discrepancies in data analysis results, impacting the accuracy and timeliness of enterprise operational decisions. Furthermore, the heterogeneity and real-time nature of device data necessitate a high degree of flexibility and intelligence within the data processing process to adapt to diverse data needs and processing scenarios.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] The embodiments of the present application provide a device data processing method and apparatus to at least solve the technical problem of difficulty in efficiently scheduling and handling exceptions for ETL tasks generated for massive device data in device management scenarios.

[0005] According to one aspect of an embodiment of the present application, a device data processing method is provided, including: obtaining device data to be processed, and determining an ETL process corresponding to the device data to be processed and a task relationship map corresponding to the ETL process, wherein the ETL process includes multiple ETL tasks, and the task relationship map includes dependencies between multiple ETL tasks; executing the ETL process, and during the execution process, obtaining a status signal generated by each ETL task, wherein the status signal is used to reflect the execution status of the ETL task; performing a rationality check on each status signal, and when any status signal has an abnormality, determining the abnormal ETL task corresponding to the abnormal status signal, performing data tracing analysis on the abnormal ETL task based on the task relationship map, and determining the abnormal source that caused the abnormality; correcting the abnormal source, and re-executing the ETL process starting from the first ETL task corresponding to the abnormal source in the task relationship map.

[0006] Optionally, obtaining the device data to be processed includes: periodically obtaining multiple sets of device data transmitted by each device in the target system through a preset transmission protocol, and storing the multiple sets of device data as the device data to be processed in a shared database, wherein the shared database is also used to store the data after processing.

[0007] Optionally, the method also includes: for each device in the target system, if the device data transmitted by the device is not received in the current cycle, generating a device abnormality alarm event; detecting the data transmission status of the device in a preset number of consecutive cycles before the current cycle; if the device data transmitted by the device has been received, terminating the device abnormality alarm event, and obtaining the device data transmitted by the device from the shared database to constitute the device data to be processed in the current cycle; if the device data transmitted by the device has not been received, issuing a device abnormality processing work order for the device abnormality alarm event.

[0008] Optionally, determining the ETL process corresponding to the device data to be processed and the task relationship graph corresponding to the ETL process includes: determining the data attributes and data processing requirements of the device data to be processed, wherein the data attributes include at least one of the following: data type, data structure; determining multiple data processing steps that need to be performed to implement the data processing requirements based on the data attributes, generating an ETL task corresponding to each data processing step, and determining the dependency relationship between multiple ETL tasks to obtain the ETL process; constructing a task relationship graph with each ETL task as a node and the dependency relationship between each ETL task as an edge.

[0009] Optionally, the status signal includes at least: a task start signal and a task end signal, and executing the ETL process includes: determining the initial ETL task located at the upstream end in the task relationship graph, and executing the ETL task; for each ETL task except the initial ETL task in the task relationship graph, when the task end signals of all upstream ETL tasks adjacent to the ETL task are obtained and each task end signal passes the rationality check, the execution of the ETL task is triggered.

[0010] Optionally, a rationality check is performed on each status signal, including: for each ETL task, a rationality check is performed on each status signal generated by the ETL task, wherein the rationality check includes: a uniqueness check and a logic closed loop check, the uniqueness check is used to check whether each status signal of the ETL task is unique, and the logic closed loop check is used to check whether all status signals of the ETL task include a closed loop from the task start signal to the task end signal.

[0011] Optionally, data traceability analysis is performed on the abnormal ETL task based on the task relationship graph to determine the source of the exception that caused the exception, including: starting from the abnormal ETL task, reversely traversing all upstream ETL tasks of the abnormal ETL task in the task relationship graph to determine the source of the exception that caused the exception, wherein the exception source includes at least one of the following: abnormal device data in the device data to be processed, and the error handling process in the ETL task.

[0012] Optionally, the abnormal source is corrected, including: when the abnormal source is abnormal device data in the device data to be processed, re-acquiring device data from the device corresponding to the abnormal device data and replacing the abnormal device data, or, correcting the abnormal device data, the correction method includes at least one of the following: abnormal value deletion, blank value interpolation; when the abnormal source is the error handling process in the ETL task, resetting the ETL process of the ETL task with the error handling process.

[0013] Optionally, the method further includes: generating a device data processing report after the ETL process is completed, wherein the device data processing report includes at least one of the following: the source of the device data to be processed, the device data to be processed, the data processing results of each ETL task, and an exception processing log.

[0014] According to another aspect of an embodiment of the present application, a device data processing apparatus is provided, comprising: an acquisition module, configured to acquire device data to be processed, and determine an ETL process corresponding to the device data to be processed and a task relationship graph corresponding to the ETL process, wherein the ETL process includes multiple ETL tasks, and the task relationship graph includes dependency relationships between the multiple ETL tasks;

[0015] The execution module is used to execute the ETL process and obtain the status signal generated by each ETL task during the execution process, where the status signal is used to reflect the execution status of the ETL task; the verification module is used to perform rationality verification on each status signal, and when there is an exception in any status signal, determine the abnormal ETL task corresponding to the abnormal status signal, perform data tracing analysis on the abnormal ETL task based on the task relationship graph, and determine the abnormal source that caused the exception; the correction module is used to correct the abnormal source and re-execute the ETL process starting from the first ETL task corresponding to the abnormal source in the task relationship graph.

[0016] According to another aspect of an embodiment of the present application, a computer program product is further provided, the computer program product comprising: a computer program, wherein when the computer program is executed by a processor, the above-mentioned device data processing method is implemented.

[0017] According to another aspect of an embodiment of the present application, an electronic device is provided, comprising: a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above-mentioned device data processing method through the computer program.

[0018] In an embodiment of the present application, by obtaining the device data to be processed and determining the corresponding ETL process, the pertinence and accuracy of data processing are ensured. At the same time, a task relationship map is constructed to clarify the dependencies between multiple ETL tasks. The execution order of tasks can be intelligently determined to avoid logical errors and redundant operations in data processing, thereby improving the degree of automation of the data processing process and accelerating the data processing speed. During the execution of the ETL process, the system obtains the status signal generated by each ETL task in real time and performs rationality verification, including signal uniqueness and logical closed-loop inspection. This verification mechanism can promptly detect abnormal situations in task execution, thereby quickly responding, avoiding further propagation of data processing errors, and ensuring the integrity and accuracy of data processing. When any status signal is abnormal, the system can accurately determine the abnormal ETL task, and perform data traceability analysis based on the task relationship map to locate the source of the abnormality. This mechanism can not only effectively solve specific problems in data processing, but also deeply analyze the root cause of the problem, providing a scientific basis for the prevention and treatment of abnormalities. After determining the source of the exception, the system will correct the source of the exception and re-execute the ETL process starting from the first ETL task corresponding to the exception source in the task relationship map to ensure the correctness and consistency of data processing. This ensures that when the system encounters an exception, it can quickly restore the data processing process, avoid data processing interruptions caused by exceptions, improve the continuity and stability of data processing, and thus solve the technical problem of difficult efficient scheduling and exception handling of ETL tasks generated for massive equipment data in equipment management scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 is a flowchart of an optional device data processing method according to an embodiment of the present application;

[0021] Figure 2 is a structural diagram of an optional device data processing apparatus according to an embodiment of the present application;

[0022] Figure 3 It is a schematic structural diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0024] It should be noted that the terms "first", "second", etc. in the specification, claims, and drawings of the present application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0025] In order to better understand the embodiments of the present application, some nouns or terms that appear in the description of the embodiments of the present application are first translated and explained as follows:

[0026] The ETL process is a core process in data integration and data warehouse development. It is used to extract data from multiple heterogeneous data sources, cleanse, transform, and consolidate it, and ultimately load the processed data into the target database or data warehouse. The ETL process ensures data consistency, accuracy, and availability and is widely used in scenarios such as enterprise-level data analysis, report generation, and machine learning model training.

[0027] The information collected in the embodiments of the present application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0028] Example 1

[0029] According to an embodiment of the present application, a device data processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0030] Figure 1 is a flow chart of a device data processing method provided according to an embodiment of the present application, such as Figure 1 As shown, the method includes the following steps:

[0031] Step S102: Obtain device data to be processed, and determine an ETL process corresponding to the device data to be processed and a task relationship graph corresponding to the ETL process, wherein the ETL process includes multiple ETL tasks, and the task relationship graph includes dependency relationships between the multiple ETL tasks;

[0032] Step S104: executing the ETL process and obtaining a status signal generated by each ETL task during the execution process, wherein the status signal is used to reflect the execution status of the ETL task;

[0033] Step S106: Perform a rationality check on each status signal. If any status signal is abnormal, determine the abnormal ETL task corresponding to the abnormal status signal, perform data tracing analysis on the abnormal ETL task based on the task relationship graph, and determine the abnormal source that caused the abnormality.

[0034] Step S108: Correct the abnormal source and re-execute the ETL process starting from the first ETL task corresponding to the abnormal source in the task relationship graph.

[0035] The following describes each step of the device data processing method in conjunction with a specific implementation process.

[0036] First, obtain the device data to be processed, and determine the ETL process corresponding to the device data to be processed and the task relationship map corresponding to the ETL process, wherein the ETL process includes multiple ETL tasks, and the task relationship map includes the dependency relationships between multiple ETL tasks.

[0037] As an optional implementation method, the following steps can be taken to obtain the device data to be processed: periodically obtain multiple sets of device data transmitted by each device in the target system through a preset transmission protocol. The periodicity here refers to automatically capturing and transmitting data at set time intervals, such as every hour, every day, etc. The multiple sets of device data can be status information, log records, performance indicators or other important data generated by various types of devices during operation. These multiple sets of device data are stored as device data to be processed in a shared database. The shared database is not only used to store the device data to be processed, but also responsible for storing the data after processing for subsequent data analysis and use.

[0038] As an optional implementation, after obtaining the device data to be processed, the process of determining the ETL process corresponding to the device data to be processed and the task relationship graph corresponding to the ETL process can take the following steps:

[0039] Determine the data attributes and data processing requirements (e.g., cleaning, conversion, loading, aggregation, etc.) of the device data to be processed, where the data attributes include at least one of the following: data type (e.g., structured, semi-structured, unstructured data), data structure (e.g., table, JSON (JavaScript Object Notation), XML (Extensible Markup Language) format, etc.);

[0040] Then, based on these data attributes, determine the multiple data processing steps that need to be performed to achieve data processing requirements, generate the ETL tasks corresponding to each data processing step, and determine the dependencies between multiple ETL tasks to obtain the ETL process;

[0041] For example, if structured data needs to be cleaned, a SQL (Structured Query Language) query task might be generated, while if semi-structured data needs to be transformed, a Python script task might be generated. Regarding the generation of dependencies between multiple ETL tasks, it can be understood that Task A may need to start only after Task B is completed, or multiple tasks can be executed in parallel, but the final results need to be aggregated into a single task. These dependencies need to be clearly defined. For example, a aggregation task may depend on the completion of multiple data cleaning tasks, and a transformation task may depend on the results of a data extraction task.

[0042] Subsequently, a task relationship graph is constructed with each ETL task as a node and the dependencies between each ETL task as edges. This graph can clearly show the flow and sequence of data processing, making it easier to understand and control the entire ETL process. It also supports backtracking and rerunning from any node.

[0043] Among them, for each device in the target system, if the device data transmitted by the device is not received in the current cycle, a device abnormality alarm event is generated; the data transmission status of the device in a preset number of consecutive cycles before the current cycle is detected. If the device data transmitted by the device has been received, the device abnormality alarm event is terminated, and the device data transmitted by the device is obtained from the shared database to form the device data to be processed in the current cycle; if the device data transmitted by the device has not been received, a device abnormality processing work order is issued for the device abnormality alarm event.

[0044] For example, the system can schedule periodic tasks, such as hourly, daily, or at specific intervals, to check whether each device in the target system has transmitted data within the current cycle. For each device in the target system, the system checks whether any device data has been received from that device within the current cycle. If no data has been received, the system generates a device anomaly alarm event, indicating a possible data transmission anomaly or failure on the device, requiring further investigation. The event may include information such as the device ID, the period in which no data was received, and the possible cause of the anomaly, allowing operations personnel to quickly locate the problem. The system can then further examine the device's data transmission status for a preset number of consecutive cycles prior to the current cycle. For example, a check can be set to examine data reception within the previous three cycles. If data from the device is detected to have been received within a previous cycle, the system terminates the current device anomaly alarm event. The system then retrieves the device data transmitted by the device in the previous cycle from a shared database and merges this data with the other received data from the current cycle to form the device data to be processed for the current cycle, ensuring the continuity and integrity of data processing. If no device data is received from the device within the historical inspection period, the system believes that the device may have ongoing data transmission problems. In response to the device abnormality alarm event, a device abnormality handling work order will be issued. The device abnormality handling work order will record the device ID, abnormality period, abnormality type and possible abnormality cause in detail, and automatically send it to the operation and maintenance team or equipment supplier to initiate the abnormality handling process, including but not limited to device inspection, data transmission protocol verification, data retransmission request, etc., to resolve the device data transmission abnormality problem.

[0045] After constructing the task relationship graph, the ETL process is executed, and during the execution, the status signal generated by each ETL task is obtained, where the status signal is used to reflect the execution status of the ETL task, and the status signal includes at least a task start signal and a task end signal.

[0046] As an optional implementation, executing the ETL process may include the following process: determining the initial ETL task located at the upstream end in the task relationship graph and executing the ETL task. The initial ETL task usually does not depend on the results of other tasks and is the starting point of the entire ETL process; for each ETL task except the initial ETL task in the task relationship graph, when the task end signals of all upstream ETL tasks adjacent to the ETL task are obtained and the task end signals of each task pass the rationality check, the execution of the ETL task is triggered.

[0047] The specific process can be referred to as follows: Each ETL task generates a start signal, namely the task start signal, when it starts, and generates an end signal, namely the task end signal, when the task is completed successfully or failed. These semaphores carry key parameters such as task ID, execution timestamp, status information (such as success, failure, running), etc. to ensure the uniqueness and traceability of the signal. The signal is transmitted between tasks through a signal processor (signal container). The signal processor is responsible for receiving task end signals from completed tasks and storing these signals. At the same time, it is also responsible for detecting the start conditions of downstream tasks, that is, when the end signals of all upstream tasks are received and verified, it will trigger the execution of downstream tasks. The system determines the upstream initial ETL task (that is, the starting point of the data processing process) through the task relationship graph and starts the task immediately. The start of the initial task usually does not depend on any upstream task, so it can be executed independently. For non-initial tasks in the task relationship graph, the system checks whether all upstream tasks have been successfully completed and whether the received task end signals have passed the rationality check. Only when the execution status of the upstream tasks meets the preset dependency conditions will the downstream tasks be triggered for execution. This semaphore-based dependency judgment mechanism ensures the consistency of the data processing flow and the correctness of task execution. Based on the dependencies between tasks in the task relationship graph, the system can intelligently select concurrent execution or serial execution strategies. For tasks without dependencies, the system can start them in parallel to improve overall processing efficiency. For tasks with dependencies, the system will follow the serial execution strategy to ensure the logical order of data processing.

[0048] Throughout the ETL process, the system monitors the execution status of each task in real time, including but not limited to task start, task completion, and task exceptions, ensuring transparency and controllability throughout the entire process. Furthermore, the system provides feedback on the execution results of ETL tasks, including the online status of equipment, exceptions, and the efficiency and accuracy of data processing, providing operations personnel with real-time feedback on equipment automation operations and data warehouse status.

[0049] After obtaining the status signal generated by each ETL task, perform a rationality check on each status signal. When any status signal is abnormal, determine the abnormal ETL task corresponding to the abnormal status signal. Perform data traceability analysis on the abnormal ETL task based on the task relationship graph to determine the source of the abnormality that caused the abnormality.

[0050] As an optional implementation, performing a rationality check on each status signal may include the following steps: for each ETL task, performing a rationality check on each status signal generated by the ETL task, wherein the rationality check includes: uniqueness check and logic closed-loop check. The uniqueness check is used to check whether each status signal of the ETL task is unique, ensuring that each status signal has a unique identifier within the task scope to avoid signal confusion and processing errors. The logic closed-loop check is used to check whether all status signals of the ETL task include a closed loop from the task start signal to the task end signal. It can be understood that each task must have a corresponding task start signal and task end signal, and the timestamps and task IDs of these two signals should match to ensure that the task is started and completed as expected. The rationality check mechanism should also include checking the signal timestamp to ensure that the generation and transmission time of the status signal are logically consistent to avoid process confusion caused by timestamp errors.

[0051] As an optional implementation, after determining the abnormal ETL task corresponding to the abnormal status signal, data traceability analysis is performed on the abnormal ETL task based on the task relationship graph to determine the source of the exception that caused the exception. The process can perform the following steps: starting from the abnormal ETL task, reversely traverse all upstream ETL tasks of the abnormal ETL task in the task relationship graph to determine the source of the exception that caused the exception, wherein the abnormal source includes at least one of the following: abnormal device data in the device data to be processed, error handling process in the ETL task, for example, data conversion rule setting error, data loading target table error, etc.

[0052] After determining the source of the exception, correct the source of the exception and re-execute the ETL process starting from the first ETL task corresponding to the exception source in the task relationship graph.

[0053] Among them, the process of correcting the abnormal source can be the following steps: when the abnormal source is abnormal device data in the device data to be processed, the device data is re-acquired from the device corresponding to the abnormal device data and the abnormal device data is replaced, or the abnormal device data is corrected, and the correction method includes at least one of the following: abnormal value deletion, blank value interpolation; when the abnormal source is the error handling process in the ETL task, the ETL process of the ETL task with the error handling process is reset. For example, this may involve modifying the SQL script, adjusting the task execution order, optimizing the data cleaning strategy, etc., to resolve errors in the process and improve the efficiency and accuracy of data processing.

[0054] After completing the correction of the abnormal source, the system will record the detailed information of the correction process, including the specific location of the abnormal source, the correction method, the comparison of data before and after correction, etc. This information will be included in the abnormal processing log to provide a reference for subsequent operation and maintenance analysis.

[0055] In order to completely resolve the exception problem, the system will re-execute the entire ETL process starting from the first ETL task corresponding to the exception source. This step ensures that the corrected data or process can be fully processed, avoiding problems left over from local corrections. The re-executed process will follow the dependencies and execution order defined in the task relationship graph to ensure coordination and consistency between tasks.

[0056] After the ETL process is complete, a device data processing report is generated. This report includes at least one of the following: the source of the device data to be processed, the device data to be processed, the data processing results of each ETL task, and an exception handling log. Device data processing reports can be archived by the system for easy historical query and comparison, which is crucial for long-term equipment maintenance and data processing process optimization. The system also provides a report sharing function, allowing operations and management to share reports with other relevant personnel via email, in-system messages, or report sharing links, facilitating cross-departmental communication and collaboration.

[0057] In an embodiment of the present application, by obtaining the device data to be processed and determining the corresponding ETL process, the pertinence and accuracy of data processing are ensured. At the same time, a task relationship map is constructed to clarify the dependencies between multiple ETL tasks. The execution order of tasks can be intelligently determined to avoid logical errors and redundant operations in data processing, thereby improving the degree of automation of the data processing process and accelerating the data processing speed. During the execution of the ETL process, the system obtains the status signal generated by each ETL task in real time and performs rationality verification, including signal uniqueness and logical closed-loop inspection. This verification mechanism can promptly detect abnormal situations in task execution, thereby quickly responding, avoiding further propagation of data processing errors, and ensuring the integrity and accuracy of data processing. When any status signal is abnormal, the system can accurately determine the abnormal ETL task, and perform data traceability analysis based on the task relationship map to locate the source of the abnormality that caused the abnormality. This mechanism can not only effectively solve specific problems in data processing, but also deeply analyze the root causes of the problems, providing a scientific basis for the prevention and treatment of abnormalities. After determining the source of the exception, the system will correct the source of the exception and re-execute the ETL process starting from the first ETL task corresponding to the exception source in the task relationship map to ensure the correctness and consistency of data processing. This ensures that when the system encounters an exception, it can quickly restore the data processing process, avoid data processing interruptions caused by exceptions, improve the continuity and stability of data processing, and thus solve the technical problem of difficult efficient scheduling and exception handling of ETL tasks generated for massive equipment data in equipment management scenarios.

[0058] Example 2

[0059] According to an embodiment of the present application, a device data processing apparatus for implementing the device data processing method in embodiment 1 is also provided. Figure 2 As shown, the device data processing apparatus at least includes: an acquisition module 21, an execution module 22, a verification module 23 and a correction module 24, wherein:

[0060] An acquisition module 21 is configured to acquire device data to be processed and determine an ETL process corresponding to the device data to be processed and a task relationship graph corresponding to the ETL process, wherein the ETL process includes multiple ETL tasks, and the task relationship graph includes dependencies between the multiple ETL tasks;

[0061] An execution module 22 is used to execute the ETL process and obtain a status signal generated by each ETL task during the execution process, wherein the status signal is used to reflect the execution status of the ETL task;

[0062] Verification module 23 is used to perform rationality verification on each status signal, and when any status signal is abnormal, determine the abnormal ETL task corresponding to the abnormal status signal, perform data tracing analysis on the abnormal ETL task based on the task relationship graph, and determine the abnormal source that caused the abnormality;

[0063] The correction module 24 is used to correct the abnormal source and re-execute the ETL process starting from the first ETL task corresponding to the abnormal source in the task relationship graph.

[0064] The functions of each module of the equipment data processing device are described below in conjunction with a specific implementation process.

[0065] The acquisition module acquires the device data to be processed and determines the ETL process corresponding to the device data to be processed and the task relationship map corresponding to the ETL process, wherein the ETL process includes multiple ETL tasks, and the task relationship map includes the dependency relationships between the multiple ETL tasks.

[0066] As an optional implementation method, to obtain the device data to be processed, the following steps can be taken: periodically obtain multiple sets of device data transmitted by each device in the target system through a preset transmission protocol, and store the multiple sets of device data as device data to be processed in a shared database, wherein the shared database is also used to store the data after processing.

[0067] As an optional implementation method, after obtaining the device data to be processed, the process of determining the ETL process corresponding to the device data to be processed and the task relationship graph corresponding to the ETL process can take the following steps: determine the data attributes and data processing requirements of the device data to be processed, wherein the data attributes include at least one of the following: data type, data structure; determine multiple data processing steps that need to be performed to implement the data processing requirements based on the data attributes, generate the ETL task corresponding to each data processing step, and determine the dependency relationship between multiple ETL tasks to obtain the ETL process; construct a task relationship graph with each ETL task as a node and the dependency relationship between each ETL task as an edge.

[0068] Among them, for each device in the target system, if the device data transmitted by the device is not received in the current cycle, a device abnormality alarm event is generated; the data transmission status of the device in a preset number of consecutive cycles before the current cycle is detected; if the device data transmitted by the device has been received, the device abnormality alarm event is terminated, and the device data transmitted by the device is obtained from the shared database to form the device data to be processed in the current cycle; if the device data transmitted by the device has not been received, a device abnormality processing work order is issued for the device abnormality alarm event.

[0069] After constructing the task relationship graph, the execution module executes the ETL process and obtains the status signal generated by each ETL task during the execution process, where the status signal is used to reflect the execution status of the ETL task.

[0070] As an optional implementation, the status signal includes at least a task start signal and a task end signal, and the execution of the ETL process may include the following process: determining the initial ETL task located at the upstream end in the task relationship graph, and executing the ETL task; for each ETL task except the initial ETL task in the task relationship graph, when the task end signals of all upstream ETL tasks adjacent to the ETL task are obtained and each task end signal passes the rationality check, the execution of the ETL task is triggered.

[0071] After obtaining the status signal generated by each ETL task, the verification module performs a rationality check on each status signal. When any status signal is abnormal, it determines the abnormal ETL task corresponding to the abnormal status signal, performs data traceability analysis on the abnormal ETL task based on the task relationship graph, and determines the source of the abnormality that caused the abnormality.

[0072] As an optional implementation, performing a rationality check on each status signal may include the following steps: for each ETL task, performing a rationality check on each status signal generated by the ETL task, wherein the rationality check includes: uniqueness check and logic closed-loop check, the uniqueness check is used to check whether each status signal of the ETL task is unique, and the logic closed-loop check is used to check whether all status signals of the ETL task include a closed loop from the task start signal to the task end signal.

[0073] As an optional implementation method, data traceability analysis is performed on the abnormal ETL task based on the task relationship graph to determine the source of the exception that caused the exception. The process can execute the following steps: starting from the abnormal ETL task, reversely traverse all upstream ETL tasks of the abnormal ETL task in the task relationship graph to determine the source of the exception that caused the exception, wherein the exception source includes at least one of the following: abnormal device data in the device data to be processed, and error handling process in the ETL task.

[0074] After determining the source of the exception, the correction module corrects the source of the exception and re-executes the ETL process starting from the first ETL task corresponding to the exception source in the task relationship graph.

[0075] Among them, the process of correcting the abnormal source can be the following steps: when the abnormal source is abnormal device data in the device data to be processed, the device data is re-acquired from the device corresponding to the abnormal device data and the abnormal device data is replaced, or the abnormal device data is corrected, and the correction method includes at least one of the following: abnormal value deletion, blank value interpolation; when the abnormal source is the error handling process in the ETL task, the ETL process of the ETL task with the error handling process is reset.

[0076] After the ETL process is completed, a device data processing report is generated, wherein the device data processing report includes at least one of the following: the source of the device data to be processed, the device data to be processed, the data processing results of each ETL task, and the exception processing log.

[0077] It should be noted that each module in the device data processing apparatus in the embodiment of the present application corresponds one-to-one to each implementation step of the device data processing method in Example 1. Since a detailed description has been given in Example 1, some details not reflected in this embodiment can be referred to Example 1 and will not be elaborated on here.

[0078] Example 3

[0079] According to an embodiment of the present application, a computer program product is further provided. The computer program product includes a computer program, wherein when the computer program is executed by a processor, the device data processing method in Example 1 is implemented.

[0080] According to an embodiment of the present application, a non-volatile storage medium is further provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the device data processing method in Example 1 by running the computer program.

[0081] According to an embodiment of the present application, a processor is further provided, which is used to run a computer program, wherein the device data processing method in Example 1 is executed when the computer program is run.

[0082] According to an embodiment of the present application, an electronic device is further provided, which includes: a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the device data processing method in Example 1 through the computer program.

[0083] Specifically, the computer program executes the following steps when it is running: obtaining the device data to be processed, and determining the ETL process corresponding to the device data to be processed and the task relationship map corresponding to the ETL process, wherein the ETL process includes multiple ETL tasks, and the task relationship map includes the dependency relationship between multiple ETL tasks; executing the ETL process, and obtaining the status signal generated by each ETL task during the execution process, wherein the status signal is used to reflect the execution status of the ETL task; performing a rationality check on each status signal, and when any status signal has an abnormality, determining the abnormal ETL task corresponding to the abnormal status signal, performing data tracing analysis on the abnormal ETL task based on the task relationship map, and determining the abnormal source that caused the abnormality; correcting the abnormal source, and re-executing the ETL process starting from the first ETL task corresponding to the abnormal source in the task relationship map.

[0084] As an optional implementation, the electronic device may be in the form of a mobile terminal, a computer terminal or a similar computing device. Figure 3 FIG1 shows a hardware structure block diagram of an electronic device for implementing a device data processing method. Figure 3 As shown, the electronic device 30 may include one or more (illustrated as 302a, 302b, ..., 302n in the figure) processors 302 (the processor 302 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 304 for storing data, and a transmission device 306 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 3 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 3 More or fewer components than shown, or with Figure 3 Different configurations shown.

[0085] It should be noted that the one or more processors 302 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the electronic device 30. As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0086] The memory 304 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the device data processing method in the embodiment of the present application. The processor 302 executes various functional applications and data processing by running the software programs and modules stored in the memory 304, that is, implementing the vulnerability detection method of the above-mentioned application. The memory 304 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 304 may further include a memory remotely located relative to the processor 302, and these remote memories may be connected to the electronic device 30 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0087] The transmission device 306 is used to receive or send data via a network. Specific examples of the aforementioned network may include a wireless network provided by the communications provider of the electronic device 30. In one embodiment, the transmission device 306 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In one embodiment, the transmission device 306 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0088] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the electronic device 30 .

[0089] The serial numbers of the above embodiments are for description only and do not represent the advantages or disadvantages of the embodiments.

[0090] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0091] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0092] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected to achieve the purpose of the present embodiment according to actual needs.

[0093] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0094] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program code.

[0095] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A device data processing method, characterized in that: include: Acquire device data to be processed, and determine an extract-transform-load (ETL) process corresponding to the device data to be processed and a task relationship map corresponding to the ETL process, wherein the ETL process includes multiple ETL tasks, and the task relationship map includes dependencies between the multiple ETL tasks; Execute the ETL process, and during the execution process, obtain a status signal generated by each ETL task, wherein the status signal is used to reflect the execution status of the ETL task; Perform a rationality check on each of the state signals, and when any state signal is abnormal, determine the abnormal ETL task corresponding to the abnormal state signal, perform data tracing analysis on the abnormal ETL task according to the task relationship graph, and determine the abnormal source causing the abnormality; The exception source is corrected, and the ETL process is re-executed starting from the first ETL task corresponding to the exception source in the task relationship graph.

2. The method according to claim 1, characterized in that Get the device data to be processed, including: Periodically obtain multiple sets of device data transmitted by each device in the target system through a preset transmission protocol, and store the multiple sets of device data as the device data to be processed in a shared database, wherein the shared database is also used to store the data after processing.

3. The method according to claim 2, characterized in that The method further comprises: For each device in the target system, if device data transmitted by the device is not received in a current cycle, a device abnormality alarm event is generated; Detecting the data transmission status of the device within a preset number of consecutive cycles before the current cycle; If the device data transmitted by the device has been received, the device abnormality alarm event is terminated, and the device data transmitted by the device is obtained from the shared database to form the device data to be processed in the current cycle; If the device data transmitted by the device has not been received, a device abnormality processing work order is issued for the device abnormality alarm event.

4. The method according to claim 1, characterized in that: Determining an ETL process corresponding to the device data to be processed and a task relationship graph corresponding to the ETL process includes: Determining data attributes and data processing requirements of the device data to be processed, wherein the data attributes include at least one of the following: data type, data structure; Determine multiple data processing steps that need to be performed to achieve the data processing requirements based on the data attributes, generate an ETL task corresponding to each of the data processing steps, and determine the dependency relationship between the multiple ETL tasks to obtain the ETL process; The task relationship graph is constructed with each of the ETL tasks as a node and the dependency relationships between the ETL tasks as an edge.

5. The method according to claim 4, characterized in that The state signal at least includes: a task start signal and a task end signal, and executing the ETL process includes: Determine the initial ETL task located at the most upstream in the task relationship graph, and execute the ETL task; For each ETL task except the initial ETL task in the task relationship graph, when task end signals of all upstream ETL tasks adjacent to the ETL task are obtained and each of the task end signals passes the rationality check, the execution of the ETL task is triggered.

6. The method according to claim 5, characterized in that Performing a rationality check on each of the status signals includes: For each ETL task, a rationality check is performed on each status signal generated by the ETL task, wherein the rationality check includes: a uniqueness check and a logic closed-loop check, wherein the uniqueness check is used to check whether each status signal of the ETL task is unique, and the logic closed-loop check is used to check whether all status signals of the ETL task include a closed loop from the task start signal to the task end signal.

7. The method according to claim 4, characterized in that Perform data tracing analysis on the abnormal ETL task according to the task relationship graph to determine the source of the abnormality, including: Starting from the abnormal ETL task, reversely traverse all upstream ETL tasks of the abnormal ETL task in the task relationship graph to determine the abnormal source causing the exception, wherein the abnormal source includes at least one of the following: abnormal device data in the device data to be processed, and the error handling process in the ETL task.

8. The method according to claim 7, characterized in that Correcting the abnormal source includes: In the case where the abnormal source is abnormal device data in the device data to be processed, reacquire device data from the device corresponding to the abnormal device data and replace the abnormal device data, or modify the abnormal device data, wherein the modification method includes at least one of the following: abnormal value deletion and blank value interpolation; In the case where the exception source is an error handling process in the ETL task, the ETL process of the ETL task having the error handling process is reset.

9. The method according to claim 1, characterized in that: The method further comprises: After the ETL process is executed, a device data processing report is generated, wherein the device data processing report includes at least one of the following: the source of the device data to be processed, the device data to be processed, the data processing results of each of the ETL tasks, and an exception processing log.

10. A device data processing apparatus, characterized in that: include: An acquisition module, used for acquiring device data to be processed, and determining an ETL process corresponding to the device data to be processed and a task relationship map corresponding to the ETL process, wherein the ETL process includes a plurality of ETL tasks, and the task relationship map includes dependency relationships between the plurality of ETL tasks; An execution module, used to execute the ETL process and, during the execution process, obtain a status signal generated by each ETL task, wherein the status signal is used to reflect the execution status of the ETL task; A verification module is used to perform a rationality check on each of the state signals, and when any state signal is abnormal, determine the abnormal ETL task corresponding to the abnormal state signal, perform data tracing analysis on the abnormal ETL task according to the task relationship graph, and determine the abnormal source causing the abnormality; The correction module is used to correct the abnormal source and re-execute the ETL process starting from the first ETL task corresponding to the abnormal source in the task relationship map.

11. A computer program product, characterized in that include: A computer program, wherein when the computer program is executed by a processor, the device data processing method according to any one of claims 1 to 9 is implemented.

12. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the device data processing method according to any one of claims 1 to 9 through the computer program.

Images