Multi-source heterogeneous data analysis method, device, computer equipment, readable storage medium and program product

Abstract

The present application relates to a multi-source heterogeneous data analysis method, device, computer equipment, readable storage medium and program product. Inspection data is obtained from the source database of the target device, the monitoring strategy is determined based on the inspection data and the strategy matching, the inspection data is reconstructed based on the monitoring strategy, and the rule configuration information and the target inspection data that meets the target format corresponding to the rule configuration information are obtained. Then, the target inspection data in the Kafka queue is written into the Kafka queue, and an abnormal analysis is performed on the target inspection data in the Kafka queue based on the preset distributed stream processing framework and the rule configuration information. Compared with the traditional manual inspection data analysis, this solution pre-configures the inspection strategy and rule template of the power grid equipment, obtains the corresponding rule configuration based on the user input, reconstructs the inspection data of different data sources, and inspects the inspection data in combination with the rule configuration, thereby improving the analysis efficiency of multi-source heterogeneous data.

Description

Technical Field

[0001] The present application relates to the field of electric power technology, and in particular to a multi-source heterogeneous data analysis method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the rapid development of the power industry, the scale of power plants is becoming larger and larger, and the data inspection tasks are increasing. In addition, the inspection of power data involves the inspection of heterogeneous data from multiple data sources. The inspection data from different data sources have different data formats, so corresponding strategies need to be adopted for analysis. At present, the inspection method for multi-source heterogeneous data in power systems is usually achieved by manually analyzing the power data from each data source. However, manual inspection of multi-source heterogeneous data will reduce the inspection efficiency.

[0003] Therefore, the current multi-source heterogeneous data analysis methods in the power sector have the defect of low efficiency. Summary of the invention

[0004] Based on this, it is necessary to provide a multi-source heterogeneous data analysis method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve efficiency in response to the above technical problems.

[0005] In a first aspect, the present application provides a multi-source heterogeneous data analysis method, the method comprising:

[0006] Acquire corresponding inspection data from a source database corresponding to the target device in the power grid, and determine the corresponding monitoring strategy according to the matching results between the inspection data and the measurement points in the candidate monitoring strategy;

[0007] Reconstruct the inspection data according to the monitoring strategy, obtain rule configuration information corresponding to the inspection data, and target inspection data that meets the target format corresponding to the rule configuration information, and write the target inspection data into the Kafka queue;

[0008] Based on a preset distributed stream processing framework, the target inspection data is obtained from the Kafka queue, and based on the rule configuration information, an abnormality analysis is performed on the target inspection data.

[0009] In one embodiment, the method further comprises:

[0010] When an inspection task configuration request is detected, the preset project configuration page is output;

[0011] Acquire the inspection cycle, information of the target device and inspection items entered by the user in the preset project configuration page;

[0012] Generate corresponding push configuration information according to the inspection cycle, the information of the target device and the inspection items;

[0013] Based on the push configuration information, an abnormality analysis result corresponding to the target device is output.

[0014] In one embodiment, the method further comprises:

[0015] Obtaining the device information, measurement points, state values ​​and judgment rules input by the user based on a preset rule template;

[0016] Obtain corresponding inspection items according to the device information;

[0017] Obtaining a trigger condition corresponding to the inspection task according to the measuring point and the state value;

[0018] Obtaining the rule configuration information according to the device information, the inspection items and the triggering conditions;

[0019] A monitoring strategy corresponding to the inspection task is obtained according to the measuring point, the state value, the judgment rule and the rule configuration information.

[0020] In one embodiment, reconstructing the inspection data according to the monitoring strategy to obtain rule configuration information corresponding to the inspection data and target inspection data satisfying a target format corresponding to the rule configuration information includes:

[0021] According to the measuring point in the monitoring strategy, the corresponding rule configuration information is determined, and the inspection data is queried according to the measuring point to obtain the inspection data of the target part corresponding to the measuring point in the inspection data;

[0022] Associating the target partial inspection data with the rule configuration information to obtain associated partial inspection data;

[0023] According to the target format corresponding to the rule configuration information, the associated partial inspection data is converted into target inspection data that meets the target format.

[0024] In one embodiment, performing abnormal analysis on the target inspection data based on the rule configuration information includes:

[0025] Determine the abnormal state value corresponding to the target inspection data according to the state value in the rule configuration information and the judgment rule;

[0026] A target state value in the target inspection data is acquired, and an abnormality analysis result corresponding to the target inspection data is obtained according to a comparison result between the target state value and the abnormal state value.

[0027] In one embodiment, after performing abnormal analysis on the target inspection data, the method further includes:

[0028] If the abnormal analysis result is that the target state value is consistent with the abnormal state value, an alarm message is generated according to the target inspection data, and the alarm message is stored in an alarm message queue;

[0029] Based on the alarm message queue, the alarm information is pushed to a management terminal corresponding to the target device.

[0030] In a second aspect, the present application provides a multi-source heterogeneous data analysis device, the device comprising:

[0031] A determination module, used to obtain corresponding inspection data from a source database corresponding to a target device in the power grid, and determine a corresponding monitoring strategy according to a matching result between the inspection data and a measurement point in a candidate monitoring strategy;

[0032] A reconstruction module, used to reconstruct the inspection data according to the monitoring strategy, obtain the rule configuration information corresponding to the inspection data, and the target inspection data satisfying the target format corresponding to the rule configuration information, and write the target inspection data into the Kafka queue;

[0033] The analysis module is used to obtain the target inspection data from the Kafka queue based on a preset distributed stream processing framework, and perform an abnormality analysis on the target inspection data based on the rule configuration information.

[0034] In a third aspect, the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0035] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0036] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0037] The above-mentioned multi-source heterogeneous data analysis method, device, computer equipment, computer-readable storage medium and computer program product obtain inspection data from the source database of the target device, determine the monitoring strategy based on the inspection data and the strategy matching, reconstruct the inspection data based on the monitoring strategy, obtain the rule configuration information and the target inspection data that meets the target format corresponding to the rule configuration information, write it into the Kafka queue, and perform abnormal analysis on the target inspection data in the Kafka queue based on the preset distributed stream processing framework and rule configuration information. Compared with the traditional manual inspection data analysis, this solution pre-configures the inspection strategy and rule template of the power grid equipment, obtains the corresponding rule configuration based on the user input, reconstructs the inspection data of different data sources, and inspects the inspection data in combination with the rule configuration, thereby improving the analysis efficiency of multi-source heterogeneous data. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A schematic diagram of a multi-source heterogeneous data analysis method in one embodiment;

[0040] Figure 2 is a flowchart of a reconstruction step in an embodiment;

[0041] Figure 3 A schematic diagram of a process of analyzing multi-source heterogeneous data in another embodiment;

[0042] Figure 4 A schematic diagram of a data modeling structure in one embodiment;

[0043] Figure 5 A schematic diagram of a process of analyzing multi-source heterogeneous data in another embodiment;

[0044] Figure 6 is a structural block diagram of a multi-source heterogeneous data analysis device in one embodiment;

[0045] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] In one embodiment, Figure 1 As shown, a multi-source heterogeneous data analysis method is provided. This embodiment uses the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server, including the following steps S202 to S206. Among them:

[0048] Step S202, obtaining corresponding inspection data from a source database corresponding to the target device in the power grid, and determining a corresponding monitoring strategy according to a matching result between the inspection data and the measuring points in the candidate monitoring strategy.

[0049] Among them, there are multiple power field devices in the power grid. These devices have different data sources and are stored in different structures. In order to ensure the normal operation of each device in the power grid, it is necessary to analyze the inspection data of these multi-source heterogeneous devices. The terminal can pre-configure the monitoring strategy for each device in the power grid, and pre-configure the rule template for analyzing the inspection data. Among them, the rule template can be used to generate a monitoring strategy containing rule configuration information. The rule template of the monitoring strategy is common to all candidate monitoring strategies and only needs to be configured once, and there is only one.

[0050] The terminal can output the above rule template based on the monitoring strategy, and the user can make corresponding selections at the corresponding options in the rule template. The terminal can pre-acquire the task attribute information for the inspection task input by the user based on the preset rule template, such as the data to be monitored and the abnormal judgment logic of the data. Among them, the target device can be a device in the target system in the power grid, and the rule configuration information can be used to determine the inspection analysis strategy for the inspection data generated by the target device. During the specific analysis, the terminal can continuously detect the source database corresponding to the target device in the power grid. When the inspection data is detected, the corresponding monitoring strategy is determined according to the matching results of the inspection points with the measurement points in multiple candidate monitoring strategies. Among them, the measurement points in the monitoring strategy can be predetermined data items that need to be analyzed.

[0051] Step S204, reconstruct the inspection data according to the monitoring strategy, obtain the rule configuration information corresponding to the inspection data, and the target inspection data that meets the target format corresponding to the rule configuration information, and write the target inspection data into the Kafka queue.

[0052] Among them, each device in the above-mentioned power grid can be distributed in different systems, and each system has a corresponding database, that is, there are multiple data sources, and the data structure of each system is different, that is, the data of each system in the power grid is multi-source heterogeneous data. The above-mentioned target device has a corresponding source database, and the source database stores the inspection data of the target device. The terminal can obtain the corresponding inspection data from the source database corresponding to the target device. Among them, since the data source and structure of the inspection data are different from the data source and structure when the terminal performs inspection data analysis, the terminal needs to reconstruct the inspection data. The terminal can reconstruct the inspection data according to the monitoring strategy, obtain the rule configuration information corresponding to the inspection data, and reconstruct the inspection data according to the monitoring strategy into the target inspection data that meets the target format corresponding to the rule configuration information. Among them, the target format of the data when performing inspection analysis can be preset in the rule configuration information, so that the inspection data is used for rule calculation. The terminal can write the above-mentioned reconstructed target inspection data into a stream processing queue, such as a Kafka queue, and perform inspection data analysis and calculation through stream processing, which can improve the accuracy, readability and maintainability of the data and reduce the development cost and risk of the system.

[0053] Step S206, based on the preset distributed stream processing framework, obtain the target inspection data from the Kafka queue, and perform anomaly analysis on the target inspection data based on the rule configuration information.

[0054] Among them, a preset distributed stream processing framework can be pre-deployed in the terminal, and the above-mentioned target inspection data can be written into the Kafka queue, so that the terminal can obtain the target inspection data from the Kafka queue based on the preset distributed stream processing framework, and interpret, calculate and analyze the target inspection data based on the inspection strategy and configuration in the rule configuration information, output the corresponding rule calculation results, and realize the abnormal analysis of the target inspection data. Among them, the terminal can save the abnormal analysis results of the target inspection data into the Kafka topic, so that the abnormal analysis results can be output through the Kafka message subscription method. Among them, the terminal can output corresponding messages according to the different abnormal analysis results. For example, for the abnormal analysis results that determine that the target inspection data has abnormalities, the terminal can output alarm information.

[0055] In the above multi-source heterogeneous data analysis method, inspection data is obtained from the source database of the target device, the monitoring strategy is determined based on the inspection data matching the strategy, and the inspection data is reconstructed based on the monitoring strategy. After obtaining the rule configuration information and the target inspection data in the target format corresponding to the rule configuration information, the target inspection data in the Kafka queue is written into the Kafka queue, and the target inspection data in the Kafka queue is analyzed for abnormalities based on the preset distributed stream processing framework and rule configuration information. Compared with the traditional manual inspection data analysis, this solution pre-configures the inspection strategy and rule template of the power grid equipment, obtains the corresponding rule configuration based on the user input, reconstructs the inspection data from different data sources, and inspects the inspection data in combination with the rule configuration, thereby improving the analysis efficiency of multi-source heterogeneous data.

[0056] In one embodiment, it also includes: when an inspection task configuration request is detected, outputting a preset project configuration page; obtaining the inspection cycle, target device information and inspection items entered by the user in the preset project configuration page; generating corresponding push configuration information according to the inspection cycle, target device information and inspection items; based on the push configuration information, outputting the corresponding abnormality analysis results for the target device.

[0057] In this embodiment, the terminal can configure the message push and output in advance, so that the terminal can determine the period in which the management end corresponding to the target device needs to call data based on the configured push configuration information. Among them, the user can trigger the corresponding configuration request, for example, by clicking a button. When the terminal detects the configuration request, it can output the preset project configuration page. Among them, the preset project configuration page can be a customized visual configuration interface, and the preset project configuration page has multiple inspection options to be filled in. The user can enter the inspection cycle, the information of the target device, and the inspection items in the preset project configuration page. Among them, the inspection cycle can represent the trigger time of outputting and pushing the abnormal analysis results of the target device to the management end of the target device, the information of the target device can be the name of the target device, the device type and the system to which it belongs, and the inspection items can be the items and data types that need to be inspected. The terminal can obtain the inspection cycle, the information of the target device, and the inspection items entered by the user in the above-mentioned preset project configuration page, so that the terminal can generate the corresponding push configuration information according to the above-mentioned inspection cycle, the information of the target device, and the inspection items.

[0058] Specifically, the terminal is provided with an inspection project management module, through which the terminal can output a visual configuration page, such as the above-mentioned preset project configuration page, so that the terminal can configure and manage inspection projects and inspection items in the preset project configuration page according to the inspection cycle, inspection project (information of the target device) and the hierarchy of inspection items to form push configuration information. In addition, the terminal can also store the above-mentioned configured push configuration information in a database associated with the terminal for persistent storage, so that the terminal can trigger the output and push of abnormal analysis results of the target device based on the saved push configuration information.

[0059] Through this embodiment, the terminal can use the preset project configuration page to configure the push configuration information for the target device, so that the terminal can output and push the inspection data results to the target device based on the push configuration information, thereby improving the efficiency of inspection data analysis in the power grid.

[0060] In one embodiment, it also includes: obtaining device information, measurement points, status values ​​and judgment rules input by the user based on a preset rule template; obtaining corresponding inspection items according to the device information; obtaining trigger conditions corresponding to the inspection task according to the measurement points and status values; obtaining rule configuration information according to the device information, the inspection items and trigger conditions; obtaining a monitoring strategy corresponding to the inspection task according to the measurement points, status values, judgment rules and rule configuration information.

[0061] In this embodiment, the terminal can configure the monitoring strategy in advance, wherein the monitoring strategy includes information such as rule configuration information for inspection information. The terminal can output a preset rule configuration page, which includes a preset rule template, so that the terminal can obtain the device information, measurement points, state values ​​and judgment rules input by the user based on the preset rule template as task attribute information. Among them, the device information represents the name, data type, device type and other information of the target device; the measurement point represents the predetermined data that needs to be inspected and analyzed in the target device; the state value represents the specific value of the data corresponding to the above measurement point, wherein the state of the data is different under different value conditions; the judgment rule represents the normal value and abnormal value of the state value and other information.

[0062] Among them, the terminal can determine multiple pieces of information in the rule configuration information based on the task attribute information. For example, the terminal can obtain the corresponding inspection items based on the device information, that is, determine the relevant information of the data that needs to be inspected and analyzed. The terminal can also obtain the trigger conditions corresponding to the inspection task based on the measurement points and status values, so that the terminal can obtain the rule configuration information based on the device information, inspection items and trigger conditions. Thus, the terminal can obtain the monitoring strategy corresponding to the inspection task based on the measurement points, status values, judgment rules and rule configuration information, where the monitoring strategy includes how to detect the status value and how to judge whether the status value is normal.

[0063] Specifically, an expert strategy management module is provided in the terminal, and the terminal outputs a customized configuration interface through the expert strategy management module, such as the above-mentioned preset rule configuration page. The terminal completes the configuration of the rule template, trigger conditions, and data inspection strategy (monitoring strategy) in sequence on the preset rule configuration page, and saves them persistently in the database associated with the terminal.

[0064] Among them, the rule template of the above inspection strategy can be common to all inspection strategies, and the terminal only needs to configure it once and only one, and the trigger condition can be reused by multiple strategies. Among them, for the rule template, the terminal can define the policy structure by configuring the rule template. For example, the configuration class of the inspection strategy "com.cimstech.st.ms.ia.common.bean.conf.RuleCaseConf_X001" specifies the policy interpreter by configuring the calculation pipeline. For example, the terminal specifies the inspection strategy interpreter through the rule configuration information to interpret the above class, that is, "X001 logical relationship determination between multiple switch quantities". The terminal can also configure the trigger condition in the rule template, which defines the trigger condition for the start or end of the execution of the strategy. For example: a switch measurement point "point_01", when the terminal detects that its state changes from 0 to 1, it starts to execute the analysis strategy. The terminal can be configured through a customized interface by selecting the corresponding device object, measurement point, state value, etc., and then converted into Aviator expression for calculation in the background, for example: "lambda(e)->str(e.state_1) == 'CLOSED'&& str(e.state_2) == 'OPEN' end , json.object(states.getMapState('mp_states','{} # point_01')) ". The above expression means that the corresponding value of the key "{}# point_01" (the key format is: rule id. measurement point id) is taken out from the state storage named "mp_states" through the states.getMapState function, and passed to the function lambad to calculate the value of the expression "str(e.state_1) == 'CLOSED' && str(e.state_2) == 'OPEN' end". Aviator is a high-performance Java expression evaluation engine based on bytecode, focusing on providing fast expression calculation capabilities.

[0065] Based on the above configurations, the terminal can form a data inspection strategy (monitoring strategy) in combination with the judgment rules. The data inspection strategy includes four parts. It includes basic information, trigger conditions, monitoring signals and inspection items. In the basic information, you can configure a custom policy name, inspection target object, rule template, etc. The selection of the rule template will determine the specific configuration items of the subsequent strategy. For the trigger condition, the terminal can select the trigger condition applicable to the current policy from the existing trigger conditions. The terminal can determine the monitoring signal based on the input judgment rule. For example, the terminal configures the judgment logic of data inspection: through the state change of the device-associated measurement point "point_02", it is judged whether the device operating environment is normal. When the measurement point state is detected to change from 1 to 0, it is judged to be abnormal. The logic of the above monitoring signal can also be configured through a customized interface and converted into an Aviator expression in the background for calculation. The terminal can also configure the corresponding inspection items, such as binding specific inspection items through the settings of the inspection cycle, inspection items, inspection items, and inspection types.

[0066] Through this embodiment, the terminal can combine information such as equipment information, status values, measurement points and judgment rules to form rule configuration information, and perform inspection and analysis of multi-source heterogeneous data in the power grid based on the rule configuration information, thereby improving analysis efficiency.

[0067] In one embodiment, inspection data is reconstructed according to a monitoring strategy to obtain rule configuration information corresponding to the inspection data and target inspection data that satisfies a target format corresponding to the rule configuration information, including: determining corresponding rule configuration information according to a measuring point in the monitoring strategy, querying the inspection data according to the measuring point, and obtaining target partial inspection data corresponding to the measuring point in the inspection data; associating the target partial inspection data with the rule configuration information to obtain associated partial inspection data; and converting the associated partial inspection data into target inspection data that satisfies the target format according to the target format corresponding to the rule configuration information.

[0068] In this embodiment, the terminal can predetermine the part of the inspection data of the target device that needs to be monitored, and the part can be represented by a measuring point. The terminal can determine the corresponding rule configuration information based on the measuring point set in the monitoring strategy, and query the inspection data corresponding to the above-mentioned target device based on the measuring point, so as to obtain the target part of the inspection data corresponding to the measuring point. The terminal can associate the above-mentioned target part of the inspection data with the rule configuration information to obtain the associated part of the inspection data. Among them, the rule configuration information has a corresponding target format, and the terminal can convert the associated part of the inspection data into the target inspection data that meets the target format according to the target format corresponding to the rule configuration information. That is, the target inspection data can be the inspection data associated with the measuring point and meeting the target format.

[0069] Specifically, a data collection module may be preset in the terminal. Through the data collection task, the terminal pulls the inspection data from the source database of the target device, such as the inspection data from the audio-visual master station system and the centralized control system. The terminal associates the inspection data corresponding to the measurement points in the inquiry year data with the inspection strategy in the rule configuration information, and Figure 2 As shown, Figure 2 : is a flowchart of the reconstruction step in an embodiment. The reconstruction is the basic data (target inspection data) suitable for rule calculation, including the measurement point pid, timestamp, measurement point attribute, rule id, rule expression, rule attribute, inspection item id, inspection item id, inspection content, inspection type, inspection data and data attribute, etc. The terminal can write the above reconstructed target inspection data into the Kafka queue as a streaming data source for subsequent rule calculation.

[0070] Through this embodiment, the terminal can reconstruct inspection data from different data sources and different structures into target inspection data in a data format that satisfies rule configuration information, and perform anomaly analysis based on the target inspection data, thereby improving the efficiency of anomaly analysis of equipment in the power grid.

[0071] In one embodiment, based on the rule configuration information, an abnormal analysis is performed on the target inspection data, including: determining the abnormal state value corresponding to the target inspection data according to the state value and judgment rule in the rule configuration information; obtaining the target state value in the target inspection data, and obtaining the abnormal analysis result corresponding to the target inspection data according to the comparison result between the target state value and the abnormal state value.

[0072] In this embodiment, the terminal can perform anomaly analysis based on the strategy in the rule configuration information. For example, the rule configuration information includes a state value and a judgment rule, and the terminal can determine the abnormal state value corresponding to the target inspection data according to the state value and the judgment rule in the rule configuration information. For example, the state value includes a normal state value and an abnormal state value, and the terminal determines the abnormal state value in each state value by parsing the judgment rule. Thus, the terminal can obtain the target state value in the target inspection data, and obtain the abnormal analysis result corresponding to the target inspection data according to the comparison result between the target state value and the abnormal state value.

[0073] Specifically, a stream computing pipeline module can be preset in the terminal, which can be a Flink computing pipeline module. Flink is an open source computing framework for distributed data stream processing and batch data processing. It supports multiple data sources and receivers, and provides developers with a multi-layer programming interface, which can support the selection of appropriate interfaces for programming according to needs and scenarios. It has the characteristics of low latency, high throughput, accurate results and good fault tolerance. The terminal presets the stream computing pipeline module, configures the computing pipeline (SQL or Python) for the data inspection strategy (rule configuration information), pulls data from the specified data source, such as the Kafka queue, and interprets and calculates the fixed-format data inspection basic data according to the configuration, and finally outputs the rule calculation results and saves them to the Kafka topic.

[0074] Among them, the abnormal analysis results include normal data and abnormal data. When the data is normal, the target state value is inconsistent with the abnormal state value; when the data is abnormal, the target state value is consistent with the abnormal state value, and the terminal can issue an alarm for the abnormal data.

[0075] In one embodiment, after performing an abnormal analysis on the target inspection data, it also includes: if the abnormal analysis result is that the target state value is consistent with the abnormal state value, generating alarm information based on the target inspection data, and storing the alarm information in an alarm message queue; based on the alarm message queue, pushing the alarm information to the management terminal corresponding to the target device.

[0076] In this embodiment, the terminal can judge the abnormal analysis result. If it is determined that the abnormal analysis result indicates that the target inspection data is abnormal, the terminal can output an alarm message. For example, if the terminal detects that the abnormal analysis result is that the above-mentioned target state value is consistent with the abnormal state value, it means that the inspection data of the target device is abnormal, so that the terminal can generate an alarm message based on the above-mentioned target inspection data and store the alarm message in the alarm message queue. The alarm message queue can be a Kafka message queue. The terminal can push the alarm message to the management terminal corresponding to the target device through the alarm message queue. The management terminal of the target device can be the management terminal of the system where the target device is located. After receiving the alarm message, the management terminal can maintain and manage the target device.

[0077] Specifically, an alarm module can be set in the terminal, and the alarm module can subscribe through Kafka messages, obtain alarm messages of data inspection from corresponding topics, store them in alarm message queues, and push them to other applications, such as the above-mentioned management terminal. Among them, the management terminal can have a message subscription relationship with the alarm module.

[0078] Through the above embodiments, the terminal can perform anomaly analysis on the target inspection data based on the inspection strategy in the rule configuration information, using the status value and judgment rules, and when it is determined that an anomaly exists, push an alarm message to the corresponding management terminal through the alarm message queue, thereby improving the inspection and analysis efficiency of multi-source heterogeneous data.

[0079] In an exemplary embodiment, Figure 3 As shown, Figure 3 It is a flow chart of a multi-source heterogeneous data analysis method in another embodiment. In this embodiment, the terminal manages the overall data inspection process by dividing it into multiple modules, such as inspection project management, expert strategy management, data aggregation, Kafka middleware, Flink streaming computing pipeline, alarm module, etc. These modules are relatively independent of each other, easy to maintain, and related to each other, and can collaboratively complete multi-source heterogeneous data inspection, including the formulation, execution and alarm of inspection strategies. This embodiment adopts data modeling, data integration, big data, cloud computing, artificial intelligence, Flink and other technologies. Among them, data modeling technology involves abstract organization of data, determining the jurisdiction of the database, data organization form, etc., until it is converted into a real database. Here, by designing the database, including table structure, data type, table association, etc., in advance to meet the data rule calculation, the storage of many data such as inspection projects, inspection items, inspection strategies, inspection results, etc. is realized. Data integration technology can organically integrate inspection data from different sources and structures, which mainly includes data cleaning, data conversion, data integration, etc. Data cleaning: Eliminate errors, duplications and outliers in data to ensure data quality. Data conversion: Convert data from different sources and formats into a unified format and structure to facilitate subsequent analysis and processing. Data integration: Integrate data from multiple data sources into a unified repository to facilitate query and analysis. Big data technology involves multiple links such as data collection, storage, processing, analysis and mining, aiming to extract valuable information from massive amounts of data.

[0080] The correlation between some key data types in the data inspection process can be as follows: Figure 4 As shown, Figure 4A schematic diagram of the structure of data modeling in one embodiment. Specifically, it includes the following data types: equipment ledger, including the main target objects of data inspection, such as parent node identification, name, plant identification, altitude, asset category code, asset code, property unit, asset status, commissioning date, equipment type, equipment identity code, equipment adjustment and management unit, installation unit, whether it is asset-level equipment, whether it is virtual, topography, latitude, longitude, unit of measurement, equipment model, equipment operation and maintenance department, equipment operation and maintenance team, equipment operation and maintenance unit, factory number, factory date, manufacturer, manufacturer's current name, property attributes, quantity, RFID (Radio Frequency Identification) code, operation number, operation location, current status of equipment, time to enter the current status, supplier, voltage level, original factory warranty period, search field, arrangement order under the same parent node, asset code, KKS code, full path of identification, whether it has been deleted, functional location type identification, functional location type name, class model identification, class model name, typical tree structure identification, label, creation time, creator user identification, last modification time, last modification user identification, functional location type, complete set of equipment, component, centralized control object, etc.

[0081] In addition to the equipment ledger, there are also spatial objects, etc.; machine vision measurement points, including associated measurement points of inspection objects. In addition, there are machine auditory measurement points, switch measurement points, etc., such as machine vision algorithm, camera preset position, description, long name, monitoring object, name, source system identifier, search field, plant station identifier, ledger object identifier associated with the measurement point, creation time, creator user identifier, measurement type, possible values ​​of data, class model identifier, class model name, label, switch quantum type, action quantity or state quantity, last modification time, last modification user identifier and collection cycle, etc.

[0082] Inspection items, including the upper-level organization of inspection items, are used to classify and manage inspection items, such as creation time, creator user ID, last modification time, last modification user ID, project name, remarks, plant station ID, and camera type.

[0083] Inspection items include task descriptions of data inspections, related inspection objects, measuring points and other information, such as basis or experience, inspection content, creation time, creator user ID, last modification time, last modifier user ID, inspection target object ID, inspection target object type, inspection cycle, project ID, plant ID, specialty, expected value, inspection method, specialty category, power grid management platform inspection item ID and inspection path, etc.

[0084] Rule templates include templates for defining policy structures and policy interpreters, such as code, creator, description, whether enabled, input parameter description, last modifier, name, output parameter description, calculation pipeline code, and configuration class name of rule configuration. For data inspection policies, the above policies all use inspection rule templates.

[0085] The trigger conditions, including the prerequisites for executing the data inspection strategy, can be reused by multiple strategies, including the start trigger condition, creator, description, end trigger condition, temporary field, auxiliary migration data, whether it is an operating condition determination, the last modifier, attached to this ledger object, the path name of the attached ledger object, name, condition name, search field, plant site identification, start trigger condition expression, end trigger condition expression, and the full path of the ledger object identification.

[0086] Inspection strategies / rules include specific data inspection strategies that can execute rule calculations after substituting measurement point data (values), which are bound to inspection items, inspection objects, measurement points and other information, such as trigger condition identification, triggering stage of start trigger, category name, fault warning, operation suggestion reminder, creator, trigger condition identification, triggering stage of start trigger, last modifier, ledger object identification, name, remarks, strategy configuration, monitoring signal, plant station identification, statistical start time, template identification, full path of ledger object identification, searched field, whether it is enabled, enabled or disabled, expert strategy template code, whether it has been deleted, mark for deletion first, clean up after expiration, rule example identification; measurement point data include specific inspection data obtained from the source data end, including measurement point ID, measurement point value, time and other information.

[0087] It also includes eventlog (status log), including event time (milliseconds), event identifier, event number, event time, status value, source status, and target status.

[0088] like Figure 5 As shown, Figure 5The figure is a flowchart of a multi-source heterogeneous data analysis method in another embodiment. When the terminal is performing data inspection, it can configure and manage inspection projects and inspection items in the inspection project management module through a visual configuration interface according to the inspection cycle, inspection project, and inspection item hierarchy, and save them persistently to the database; in the expert strategy management module, the terminal completes the configuration of rule templates, trigger conditions, and data inspection strategies in sequence through a customized configuration interface to form rule configuration information, and saves it persistently to the database; in the data collection module, the terminal pulls inspection data from source data systems, such as audio-visual master station systems, centralized control systems, etc., through data collection tasks, associates it with inspection strategies through measurement points, and reconstructs it into applicable The basic data for rule calculation is written into the Kafka topic as a streaming data source for subsequent rule calculations. In the Flink computing pipeline module, for the computing pipeline configured for the data inspection strategy, the terminal pulls data from the specified data source, such as Kafka, and interprets and calculates the fixed-format data inspection basic data according to the configuration, and finally outputs the rule calculation results to form anomaly analysis results, which are saved to the Kafka topic. Among them, the alarm module subscribes to Kafka messages, and the terminal can obtain data inspection alarm messages from the corresponding topics, store them in the alarm message queue, and push them to other applications, such as the management terminal of other source data systems.

[0089] Through the above embodiments, the terminal can pre-configure the inspection tasks and rule templates of the power grid equipment, obtain the corresponding rule configuration based on the user input, reconstruct the inspection data from different data sources, and inspect the inspection data in combination with the rule configuration, thereby improving the analysis efficiency of multi-source heterogeneous data.

[0090] In addition, in the configuration of multi-source heterogeneous data inspection strategies, the strategy structure is defined through configuration classes (rule templates). Users only need to select the corresponding rule template to automatically generate the configuration items of the corresponding rules, avoiding the tediousness of manual selection and configuration; in the data persistence of multi-source heterogeneous data inspection strategies, through high cohesion and low coupling model design, repetitive data is extracted and the data type is refined, which not only reduces the complexity of the data structure, but also avoids a large amount of data redundancy and is easier to maintain; in the execution of multi-source heterogeneous data inspection strategies, based on Aviator expressions and Flink computing pipelines, the separation of computing logic and executors is achieved; users only need to configure the inspection strategy according to the guidance, and the flink computing pipeline is responsible for the interpretation and execution of the strategy; logical changes in individual strategies do not affect the execution of other strategies.

[0091] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0092] Based on the same inventive concept, the embodiment of the present application also provides a multi-source heterogeneous data analysis device for implementing the multi-source heterogeneous data analysis method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more multi-source heterogeneous data analysis device embodiments provided below can refer to the limitations of the multi-source heterogeneous data analysis method above, and will not be repeated here.

[0093] In an exemplary embodiment, Figure 6 As shown, a multi-source heterogeneous data analysis device is provided, including: a determination module 500, a reconstruction module 502 and an analysis module 504, wherein:

[0094] The determination module 500 is used to obtain corresponding inspection data from a source database corresponding to the target device in the power grid, and determine the corresponding monitoring strategy according to the matching results between the inspection data and the measurement points in the candidate monitoring strategy.

[0095] The reconstruction module 502 is used to reconstruct the inspection data according to the monitoring strategy, obtain the rule configuration information corresponding to the inspection data, and the target inspection data that meets the target format corresponding to the rule configuration information, and write the target inspection data into the Kafka queue.

[0096] The analysis module 504 is used to obtain target inspection data from the Kafka queue based on a preset distributed stream processing framework, and perform abnormal analysis on the target inspection data based on rule configuration information.

[0097] In one embodiment, the above-mentioned device also includes: a first configuration module, which is used to output a preset project configuration page when a patrol task configuration request is detected; obtain the patrol cycle, target device information and patrol items entered by the user in the preset project configuration page; generate corresponding push configuration information according to the patrol cycle, target device information and patrol items; based on the push configuration information, output the abnormality analysis results corresponding to the target device.

[0098] In one embodiment, the above-mentioned device also includes: a second configuration module, which is used to obtain the equipment information, measurement points, status values ​​and judgment rules input by the user based on a preset rule template; obtain the corresponding inspection items according to the equipment information; obtain the trigger conditions corresponding to the inspection task according to the measurement points and status values; obtain the rule configuration information according to the equipment information, inspection items and trigger conditions; obtain the monitoring strategy corresponding to the inspection task according to the measurement points, status values, judgment rules and rule configuration information.

[0099] In one embodiment, the reconstruction module 502 is used to determine the corresponding rule configuration information according to the measuring points in the monitoring strategy, query the inspection data according to the measuring points, and obtain the target partial inspection data corresponding to the measuring points in the inspection data; associate the target partial inspection data with the rule configuration information to obtain the associated partial inspection data; and convert the associated partial inspection data into target inspection data that meets the target format according to the target format corresponding to the rule configuration information.

[0100] In one embodiment, the above-mentioned analysis module 504 is used to determine the abnormal state value corresponding to the target inspection data according to the state value and judgment rule in the rule configuration information; obtain the target state value in the target inspection data, and obtain the abnormal analysis result corresponding to the target inspection data according to the comparison result between the target state value and the abnormal state value.

[0101] In one embodiment, the above-mentioned device also includes: an alarm module, which is used to generate alarm information according to the target inspection data if the abnormal analysis result is that the target state value is consistent with the abnormal state value, and store the alarm information in the alarm message queue; based on the alarm message queue, the alarm information is pushed to the management terminal corresponding to the target device.

[0102] Each module in the above-mentioned multi-source heterogeneous data analysis device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0103] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 7As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a multi-source heterogeneous data analysis method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0104] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0105] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above-mentioned multi-source heterogeneous data analysis method when executing the computer program.

[0106] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the multi-source heterogeneous data analysis method described above is implemented.

[0107] In one embodiment, a computer program product is provided, including a computer program, which implements the above-mentioned multi-source heterogeneous data analysis method when executed by a processor.

[0108] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0109] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0110] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0111] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A multi-source heterogeneous data analysis method, characterized in that: The method comprises: Obtaining corresponding inspection data from a source database corresponding to a target device in the power grid, and determining a corresponding monitoring strategy based on a matching result between the inspection data and a measurement point in a candidate monitoring strategy; the measurement point represents a predetermined data item that needs to be analyzed; According to the measuring point in the monitoring strategy, the corresponding rule configuration information is determined, and the inspection data is queried according to the measuring point to obtain the target partial inspection data corresponding to the measuring point in the inspection data; the target partial inspection data is associated with the rule configuration information to obtain the associated partial inspection data; according to the target format corresponding to the rule configuration information, the associated partial inspection data is converted into the target inspection data that meets the target format; and the target inspection data is written into the Kafka queue; Based on a preset distributed stream processing framework, the target inspection data is obtained from the Kafka queue, and based on the rule configuration information, an abnormality analysis is performed on the target inspection data.

2. The method according to claim 1, characterized in that The method further comprises: Obtain device information, measurement points, status values ​​and judgment rules input by users based on preset rule templates; Obtain corresponding inspection items according to the device information; According to the measuring point and the state value, a trigger condition corresponding to the inspection task is obtained; Obtaining the rule configuration information according to the device information, the inspection items and the triggering conditions; A monitoring strategy corresponding to the inspection task is obtained according to the measuring point, the state value, the judgment rule and the rule configuration information.

3. The method according to claim 2, characterized in that The performing abnormal analysis on the target inspection data based on the rule configuration information includes: Determine the abnormal state value corresponding to the target inspection data according to the state value in the rule configuration information and the judgment rule; A target state value in the target inspection data is acquired, and an abnormality analysis result corresponding to the target inspection data is obtained according to a comparison result between the target state value and the abnormal state value.

4. The method according to claim 3, characterized in that After performing abnormal analysis on the target inspection data, the method further includes: If the abnormal analysis result is that the target state value is consistent with the abnormal state value, an alarm message is generated according to the target inspection data, and the alarm message is stored in an alarm message queue; Based on the alarm message queue, the alarm information is pushed to a management terminal corresponding to the target device.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: When an inspection task configuration request is detected, the preset project configuration page is output; Obtaining the inspection cycle, information of the target device, and inspection items entered by the user in the preset project configuration page; Generate corresponding push configuration information according to the inspection cycle, the information of the target device and the inspection items; Based on the push configuration information, an abnormality analysis result corresponding to the target device is output.

6. A multi-source heterogeneous data analysis device, characterized in that: The device comprises: A determination module, used to obtain corresponding inspection data from a source database corresponding to a target device in the power grid, and determine a corresponding monitoring strategy according to a matching result between the inspection data and a measurement point in a candidate monitoring strategy; the measurement point represents a predetermined data item that needs to be analyzed; A reconstruction module is used to determine the corresponding rule configuration information according to the measuring point in the monitoring strategy, query the inspection data according to the measuring point, and obtain the target partial inspection data corresponding to the measuring point in the inspection data; associate the target partial inspection data with the rule configuration information to obtain the associated partial inspection data; convert the associated partial inspection data into target inspection data that meets the target format according to the target format corresponding to the rule configuration information; and write the target inspection data into a Kafka queue; The analysis module is used to obtain the target inspection data from the Kafka queue based on a preset distributed stream processing framework, and perform an abnormality analysis on the target inspection data based on the rule configuration information.

7. The device according to claim 6, characterized in that The device further includes: a second configuration module, configured to: Obtain device information, measurement points, status values ​​and judgment rules input by users based on preset rule templates; Obtain corresponding inspection items according to the device information; According to the measuring point and the state value, a trigger condition corresponding to the inspection task is obtained; Obtaining the rule configuration information according to the device information, the inspection items and the triggering conditions; A monitoring strategy corresponding to the inspection task is obtained according to the measuring point, the state value, the judgment rule and the rule configuration information.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images