A power transmission data consistency detection method and system

Abstract

The application discloses a power transmission data consistency detection method and system, wherein the method acquires a power transmission data stream of a heterogeneous data link of a power transmission system and a holographic topology atlas corresponding to the heterogeneous data link of the power transmission system; the power transmission data stream comprises a plurality of data blocks; according to the holographic topology atlas, fuzzy fingerprint generation is performed on the power transmission data stream to obtain a first fingerprint stream and a plurality of second fingerprint streams; the first atlas node is a terminal atlas node in all atlas nodes of the holographic topology atlas; according to the first fingerprint stream, terminal consistency detection is performed on all the second fingerprint streams to obtain a consistency detection result of the power transmission data stream. The method can improve the accuracy of power transmission data consistency detection. The application relates to the technical field of power system big data management.

Description

Technical Field

[0001] This invention relates to the field of big data governance technology in power systems, and in particular to a method and system for detecting the consistency of power transmission data. Background Technology

[0002] With the widespread application of distributed intelligent tools in power transmission systems, massive amounts of real-time, dynamic data are frequently flowing between multiple security zones. This data often requires multiple cleaning, aggregation, and format conversions as it traverses different security zones and heterogeneous systems (such as source acquisition, middleware transmission, and database storage). However, due to the large number of systems involved and the fact that the data links are in a "black box" state, data is highly susceptible to inconsistencies in synchronization, logical loss, or accuracy deviations during transmission, making the consistency detection of power transmission data increasingly important.

[0003] Currently, relevant technologies typically rely on single-point detection with fixed thresholds to achieve data consistency testing across nodes in a power transmission system. However, due to the existence of data silos across security zones within the power transmission system, this method struggles to trace the data lineage within the system and is ill-suited to adapting to the varying accuracy of heterogeneous databases, resulting in unsatisfactory accuracy in power transmission data consistency testing.

[0004] Therefore, the problems with the relevant technologies still need to be solved and optimized. Summary of the Invention

[0005] The purpose of this invention is to at least partially solve one of the technical problems existing in the related art.

[0006] Therefore, one objective of this invention is to provide a method and system for detecting the consistency of power transmission data, wherein the method can improve the accuracy of power transmission data consistency detection.

[0007] To achieve the above-mentioned technical objectives, the technical solutions adopted in the embodiments of this application include: In a first aspect, embodiments of this application provide a method for detecting the consistency of power transmission data, including: The transmission data stream of the heterogeneous data link of the power transmission system and the holographic topology map corresponding to the heterogeneous data link of the power transmission system are obtained; the transmission data stream includes several data blocks; Based on the holographic topology map, fuzzy fingerprints are generated for the power transmission data stream to obtain a first fingerprint stream and several second fingerprint streams. The first fingerprint stream is used to characterize the combination of several fuzzy fingerprints of the power transmission data stream at a first map node, and the second fingerprint stream is used to characterize the combination of several fuzzy fingerprints of the power transmission data stream at a second map node. The first map node is the end map node among all map nodes of the holographic topology map; the second map node is the map node other than the first map node among all map nodes of the holographic topology map; each fuzzy fingerprint corresponds to one data block. Based on the first fingerprint stream, end-to-end consistency detection is performed on all second fingerprint streams to obtain the consistency detection result of the power transmission data stream.

[0008] In addition, the method according to the above embodiments of this application may also have the following additional technical features: Furthermore, in one embodiment of this application, obtaining a holographic topology map corresponding to the heterogeneous data links of the power transmission system includes: Obtain the entity data of each entity object in the heterogeneous data link; Entity relationship analysis is performed on all the entity data to obtain relationship analysis information; Based on the relationship analysis information, a graph is generated for all the entity objects to obtain the holographic topological graph.

[0009] Furthermore, in one embodiment of this application, the method further includes: The step of generating a fuzzy fingerprint of the power transmission data stream based on the holographic topology map to obtain a first fingerprint stream and several second fingerprint streams includes: Obtain several node blocks, which are used to represent data blocks in the power transmission data stream of the third graph node; Based on the power transmission data stream, fingerprint synthesis is performed on all the node blocks to obtain a fuzzy fingerprint for each node block; The third fingerprint stream is obtained based on the fuzzy fingerprints of all the node blocks. If the third graph node is the first graph node, then the third fingerprint stream is the first fingerprint stream; or, if the third graph node is the second graph node, then the third fingerprint stream is the second fingerprint stream.

[0010] Furthermore, in one embodiment of this application, the step of performing fingerprint synthesis on all the node blocks based on the power transmission data stream to obtain a fuzzy fingerprint for each node block includes: Statistical features are extracted from the node blocks to obtain block feature vectors; The block feature vector is subjected to noise-resistant quantization to obtain block quantized features; Based on the timestamp of the power transmission data stream, a block fingerprint is generated from the block quantization features to obtain the fuzzy fingerprint of the node block.

[0011] Further, in one embodiment of this application, the step of performing end-to-end consistency detection on all second fingerprint streams based on the first fingerprint stream to obtain the consistency detection result of the power transmission data stream includes: Acquire a tolerance window, a first data fingerprint, and several second data fingerprints; the first data fingerprint is any one of the fuzzy fingerprints in the first fingerprint stream; the second data fingerprint is the fuzzy fingerprint in the second fingerprint stream that corresponds to the first data fingerprint. Fingerprint prediction analysis is performed on all the second data fingerprints to obtain the first expected fingerprint; Based on the tolerance window and the first data fingerprint, spatiotemporal tolerance and fingerprint difference detection are performed on the first expected fingerprint to obtain the consistency detection result.

[0012] Furthermore, in one embodiment of this application, if the consistency detection result is an anomaly, the method further includes: Obtain abnormal blocks, wherein the abnormal blocks are data blocks that are abnormal in the power transmission data stream of the first graph node; Based on the abnormal block, upstream related paths are extracted from the holographic topology map to obtain a candidate fault path set; the candidate fault path set includes several path nodes; Perform hop-by-hop fingerprint gradient difference analysis on the candidate fault path set to obtain the consistency gradient of each path node; Fault location is performed on all the aforementioned consistency gradients to obtain fault objects, which are used to characterize fault map nodes and / or fault edges in the holographic topology map.

[0013] Furthermore, in this embodiment of the application, the method further includes: Obtain the environmental context characteristics of the faulty object and several upstream nodes; Based on the fuzzy fingerprints of all the upstream nodes, a second expected fingerprint corresponding to the faulty object is obtained; Based on the fuzzy fingerprint of the faulty object, perform fingerprint difference analysis on the second expected fingerprint to obtain fingerprint difference features; Based on the environmental context features and the fingerprint difference features, a joint fault analysis is performed to obtain the fault analysis results of the faulty object.

[0014] Secondly, embodiments of this application provide a power transmission data consistency detection system, comprising: The first processing unit is used to acquire the transmission data stream of the heterogeneous data link of the power transmission system, and the holographic topology map corresponding to the heterogeneous data link of the power transmission system; the transmission data stream includes several data blocks; The second processing unit is configured to generate fuzzy fingerprints for the power transmission data stream based on the holographic topology map, resulting in a first fingerprint stream and several second fingerprint streams. The first fingerprint stream characterizes a combination of several fuzzy fingerprints of the power transmission data stream at a first map node, and the second fingerprint stream characterizes a combination of several fuzzy fingerprints of the power transmission data stream at a second map node. The first map node is the end map node among all map nodes in the holographic topology map; the second map node is the map node other than the first map node among all map nodes in the holographic topology map; each fuzzy fingerprint corresponds to one data block. The third processing unit is used to perform end-to-end consistency detection on all the second fingerprint streams based on the first fingerprint stream, and obtain the consistency detection result of the power transmission data stream.

[0015] Thirdly, embodiments of this application also provide an electronic device, including: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0016] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a processor-executable program, which, when executed by the processor, is used to implement the above-described method.

[0017] Fifthly, embodiments of this application also provide a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer program from the computer-readable storage medium and executes the computer program, causing the electronic device to perform the method described above.

[0018] The advantages and beneficial effects of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application: This application discloses a method and system for power transmission data consistency detection. The method acquires power transmission data streams from heterogeneous data links in a power transmission system, and a holographic topology map corresponding to the heterogeneous data links. The power transmission data streams include several data blocks. Based on the holographic topology map, fuzzy fingerprints are generated from the power transmission data streams to obtain a first fingerprint stream and several second fingerprint streams. The first fingerprint stream characterizes a combination of several fuzzy fingerprints of the power transmission data stream at a first map node, and the second fingerprint stream characterizes a combination of several fuzzy fingerprints of the power transmission data stream at a second map node. The first map node is an end map node among all map nodes in the holographic topology map; the second map nodes are map nodes other than the first map node among all map nodes in the holographic topology map; each fuzzy fingerprint corresponds to one data block; based on the first fingerprint stream, end consistency detection is performed on all second fingerprint streams to obtain the consistency detection result of the power transmission data streams. This method acquires a holographic topology map of heterogeneous data links in the power transmission system, which can make the links transparent and facilitate the tracking of the data lineage of power transmission data in the power transmission system. In addition, by generating fuzzy fingerprints of the power transmission data stream, this method can eliminate the accuracy error of heterogeneous data in the power transmission system to a certain extent, which is beneficial to improving the accuracy of subsequent power transmission data consistency detection. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of this application or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions in this application. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0020] Figure 1 A flowchart illustrating a power transmission data consistency detection method provided in an embodiment of this application; Figure 2 A schematic diagram of the framework of a power transmission data consistency detection system provided in this application embodiment; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0023] Currently, relevant technologies typically rely on single-point detection with fixed thresholds to detect data consistency across nodes in a power transmission system. However, due to the existence of data silos across security zones within the power transmission system, and the fact that single-point detection can only perform longitudinal analysis of transmission data at each node, lacking the ability to detect data laterally along the transmission link, this approach cannot determine whether unexpected logical tampering or packet loss has occurred during the data flow from source to end. This makes it difficult to trace the data lineage of transmission data within the system. Furthermore, this method is ill-suited to the accuracy differences of heterogeneous databases and is susceptible to network jitter, resulting in a high false alarm rate under the influence of accuracy differences and transmission delay interference. In other words, the accuracy of power transmission data consistency detection is unsatisfactory.

[0024] Furthermore, in the power transmission data governance scenario, there are some related technologies that locate faults through explicit abnormal alarm events. However, since many data quality issues in the power transmission data governance scenario (such as loss of floating-point calculation precision and ETL logic errors) do not trigger alarms, this method fails when faced with hidden faults such as "network connectivity but data errors". As a result, it is impossible to trace the source at the data content level, making the reliability of power transmission data services unsatisfactory.

[0025] It should be noted that the aforementioned related technologies are only used to assist in understanding the technical solutions of this application and do not mean that they belong to the publicly disclosed prior art.

[0026] In view of this, embodiments of this application provide a method and system for power transmission data consistency detection. The method acquires a holographic topology map of heterogeneous data links in the power transmission system, which enables link transparency and facilitates tracing the data lineage of power transmission data within the system. Furthermore, the method generates fuzzy fingerprints from the power transmission data streams. Specifically, it uses the timestamps of the power transmission data streams and the block quantization features obtained through anti-noise quantization to generate block fingerprints. This ensures that the generated fingerprints are noise-resistant and can, to some extent, eliminate accuracy errors in heterogeneous data within the power transmission system, providing a robust basis for subsequent consistency verification and thus improving the accuracy of subsequent power transmission data consistency detection.

[0027] Furthermore, this method performs end-to-end consistency detection on the second fingerprint stream using the first fingerprint stream. For any data block received by a data link node, specifically, the fingerprint of the current data link node's data block is predicted by comparing the fingerprint of the data block from the previous node (i.e., the first expected fingerprint). Combined with the actually obtained first data fingerprint and the tolerance window, spatiotemporal tolerance and fingerprint analysis are performed. This can overcome transmission delay interference, thereby achieving low-bandwidth, highly robust consistency verification, which is beneficial to improving the accuracy of power transmission data consistency detection.

[0028] Furthermore, when the consistency detection result is abnormal, this method can achieve fault location and automatic attribution by performing fingerprint gradient differential analysis on each candidate fault path set and subsequent joint fault analysis, effectively analyzing hidden faults and improving the reliability of power transmission data services.

[0029] Reference Figure 1 In this embodiment of the application, a method for detecting the consistency of power transmission data includes: Step 110: Obtain the transmission data stream of the heterogeneous data link of the power transmission system, and the holographic topology map corresponding to the heterogeneous data link of the power transmission system; the transmission data stream includes several data blocks; In this embodiment, the heterogeneous data link of the power transmission system can be composed of several entity objects located in multiple security zones of the power system. The security zone refers to the power system security partition. Based on power industry standards, the security partition can usually be divided into production control zones (security zones I and II) and management information zones (security zones III and IV). The entity objects are used to represent the system hardware and software nodes that participate in power transmission data storage, computing processing and network transmission. The entity objects specifically include database entity objects, data processing job entity objects and network channel entity objects. Among them, database entity objects refer to business database systems such as Oracle and Hive, data processing job entity objects refer to ETL tools (such as Informatica and Kettle) or stream computing engines (such as Flink and Spark), and network channel entity objects refer to network devices such as cross-regional physical isolation gateways (GAPs) and routers.

[0030] Understandably, a power transmission data stream can be a collection of power transmission data flowing through heterogeneous data links. This data stream consists of several data blocks, each representing a block of power transmission data within a corresponding time window. Each data block corresponds to a different time window on the time axis. A holographic topology map is used to represent the topology of heterogeneous data links; it can be determined from the heterogeneous data links based on map generation techniques. In some embodiments, acquiring a holographic topology map corresponding to the heterogeneous data links of the power transmission system includes: Obtain the entity data of each entity object in the heterogeneous data link; Entity relationship analysis is performed on all the entity data to obtain relationship analysis information; Based on the relationship analysis information, a graph is generated for all the entity objects to obtain the holographic topological graph.

[0031] In this embodiment, data acquisition probes can be deployed in various security areas of the power transmission system to collect data on different types of entity objects, thereby obtaining entity data for each entity object in each heterogeneous data link. Specifically, if the entity object is a database entity object, entity data such as table name, field list, field type, and primary / foreign key constraints can be obtained by accessing the database system tables or metadata repository; or, if the entity object is a data processing job entity object, the task configuration file (XML or JSON format) of the ETL tool (such as Informatica, Kettle) and the submission script of the stream computing engine (Flink / Spark) can be read through a file parser; or, if the entity object is a network channel entity object, the entity data can be the mapping rule file of the cross-regional network gateway and the access control list (ACL) of the router.

[0032] Understandably, entity relationship analysis can determine the relationships between entity objects from entity data. For example, it can parse entity data from data processing job entity objects to obtain relationship analysis information. Specifically, for task configuration files in XML / JSON format, standard DOM or JSON parsing libraries can be used, based on preset node tags (such as...). <source> , <target>Alternatively, for key-value pairs, the corresponding table name information can be directly extracted from the task configuration file to obtain the input source table and output target table of the data processing job. The input source table records the upstream entity objects of the data processing job entity object, while the output target table records the downstream entity objects of the data processing job entity object. Or, for SQL scripts of stream processing engines such as Flink / Spark, the input source table and output target table can be obtained by regular expression matching or by constructing an abstract syntax tree (AST) using an SQL parser, extracting the identifiers after keywords such as FROM, INSERT, and INTO.

[0033] For network channel entity objects, IP address mapping relationships and port open status can be easily obtained from rule mapping files and access control lists, thus determining the link relationships between the network channel entity object and other entity objects. The content of database entity objects is similar to that of the aforementioned data processing job entity objects and can be easily deduced. After obtaining the link relationships between each entity object and other entity objects, simple integration is sufficient to determine the relationship analysis information.

[0034] Graph generation can be based on graph generation technology, mapping each entity object to a graph node, and establishing connection edges between graph nodes based on the link relationships between each entity object in the relationship analysis information, thereby obtaining a holographic topological graph.

[0035] Step 120: Based on the holographic topology map, generate fuzzy fingerprints for the power transmission data stream to obtain a first fingerprint stream and several second fingerprint streams; the first fingerprint stream is used to characterize the combination of several fuzzy fingerprints of the power transmission data stream at the first map node, and the second fingerprint stream is used to characterize the combination of several fuzzy fingerprints of the power transmission data stream at the second map node; the first map node is the end map node among all map nodes of the holographic topology map; the second map node is the map node other than the first map node among all map nodes of the holographic topology map; each fuzzy fingerprint corresponds to one data block; In this embodiment, a combination of fuzzy fingerprints (i.e., fingerprint streams) of the power transmission data stream at each node it flows through can be generated based on the holographic topology map. For ease of understanding, this embodiment will subsequently use a holographic topology map including cascaded nodes A, B, C, and D as an example, where node D is the terminal node (i.e., the first node) of the holographic topology map, and the fingerprint stream of the power transmission data stream at node D is the first fingerprint stream; while nodes A, B, and C are each a second node, and the fingerprint stream of each second node is the second fingerprint stream.

[0036] It is understood that the examples of holographic topological spectra in this application are merely simple examples and are not intended to limit the scope of holographic topological spectra. The number of spectra nodes included in the holographic topological spectra, as well as the relationship structure between spectra nodes (such as cascade relationship, parallel relationship, etc.), can be obtained according to the actual situation.

[0037] In some embodiments, the step of generating a fuzzy fingerprint of the power transmission data stream based on the holographic topology map to obtain a first fingerprint stream and several second fingerprint streams includes: Obtain several node blocks, which are used to represent data blocks in the power transmission data stream of the third graph node; In practical applications, the power transmission data stream sequentially travels from graph node A to graph nodes B, C, and D. Therefore, fuzzy fingerprint generation can be performed on the power transmission data stream at each graph node to obtain the second fingerprint streams for graph nodes A, B, and C, and the first fingerprint stream for graph node D. Specifically, any graph node in the holographic topology graph can be designated as the third graph node, and each data block in the power transmission data stream at the third graph node can be defined as a node block.

[0038] Based on the power transmission data stream, fingerprint synthesis is performed on all the node blocks to obtain a fuzzy fingerprint for each node block; Further, the step of performing fingerprint synthesis on all the node blocks based on the power transmission data stream to obtain a fuzzy fingerprint for each node block includes: Statistical features are extracted from the node blocks to obtain block feature vectors; The block feature vector is subjected to noise-resistant quantization to obtain block quantized features; Based on the timestamp of the power transmission data stream, a block fingerprint is generated from the block quantization features to obtain the fuzzy fingerprint of the node block.

[0039] In this embodiment of the application, for any node block among all node blocks, statistical feature extraction can be performed by extracting statistical features in the node block that are insensitive to numerical drift. The feature vector (i.e., the block feature vector) of this statistical feature can be represented as:

[0040] in, For block feature vectors; The number of data rows in the node block; The sum of the values ​​of K key measurement fields. This is the kth key metering field, which can be voltage, active power, etc. in the node block; Histogram features representing the numerical distribution of node block data are used to capture the macroscopic form of the data content.

[0041] It is understandable that, compared to the traditional MD5 full hash of data blocks, the embodiments of this application extract statistical features of node blocks and utilize the insensitivity of features in the block feature vector to small numerical drifts (such as floating-point precision differences). This allows the system to tolerate precision loss between heterogeneous data sources while maintaining the overall fingerprint consistency of the data content. This heterogeneous data source can be used to characterize different types of entity objects in heterogeneous data links.

[0042] Noise-resistant quantization can be achieved by fuzzing the numerical elements (i.e., key measurement fields) in a block feature vector. Specifically, this is done by introducing a dynamic quantization function that maps continuous floating-point values ​​to discrete integer buckets, thus obtaining block quantized features. This noise-resistant quantization can eliminate minute precision errors in the block feature vector within the quantization buckets. The dynamic quantization function can be expressed as:

[0043] in, For dynamic quantization functions; The original statistical values ​​of the numerical elements of the block feature vector, such as the sum of fields; This is a preset tolerance accuracy threshold, which can be any value such as 0.01, 0.001, etc.

[0044] It should be noted that power transmission data experiences nondeterministic delays during cross-regional network transmission via heterogeneous data links, which can easily lead to misalignment between the actual stamps at the source and destination ends. Block fingerprint generation can begin by obtaining the time window index of the node block, which can be represented as:

[0045] in, Index for time windows; The event timestamps are inherent to the power transmission data stream; The timestamp of the node block can be either the timestamp of the power transmission data stream arriving at the third graph node, or the timestamp of the third graph node processing the node block. The sliding width of the time window can be 3 seconds, 5 seconds, etc.

[0046] Subsequently, combining the time window index and block quantization features, an irreversible hash algorithm is used to generate a unique fuzzy fingerprint for the node block. This fuzzy fingerprint is identical between the source and destination nodes within an allowable range of precision loss and minor time drift, providing a robust basis for subsequent consistency verification. Specifically, the source node can be a graph node sending transmission data, while the destination node is a graph node receiving transmission data. For example, if the transmission data stream flows from graph node A to graph node B, then graph node A can be the source node, and graph node B can be the destination node; or, if the transmission data stream flows from graph node C to graph node D, then graph node C can be the source node, and graph node D can be the destination node. This unique fuzzy fingerprint can be represented as:

[0047] in, A fuzzy fingerprint of a node block; This is a functional representation of an irreversible hash algorithm; This serves as a globally unique identifier for the third graph node within the heterogeneous data link of the power transmission system. The symbolic representation for the concat operation.

[0048] The third fingerprint stream is obtained based on the fuzzy fingerprints of all the node blocks. If the third graph node is the first graph node, then the third fingerprint stream is the first fingerprint stream; or, if the third graph node is the second graph node, then the third fingerprint stream is the second fingerprint stream.

[0049] In this embodiment, the acquisition of fuzzy fingerprints for the remaining data blocks of the third graph node is similar to the acquisition of fuzzy fingerprints for the aforementioned node blocks of the third graph node, and can be easily deduced by analogy. After obtaining the fuzzy fingerprints of each data block of the power transmission data stream in the third graph node, the corresponding third fingerprint stream can be easily integrated.

[0050] Step 130: Based on the first fingerprint stream, perform end consistency detection on all second fingerprint streams to obtain the consistency detection result of the power transmission data stream.

[0051] In this embodiment of the application, when the power transmission data stream reaches the terminal map node (i.e. the first map node), the power transmission system can perform consistency detection based on the first fingerprint stream of the power transmission data stream at the first map node and the first fingerprint stream of the power transmission data stream at each second map node to obtain the consistency detection result of the power transmission data stream.

[0052] It is understandable that in the power transmission data governance scenario, traditional power transmission data consistency detection usually relies on full detection of the power transmission data itself. However, due to the large scale of power transmission data in the power transmission system, this method requires high transmission bandwidth and high computing resources for power transmission data consistency detection. In contrast, the embodiments of this application perform consistency detection on the fingerprint stream of power transmission data, which can reduce the transmission bandwidth required for power transmission data consistency detection and has low computing resource requirements.

[0053] In some embodiments, the step of performing end-to-end consistency detection on all second fingerprint streams based on the first fingerprint stream to obtain the consistency detection result of the power transmission data stream includes: Acquire a tolerance window, a first data fingerprint, and several second data fingerprints; the first data fingerprint is any one of the fuzzy fingerprints in the first fingerprint stream; the second data fingerprint is the fuzzy fingerprint in the second fingerprint stream that corresponds to the first data fingerprint. Fingerprint prediction analysis is performed on all the second data fingerprints to obtain the first expected fingerprint; Based on the tolerance window and the first data fingerprint, spatiotemporal tolerance and fingerprint difference detection are performed on the first expected fingerprint to obtain the consistency detection result.

[0054] In this embodiment, for any data block in the first graph node of the power transmission data stream, the power transmission system can obtain the fuzzy fingerprint of that data block, denoted as the first fuzzy fingerprint; simultaneously, the power transmission system obtains the fuzzy fingerprints of that data block in the remaining second graph nodes, denoted as the second data fingerprints. Fingerprint prediction analysis can be based on all the second data fingerprints to determine the first expected fingerprint.

[0055] Specifically, in the first implementation, if the transmission path of the power transmission data stream containing the data block in the heterogeneous data link is transparent (i.e., there is no aggregation calculation or change in calculation logic during transmission), then under normal data transmission conditions, the fuzzy fingerprint of the data block is the same at each graph node. The first expected fingerprint under the first graph node can be the fuzzy fingerprint of its source end, specifically, it can be any one of the second data fingerprints. This first expected fingerprint can be represented as:

[0056] in, This is the first expected fingerprint; The source fingerprint is an fuzzy fingerprint.

[0057] Alternatively, in the second embodiment, if the power transmission data stream containing the data block has aggregated or calculated graph nodes in the transmission path of the heterogeneous data link, these nodes are denoted as processing nodes. For ease of understanding, this application embodiment takes an example where the number of processing nodes is equal to 1. Then, the first expected fingerprint can be represented as:

[0058] in, This is the fingerprint transfer function for the corresponding processing node. This fingerprint transfer function can be easily determined based on the actual processing logic of the processing node (such as summation, averaging, field mapping, etc.).

[0059] The first expected fingerprint when the number of processing nodes is greater than or equal to 2 is similar to the first expected fingerprint when the number of processing nodes is equal to 1, and can be easily deduced by analogy. This application will not elaborate further here.

[0060] In practical applications, cross-regional transmission of power transmission data in heterogeneous data links within a power transmission system often involves random network delays. This can cause data from the source at second T to arrive at the destination at second T+β, leading to false alarms when directly performing consistency checks. Therefore, a dynamic tolerance window mechanism can be used to obtain a tolerance window. This tolerance window allows the power transmission system to avoid strictly aligning the time windows of the data streams at the source and destination, instead performing sliding matching within a preset time offset range [-g, g]. The matching logic of the power transmission system is as follows: if a time offset j exists within the tolerance window such that the fingerprint difference is less than a preset threshold, the verification passes, and the system returns to the steps of obtaining the tolerance window, the first data fingerprint, and several second data fingerprints. A consistency check result indicating normal consistency is generated when all data blocks pass the verification; otherwise, the verification fails, and a consistency check result indicating abnormality is generated. The functional representation of the matching logic is as follows:

[0061] in, This is a functional representation of the consistency check results; Characterization verification passed. Characterization verification failed; This represents the maximum tolerable number of sliding windows, and its value can be any one of 1, 2, etc. The actual measurement fuzzy fingerprint (i.e., the first data fingerprint) of the power transmission data stream under the time window index corresponding to the first map node (i.e. the end map node); This is the allowable consistency error threshold for the power transmission system. The value can be set according to actual needs, for example, it can be dynamically adjusted within the range of [0.001, 0.1]. This is a difference measurement function.

[0062] The difference measurement function can be:

[0063] in, For difference measurement function Another functional representation of ; This represents the number of data rows in the first data fingerprint. The number of data rows for the first expected fingerprint; and These are the noise-resistant quantized numerical feature vectors from the first data fingerprint and the first expected fingerprint, respectively. These noise-resistant quantized numerical feature vectors are the aforementioned noise-resistant quantized... .

[0064] In some embodiments, if the consistency detection result is an anomaly, the method further includes: Obtain abnormal blocks, wherein the abnormal blocks are data blocks that are abnormal in the power transmission data stream of the first graph node; Based on the abnormal block, upstream related paths are extracted from the holographic topology map to obtain a candidate fault path set; the candidate fault path set includes several path nodes; Perform hop-by-hop fingerprint gradient difference analysis on the candidate fault path set to obtain the consistency gradient of each path node; Fault location is performed on all the aforementioned consistency gradients to obtain fault objects, which are used to characterize fault map nodes and / or fault edges in the holographic topology map.

[0065] In this embodiment, if the consistency check result is abnormal, it indicates that the transmission data flow in the heterogeneous data link of the transmission system is abnormal. In this case, the data block that failed the check can be identified as an abnormal block. The upstream associated path extraction can be performed by utilizing the directed connection features of the holographic topology map. Starting from the first map node, a reverse depth-first search algorithm (Reverse DFS) is executed in the holographic topology map. By traversing only the upstream map nodes associated with the abnormal data block and identifying each associated map node as a path node, a candidate fault path set containing several path nodes is constructed.

[0066] It is understandable that hop-by-hop fingerprint gradient differential analysis can be performed by determining the consistency gradient of each path node along the candidate fault path. The consistency gradient can be obtained by the following formula:

[0067] in, For a consistent gradient; This is a similarity function, taking values ​​of 0 or 1, where 0 indicates an anomaly and 1 indicates normality. This is a fuzzy fingerprint of the anomalous block at the upstream node; The fuzzy fingerprint of the anomalous block at the downstream node; and These are the expected fingerprints corresponding to the abnormal block in the upstream and downstream nodes, respectively. The method of obtaining these expected fingerprints is similar to that of obtaining the first expected fingerprint mentioned above, and can be easily deduced by analogy.

[0068] It should be noted that fault localization can be achieved by analyzing the specific values ​​of the consistency gradient of each path node to determine the faulty object. Specifically, in the first implementation, if the candidate faulty path set includes path node E, path node F, and path node G in sequence, where path node E is a downstream node of path node F and path node G is an upstream node of path node F; for path node F, if the fingerprint of the upstream node is normal (Sim=1), while the fingerprint of the immediately adjacent downstream node is abnormal (Sim=0), then the gradient... At this point, the fault can be located to the connection edge between path node G and path node F, which is recorded as the first fault edge. Path node F is recorded as the fault graph node, and the connection edge between path node F and path node E is recorded as the second fault edge. Based on all the obtained fault edges and fault graph nodes, the fault object can be determined.

[0069] In the second implementation, for path node F, path node F can be determined as a downstream node, and path node G as an upstream node, to obtain the consistency gradient of path node F. Specifically, if the fingerprint of the upstream node is normal (Sim=1), while the fingerprint of the immediately adjacent downstream node is abnormal (Sim=0), then the gradient... At this point, the fault can be located to the connecting edge between path node G and path node F, which is recorded as the fault edge, and the fault edge is identified as the fault object.

[0070] In practical applications, consistency gradient analysis can be performed using either of the two implementation methods described above, or a combination of both. Furthermore, if the fingerprint of the upstream node (e.g., path node G) is still abnormal (Sim=0) when tracing back to the upstream node, the fault can be identified as a problem with the quality of the external input source data.

[0071] In some embodiments, the method further includes: Obtain the environmental context characteristics of the faulty object and several upstream nodes; Based on the fuzzy fingerprints of all the upstream nodes, a second expected fingerprint corresponding to the faulty object is obtained; Based on the fuzzy fingerprint of the faulty object, perform fingerprint difference analysis on the second expected fingerprint to obtain fingerprint difference features; Based on the environmental context features and the fingerprint difference features, a joint fault analysis is performed to obtain the fault analysis results of the faulty object.

[0072] In this embodiment, the environmental context features can be obtained from the power transmission system operation and maintenance monitoring center (such as Prometheus or Zabbix) under the abnormal block time window index, including at least one of the following: the computing resource load rate of the faulty object, the network packet loss rate and bit error rate of the faulty object, the number of Error-level log entries of the process associated with the faulty object in the corresponding time period, and the backlog depth of the message queue of the faulty object. The method for obtaining the second expected fingerprint is similar to that for obtaining the first expected fingerprint, and can be easily deduced by analogy. The fingerprint difference features are similar to the aforementioned difference measurement function, and can be easily deduced by analogy.

[0073] It is understandable that fingerprint difference analysis can involve constructing a fault classification decision tree or rule inference engine, jointly analyzing fingerprint difference features with environmental context features to obtain fault analysis results. The fault type discrimination logic within this fault classification decision tree or rule inference engine may include: 1) Type I: Network Fault Judgment Logic: The fingerprint exhibits "complete loss of fingerprint entries," and environmental indicators show that the network packet loss rate and bit error rate exceed a preset threshold. This preset threshold can be any one of 0.2, 0.5, 0.8, or 0.9. This "complete loss of fingerprint entries" is used to characterize the faulty object. (approaching or equal to 0) Fault analysis results: Physical link interruption or packet loss by the network gateway.

[0074] 2) Type II: Logic Fault Criterion: The fingerprint exhibits "matching number of fingerprints but failed numerical verification" and the number of error-level log entries is greater than 0 (a code exception stack trace exists). This "matching number of fingerprints but failed numerical verification" is used to characterize the fingerprint's discrepancies. Same but A mutation occurred.

[0075] Fault analysis results: ETL script logic error, data type conversion overflow, or code bug.

[0076] 3) Type III: Performance Fault Criteria: The fingerprint exhibits a "severely delayed time window" and a persistently high queue depth, or the computational resource load rate exceeds a preset threshold. This preset threshold can be any one of 0.8, 0.9, 0.95, etc. The "severely delayed time window" indicates that the aforementioned matching offset j has reached its maximum tolerance value (i.e., the preset time offset range). The specific condition for the persistently high queue depth can be that the amount of data contained in the faulty object's message queue exceeds a preset data amount. The specific value of this preset data amount can be flexibly set according to actual conditions; for example, it could be a maximum message count of 10,000 or a total byte limit of 1GB for the message queue.

[0077] Fault analysis results: Data backlog and delay caused by insufficient system resources.

[0078] It should be noted that the examples of fault type discrimination logic in the embodiments of this application are only for the purpose of understanding and are not exhaustive, nor are they intended to limit the specific discrimination logic.

[0079] Reference Figure 2 The power transmission data consistency detection system proposed in this application includes: The first processing unit 101 is used to acquire the power transmission data stream of the heterogeneous data link of the power transmission system, and the holographic topology map corresponding to the heterogeneous data link of the power transmission system; the power transmission data stream includes several data blocks; The second processing unit 102 is configured to generate fuzzy fingerprints for the power transmission data stream based on the holographic topology map, resulting in a first fingerprint stream and several second fingerprint streams. The first fingerprint stream characterizes a combination of several fuzzy fingerprints of the power transmission data stream at a first map node, and the second fingerprint stream characterizes a combination of several fuzzy fingerprints of the power transmission data stream at a second map node. The first map node is the end map node among all map nodes in the holographic topology map; the second map node is the map node other than the first map node among all map nodes in the holographic topology map; each fuzzy fingerprint corresponds to one data block. The third processing unit 103 is used to perform end consistency detection on all the second fingerprint streams based on the first fingerprint stream to obtain the consistency detection result of the power transmission data stream.

[0080] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0081] Reference Figure 3 This application also provides an electronic device, including: At least one processor 201; At least one memory 202 is used to store at least one program; When the at least one program is executed by the at least one processor 201, the at least one processor 201 implements the method embodiment described above.

[0082] Similarly, it can be understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0083] This application also provides a computer-readable storage medium storing a program executable by a processor 201, which, when executed by the processor 201, is used to implement the above-described method embodiments.

[0084] Similarly, the content of the above method embodiments is applicable to the present computer-readable storage medium embodiments. The specific functions implemented by the present computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0085] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0086] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods.

[0087] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this application are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0088] Furthermore, although this application is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding this application. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional technology for an engineer. Therefore, those skilled in the art can implement the application set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of this application, which is determined by the full scope of the appended claims and their equivalents.

[0089] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods in the embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0090] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0091] More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0092] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0093] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0094] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

[0095] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.< / target>

Claims

1. A method for detecting power delivery data consistency, the method comprising: include: The transmission data stream of the heterogeneous data link of the power transmission system and the holographic topology map corresponding to the heterogeneous data link of the power transmission system are obtained; the transmission data stream includes several data blocks; Based on the holographic topology map, fuzzy fingerprints are generated for the power transmission data stream to obtain a first fingerprint stream and several second fingerprint streams. The first fingerprint stream is used to characterize the combination of several fuzzy fingerprints of the power transmission data stream at a first map node, and the second fingerprint stream is used to characterize the combination of several fuzzy fingerprints of the power transmission data stream at a second map node. The first map node is the end map node among all map nodes of the holographic topology map; the second map node is the map node other than the first map node among all map nodes of the holographic topology map; each fuzzy fingerprint corresponds to one data block. Based on the first fingerprint stream, end-to-end consistency detection is performed on all second fingerprint streams to obtain the consistency detection result of the power transmission data stream.

2. The method according to claim 1, characterized in that, Obtaining a holographic topology map corresponding to the heterogeneous data links of the power transmission system includes: Obtain the entity data of each entity object in the heterogeneous data link; Entity relationship analysis is performed on all the entity data to obtain relationship analysis information; Based on the relationship analysis information, a graph is generated for all the entity objects to obtain the holographic topological graph.

3. The method according to claim 1, characterized in that, The step of generating a fuzzy fingerprint of the power transmission data stream based on the holographic topology map to obtain a first fingerprint stream and several second fingerprint streams includes: Obtain several node blocks, which are used to represent data blocks in the power transmission data stream of the third graph node; Based on the power transmission data stream, fingerprint synthesis is performed on all the node blocks to obtain a fuzzy fingerprint for each node block; The third fingerprint stream is obtained based on the fuzzy fingerprints of all the node blocks. If the third graph node is the first graph node, then the third fingerprint stream is the first fingerprint stream; or, if the third graph node is the second graph node, then the third fingerprint stream is the second fingerprint stream.

4. The method according to claim 3, characterized in that, The step of performing fingerprint synthesis on all the node blocks based on the power transmission data stream to obtain a fuzzy fingerprint for each node block includes: Statistical features are extracted from the node blocks to obtain block feature vectors; The block feature vector is subjected to noise-resistant quantization to obtain block quantized features; Based on the timestamp of the power transmission data stream, a block fingerprint is generated from the block quantization features to obtain the fuzzy fingerprint of the node block.

5. The method according to claim 1, characterized in that, The step of performing end-to-end consistency detection on all second fingerprint streams based on the first fingerprint stream to obtain the consistency detection result of the power transmission data stream includes: Acquire a tolerance window, a first data fingerprint, and several second data fingerprints; the first data fingerprint is any one of the fuzzy fingerprints in the first fingerprint stream; the second data fingerprint is the fuzzy fingerprint in the second fingerprint stream that corresponds to the first data fingerprint. Fingerprint prediction analysis is performed on all the second data fingerprints to obtain the first expected fingerprint; Based on the tolerance window and the first data fingerprint, spatiotemporal tolerance and fingerprint difference detection are performed on the first expected fingerprint to obtain the consistency detection result.

6. The method according to any one of claims 1-5, characterized in that, If the consistency detection result is an anomaly, the method further includes: Obtain abnormal blocks, wherein the abnormal blocks are data blocks that are abnormal in the power transmission data stream of the first graph node; Based on the abnormal block, upstream related paths are extracted from the holographic topology map to obtain a candidate fault path set; the candidate fault path set includes several path nodes; Perform hop-by-hop fingerprint gradient difference analysis on the candidate fault path set to obtain the consistency gradient of each path node; Fault location is performed on all the aforementioned consistency gradients to obtain fault objects, which are used to characterize fault map nodes and / or fault edges in the holographic topology map.

7. The method according to claim 6, characterized in that, The method further includes: Obtain the environmental context characteristics of the faulty object and several upstream nodes; Based on the fuzzy fingerprints of all the upstream nodes, a second expected fingerprint corresponding to the faulty object is obtained; Based on the fuzzy fingerprint of the faulty object, perform fingerprint difference analysis on the second expected fingerprint to obtain fingerprint difference features; Based on the environmental context features and the fingerprint difference features, a joint fault analysis is performed to obtain the fault analysis results of the faulty object.

8. A power transmission data consistency detection system, characterized in that, include: The first processing unit is used to acquire the transmission data stream of the heterogeneous data link of the power transmission system, and the holographic topology map corresponding to the heterogeneous data link of the power transmission system; the transmission data stream includes several data blocks; The second processing unit is configured to generate fuzzy fingerprints for the power transmission data stream based on the holographic topology map, resulting in a first fingerprint stream and several second fingerprint streams. The first fingerprint stream characterizes a combination of several fuzzy fingerprints of the power transmission data stream at a first map node, and the second fingerprint stream characterizes a combination of several fuzzy fingerprints of the power transmission data stream at a second map node. The first map node is the end map node among all map nodes in the holographic topology map; the second map node is the map node other than the first map node among all map nodes in the holographic topology map; each fuzzy fingerprint corresponds to one data block. The third processing unit is used to perform end-to-end consistency detection on all the second fingerprint streams based on the first fingerprint stream, and obtain the consistency detection result of the power transmission data stream.

9. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The processor stores a program that is executable by a processor, which, when executed by the processor, is used to implement the method as described in any one of claims 1-7.

Images