Batch processing method for service data of power distribution Internet of Things

Abstract

The invention belongs to the technical field of power distribution network service management methods, and particularly relates to a power distribution Internet of Things service data batch processing method. Comprising the following steps: establishing definition and description rules of business data and attributes thereof, and configuring data formats of different types of business data; carrying out multi-source data matching based on an attribute similarity principle, and combining or removing source data of which the similarity exceeds a preset threshold value; and the data storage unit stores various types of business data in batches based on the business data structure types. According to the power distribution Internet of Things service data batch processing method, the similarity matching scheme based on the power distribution network service data source data and the distributed classification storage method, a power distribution network service data management scheme can be established better and more efficiently, the classification storage efficiency of different service data is improved, efficient analysis of redundant data is realized, and the service data quality is improved. And a technical basis is provided for establishing power distribution network business data batch management and maintenance.

Description

Technical Field

[0001] This invention belongs to the technical field of power distribution network business management methods, and particularly relates to a method for batch processing of power distribution Internet of Things (IoT) business data. Background Technology

[0002] The distribution network has a wide variety of business types, generating a massive amount of business data. This poses a significant challenge to the batch storage, management, and analysis of this data. On the one hand, the source data involved in different distribution network business data are often collected through different devices or systems. However, there is a large amount of identical or consistent data among these source data, leading to the redundant consumption of data storage and management resources. On the other hand, different process stages of some distribution network businesses involve a large number of calls to the same type of data. An unreasonable storage management method will lead to the rapid consumption of system hardware and software resources during the processing. The existence of the above problems poses a challenge to the increasingly large business data management operations of the distribution network. Summary of the Invention

[0003] The purpose of this invention is to propose a batch processing method for distribution network business data based on actual needs and taking into account the characteristics of distribution network business data. This method aims to improve the efficiency of batch management, storage, and analysis of distribution network business data, and enhance the ability of distribution network data management and maintenance systems to handle large amounts of business data.

[0004] To achieve the above objectives, the present invention adopts the following technical solution.

[0005] A method for batch processing of distribution Internet of Things (IoT) business data includes:

[0006] A1. Establish rules for defining and describing business data and its attributes, and configure data formats for different types of business data;

[0007] A2. Based on the principle of attribute similarity, perform multi-source data matching, and merge or remove source data whose similarity exceeds a preset threshold;

[0008] A3. The data storage unit performs batch storage of various types of business data based on the business data structure type. The data storage unit uses different data formats to store different types of data separately. The data storage data linked list establishes a business data chain for locating and tracking source data based on the relationship attributes between source data within the data storage unit, and stores it in a linked list manner.

[0009] In a further improved or preferred embodiment of the aforementioned batch processing method for distribution IoT business data, the different types of business data in A1 include: source data, basic business information data, and business synchronization information data.

[0010] A further improvement or preferred embodiment of the aforementioned batch processing method for distribution network IoT business data includes the following source data: distribution network equipment or system measurement business data, distribution network topology and network structure attribute data, and equipment or structure parameter attribute data; the data format refers to data formats such as XML, DAT, SI, DB, BPA, and SWI; among which, XML and DB are mainly used to store large batches of measurement business data involved in the distribution network business process; DAT and SWI are mainly used to store parameter data such as the attribute status of equipment or systems involved in the distribution network business process; SI and BPA are mainly used to store distribution network topology and power flow analysis data.

[0011] A further improvement or preferred embodiment of the aforementioned method for batch processing of distribution IoT business data, wherein step A2 specifically includes the following steps:

[0012] 2a. Based on the distribution network business, establish attribute IDs to describe the basic attributes of metadata. Metadata refers to the basic business data units or business data unit groups used to constitute source data. Unify the attribute ID format and naming method, establish an attribute ID table, and for each source data, extract the basic attribute IDs in sequence according to the generation process of metadata or metadata groups to generate its attribute ID chain. The order of metadata groups does not affect the ID of the metadata group.

[0013] 2b. Classify the source data according to the type of distribution network business involved, and generate source databases for serving different distribution network businesses. For common source data D involving multiple distribution network businesses, add it to each source database respectively.

[0014] 2c. For any two source data in each source database, perform attribute ID chain similarity matching. Based on a preset similarity threshold, determine whether the two source data are equivalent. If so, merge them into a single source data. Simultaneously, record the source data corresponding to all shared source data D after matching. Repeat the aforementioned matching process until all source data in each source database are distinct.

[0015] 2d. Locate the source databases that share the same source data, extract the shared source data or the source data generated by merging and unifying them, and merge them to generate merged shared source data. Replace the original shared source data with the merged shared source data. In each source database, perform similarity matching between the merged shared source data and the remaining source data in turn. If they are equal, remove the business data from the remaining source data. If they are not equal, retain it.

[0016] A further improvement or preferred embodiment of the aforementioned batch processing method for distribution IoT business data, wherein the similarity matching of the attribute ID chain specifically includes:

[0017] Structural similarity matching based on the composition and generation order of source data attribute IDs

[0018] Since the composition and generation order of attribute IDs are in the form of a linked list, the ease of conversion between linked lists can be used to determine their suitability for two source data. Attribute ID chain Assuming It contains M attributes, ID, It contains M attribute IDs, and analyzes the attribute ID chain by editing the attribute ID position and quantity. Convert to attribute ID chain Difficulty; Definition The first i attributes ID and The conversion difficulty of the first j attribute IDs is The attribute ID chain is compared in a fixed order (either from front to back or from back to front). If they differ, they are transformed sequentially to ensure that the attribute ID at that position remains consistent. The update function for the transformation difficulty during each transformation is:

[0019]

[0020] When the conversion begins, , The conversion difficulty is obtained by sequentially converting and calculating. Then for any two source data in any source database, The similarity based on the composition and generation order of the source data attribute IDs can be represented as: ;

[0021] Local similarity matching based on shared metadata linked lists in the source data

[0022] The shared metadata linked list in the source data represents the local similarity of the source data. In the attribute ID chain, this is represented by the length of the same local chain. The longer the length of the same local chain, the higher the local similarity. For two source data... Attribute ID chain , Assuming The first m characters and The length of the common local chain of the first n characters is ,but The local similarity is calculated sequentially in a fixed order (from front to back or from back to front), and the update function for this function is:

[0023] ;

[0024] After updating sequentially, the lengths of local chains with the same source data are obtained. Based on two source data The local similarity of the metadata linked list in the CPC can be represented as: ,in This refers to the length of the attribute ID chain.

[0025] Weights were assigned to structural similarity and local similarity analysis respectively. , , , Based on the structural similarity and local similarity determined in steps 4a and 4b, two source data are identified. Overall similarity .

[0026] Its beneficial effects are as follows:

[0027] The batch processing method for distribution network Internet of Things (IoT) business data in this application, based on a similarity matching scheme for distribution network business data sources and a distributed classification storage method, can better and more efficiently establish a distribution network business data management scheme, improve the efficiency of classified storage of different business data, and realize efficient analysis of redundant data, thus providing a technical foundation for establishing batch management and maintenance of distribution network business data. Attached Figure Description

[0028] Figure 1 It contains heterogeneous source data from some power distribution network business processes. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0030] This application relates to a batch processing method for distribution network Internet of Things (IoT) business data, mainly used to solve the problem that a large amount of redundant data or essentially the same source data is entered during the storage, management and maintenance of large-scale business data in the distribution network, resulting in wasted storage and processing resources. The method mainly includes the following steps:

[0031] A1. Establish rules for defining and describing business data and its attributes, and configure the data format and relational attributes for different types of business data;

[0032] Based on the characteristics of distribution network business data, the different types of business data in this application include: source data, basic business information data, and business synchronization information data. Source data mainly consists of the basic data used to complete various distribution network businesses, such as the attribute parameters and operating status parameters of basic equipment or systems in the distribution network. This primarily includes: distribution network equipment or system measurement business data, distribution network topology and network structure attribute data, and equipment or structure parameter attribute data. Basic business information data refers to the standardized label data used to define and standardize distribution network business attributes, including: distribution network business ID, code, business process code, etc. Business synchronization information data, based on the basic business information, is used to connect and associate a series of distribution network operation tasks and supporting associated data for cloud-edge business collaboration.

[0033] Distribution network business data often involves multiple types of management and maintenance systems within the distribution network, including power supply and distribution management systems, energy management systems, etc. In order to achieve batch management and maintenance of business data, it is necessary to establish corresponding data formats based on the inherent attributes of different types of data, and to establish relational attribute charts based on the correlation attributes between data.

[0034] In fact, based on the current data management configuration standards in various management systems, various types of business data have one or more relatively fixed format specifications. Their basic formats are mainly divided into tables, graphs, text, and specific formats. In order to improve the efficiency of batch data processing, the configuration of several data formats mentioned in this application mainly refers to standardizing the data that is easy to be digitally expressed, in addition to the solidified models and graphic data, and using a data management form that matches the storage management library for standardized processing.

[0035] A relational attribute diagram is a data diagram composed of definitional data files of business and source data object relational attributes, such as business data association tables, data association directories, business process sequence tables, and business data association directed graphs within the distribution network business management system.

[0036] For example, data such as voltage and current are expressed using the same units and mathematical methods. For binary variables, dual states such as on-off, positive-negative, up-down, and AB in the source data are expressed using 0-1 and other methods, and corresponding data format specifications are established based on this. Another example is the distribution network topology diagram. Since the distribution network equipment and nodes in the topology structure can be located by their corresponding IDs, and the coordinates of equipment and nodes can be expressed by numbers in the Earth system coordinate system, the IDs can be used in conjunction with their structure vector table for processing. In specific implementation, targeted designs are made according to the distribution network business type and its requirements. There are already relatively mature data standardization processing solutions and examples in this regard, so they will not be elaborated further.

[0037] A2. Perform multi-source data matching based on attribute similarity principles.

[0038] like Figure 1 As shown, because the distribution network business process can obtain source data from different systems or devices, there is a large amount of multi-source data for the same business object in the distribution network source data. This multi-source data contains a large number of differentiated descriptions of the same business attribute. In order to ensure the consistency of the source data and reduce the burden of business data processing,

[0039] Specifically, it refers to:

[0040] 2a. Based on the distribution network business, establish attribute IDs to describe the basic attributes of metadata in the business data. Metadata refers to the basic business data units or business data unit groups used to constitute the source data. Unify the attribute ID format and naming method, establish an attribute ID table, and for each source data, generate the attribute ID chain of the source data by extracting the basic attribute IDs in sequence according to the generation process of metadata or metadata groups.

[0041] For example, for source data ABC consisting of metadata A, B, and C, arranging and extracting attribute IDs according to the order in which the metadata or metadata groups were generated, assuming the generation order is A→B→C, and A+B, B+C, and A+C are all metadata groups with defined attribute IDs, then the attribute ID chain of source data ABC can be represented as follows: ;in The attribute ID is presented in a uniform format, with the subscript indicating the specific attribute ID value. The order of metadata groups does not affect the ID of the metadata group, i.e., metadata group A + C = metadata group C + A.

[0042] 2b. Classify the source data according to the type of distribution network business involved, and generate source databases for serving different distribution network businesses. For common source data D involving multiple distribution network businesses, add it to each source database respectively.

[0043] 2c. For any two source data in each source database, perform attribute ID chain similarity matching. Based on a preset similarity threshold, determine whether the two source data are equivalent. If so, merge them into a single source data. Simultaneously, record the source data corresponding to all shared source data D after matching. Repeat the aforementioned matching process until all source data in each source database are distinct.

[0044] Typically, the similarity of attribute ID chains between two source data points involves the composition and arrangement of attribute IDs. The composition of attribute IDs represents the similarity of source data components, indicating the degree of proximity in the source data's origin and business attributes. The similarity of the attribute ID chain arrangement represents the similarity of the source data's generation path in the power distribution network business process, indicating the degree to which the source data is shared within the business process. Therefore, this application performs attribute ID chain similarity matching based on the following method, specifically including two parts of matching:

[0045] Firstly, structural similarity matching based on the composition and generation order of source data attribute IDs is out of the question.

[0046] Since the composition and generation order of attribute IDs are in the form of a linked list, the ease of conversion between linked lists can be used to determine their suitability for two source data. Attribute ID chain Assuming It contains M attributes, ID, It contains M attribute IDs, and analyzes the attribute ID chain by editing the attribute ID position and quantity. Convert to attribute ID chain The difficulty is expressed as ;definition The first i attributes ID and The conversion difficulty of the first j attribute IDs is The attribute ID chain is compared in a fixed order (either from front to back or from back to front). If they differ, they are transformed sequentially to ensure that the attribute ID at that position remains consistent. The update function for the transformation difficulty during each transformation is:

[0047]

[0048] When the conversion begins, , The conversion difficulty is obtained by sequentially converting and calculating. Then for any two source data in any source database, Structural similarity based on the composition and generation order of source data attribute IDs can be represented as: ;

[0049] Secondly, local similarity matching based on the shared metadata linked list in the source data.

[0050] The shared metadata linked list in the source data represents the local similarity of the source data. In the attribute ID chain, this is represented by the length of the same local chain. The longer the length of the same local chain, the higher the local similarity. For two source data... Attribute ID chain , Assuming The first m characters and The length of the common local chain of the first n characters is ,but The local similarity is calculated sequentially in a fixed order (from front to back or from back to front), and the update function for this function is:

[0051] ;

[0052] After updating sequentially, the lengths of local chains with the same source data are obtained. Based on two source data The local similarity of the metadata linked list in the CPC can be represented as: ,in This refers to the length of the attribute ID chain;

[0053] Weights were assigned to structural similarity and local similarity analysis respectively. , , , Based on the structural similarity and local similarity determined in steps 4a and 4b, two source data are identified. Overall similarity .

[0054] 2d. Locate the source databases that share the same source data, extract the shared source data or the source data generated by merging and unifying them, and merge them to generate merged shared source data. Replace the original shared source data with the merged shared source data. In each source database, perform similarity matching between the merged shared source data and the remaining source data in turn. If they are equal, remove the business data from the remaining source data. If they are not equal, retain it.

[0055] A3. Establish a data storage unit based on the source data structure type and configure the business data storage data linked list;

[0056] The data storage unit performs batch storage of various types of business data based on the business data structure type. The data storage unit also stores data in different formats based on the data type, including XML, DAT, SI, DB, BPA, and SWI. XML and DB are primarily used for storing large volumes of measurement-related business data involved in the distribution network business process; DAT and SWI are primarily used for storing parameter data such as the attribute status of equipment or systems involved in the distribution network business process; and SI and BPA are primarily used for storing distribution network topology and power flow analysis data.

[0057] The data storage linked list establishes a business data chain based on the relationship attributes between source data within the data storage unit, which is used for locating and tracking business data associated with source data, and stores it in a linked list manner;

[0058] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the essence and scope of the technical solutions of the present invention.

Claims

1. A method for batch processing of distribution Internet of Things (IoT) business data, characterized in that, include: A1. Establish rules for defining and describing business data and its attributes, and configure data formats for different types of business data; Different types of business data include: source data, basic business information data, and business synchronization information data; A2. Based on the principle of attribute similarity, perform multi-source data matching, and merge or remove source data whose similarity exceeds a preset threshold; A3. The data storage unit performs batch storage of various types of business data based on the business data structure type. The data storage unit uses different data formats to store different types of data separately. The data storage data linked list establishes a business data chain for locating and tracking source data based on the relationship attributes between source data within the data storage unit, and stores it in a linked list manner.

2. The method for batch processing of distribution IoT business data according to claim 1, characterized in that, The source data includes: distribution network equipment or system measurement business data, distribution network topology and network structure attribute data, and equipment or structure parameter attribute data; the data format refers to data formats such as XML, DAT, SI, DB, BPA, and SWI; among them, XML and DB are mainly used to store large-volume measurement business data involved in the distribution network business process; DAT and SWI are mainly used to store attribute status and parameter business data of equipment or systems involved in the distribution network business process; SI and BPA are mainly used to store distribution network topology and power flow analysis data.

3. The method for batch processing of distribution IoT business data according to claim 1, characterized in that, The specific steps of step A2 include: 2a. Based on the distribution network business, establish attribute IDs to describe the basic attributes of metadata. Metadata refers to the basic business data units or business data unit groups used to constitute source data. Unify the attribute ID format and naming method, establish an attribute ID table, and for each source data, extract the basic attribute IDs in sequence according to the generation process of metadata or metadata groups to generate its attribute ID chain. The order of metadata groups does not affect the ID of the metadata group. 2b. Classify the source data according to the type of distribution network business involved, and generate source databases for serving different distribution network businesses. For common source data D involving multiple distribution network businesses, add it to each source database respectively. 2c. For any two source data in each source database, perform attribute ID chain similarity matching. Based on a preset similarity threshold, determine whether the two source data are equivalent. If so, merge them into a single source data. Simultaneously, record the source data corresponding to all shared source data D after matching. Repeat the aforementioned matching process until all source data in each source database are no longer identical. 2d. Locate the source databases that share the same source data, extract the shared source data or the source data generated by merging and unifying them, and merge them to generate merged shared source data. Replace the original shared source data with the merged shared source data. In each source database, perform similarity matching between the merged shared source data and the remaining source data in turn. If they are equal, remove the business data from the remaining source data. If they are not equal, retain it.

4. The method for batch processing of distribution IoT business data according to claim 3, characterized in that, The similarity matching of the attribute ID chain specifically includes: Structural similarity matching based on the composition and generation order of source data attribute IDs Since the composition and generation order of attribute IDs are in the form of a linked list, the ease of conversion between linked lists can be used to determine their suitability for two source data. Attribute ID chain Assuming It contains M attributes, ID, It contains M attribute IDs, and analyzes the attribute ID chain by editing the attribute ID position and quantity. Convert to attribute ID chain Difficulty; Definition The first i attributes ID and The conversion difficulty of the first j attribute IDs is The attribute ID chain is compared in a fixed order (either from front to back or from back to front). If they differ, they are transformed sequentially to ensure that the attribute ID at that position remains consistent. The update function for the transformation difficulty during each transformation is: When the conversion begins, , The conversion difficulty is obtained by sequentially converting and calculating. Then for any two source data in any source database, The similarity based on the composition and generation order of the source data attribute IDs can be represented as: ; Local similarity matching based on shared metadata linked lists in the source data The shared metadata linked list in the source data represents the local similarity of the source data. In the attribute ID chain, this is represented by the length of the same local chain. The longer the length of the same local chain, the higher the local similarity. For two source data... Attribute ID chain , Assuming The first m characters and The length of the common local chain of the first n characters is ,but The local similarity is calculated sequentially in a fixed order (from front to back or from back to front), and the update function for this function is: ； After updating sequentially, the lengths of local chains with the same source data are obtained. Based on two source data The local similarity of the metadata linked list in the CPC can be represented as: ,in This refers to the length of the attribute ID chain; Weights were assigned to structural similarity and local similarity analysis respectively. , , , Based on the structural similarity and local similarity determined in steps 4a and 4b, two source data are identified. Overall similarity .

5. The method for batch processing of distribution IoT business data according to claim 1, characterized in that, The source data is the basic data used to complete various business operations of the distribution network, including: distribution network equipment or system measurement business data, distribution network topology and network structure attribute data, and equipment or structure parameter attribute data; Basic business information data refers to the planned label data used to define and standardize the attributes of distribution network business, including: distribution network business ID, code, and business process code; The business synchronization information data is based on the basic business information and is used to connect and associate a series of power distribution network operation tasks as well as supporting associated data for cloud edge business collaboration.

6. The method for batch processing of distribution IoT business data according to claim 1, characterized in that, Configuring data formats for different types of business data means, in addition to the solidified models and graphical data, standardizing the data that is easy to express digitally and processing it in a standardized manner using a data management method that matches the storage management library.

Images