Power data processing method and system based on system learning

Abstract

The invention discloses an electric power data processing method and system based on system learning, relates to the technical field of electric power planning, and aims to solve the problem that an existing electric power data processing method cannot perform association processing on different types of data. The method comprises the following steps: receiving dispersed data, and processing non-structural data to generate second structural data; classifying the first structural data and the second structural data according to the power data elements to form a plurality of to-be-processed data sets, and performing repetition screening on the to-be-processed data sets to generate third structural data; performing association analysis on the third structure data to generate relational data; and generating a three-dimensional data set according to the structural data and the relational data, performing first digital graph conversion and splicing processing on the three-dimensional data set to generate an overall graph, and sending and restoring the overall graph for query. According to the method, the association relationship of the data can be fully mined, so that different types of power data are complementary to each other, and a data reference is provided for subsequent power planning and design.

Description

Technical Field

[0001] The present invention relates to the technical field of electric power planning, and in particular to an electric power data processing method and system based on system learning. Background Art

[0002] In the process of power planning and design, the processing of various types of power data is particularly important as a prerequisite. The existing power data processing mainly processes single types of data and cannot associate different types of data. As a result, the existing power data processing process is inefficient and cannot meet the growing needs of power planning and design.

[0003] For example, the Chinese patent with announcement number CN117633881A relates to a method for optimizing and processing power data, including: the user performs identity verification through the client's Web browsing page to determine whether to log into the system, and obtains corresponding access and management permissions according to the role assigned by the user identity; the file package of each collection point is encrypted by the public key issued by the central processor, and uploaded to the cloud through the wireless network, and the cloud decrypts the file package through the private key and stores it; the improved clustering algorithm uses the median of the power data in each file package as the clustering centroid, iteratively aggregates the power data in the file package, and displays abnormal power data that is not included in the cluster value set, and generates warning information and new commands to realize encrypted transmission and access of power data and highlight abnormal data. However, the Chinese patent with announcement number CN117633881A focuses on encrypted access to power data, and does not involve data processing processes of different types or types. Summary of the invention

[0004] The present invention solves the problem that existing power data processing methods are unable to perform correlation processing on data of different categories or types, and proposes a power data processing method and system based on system learning, which can comprehensively process structured data and unstructured data with the help of system learning processing methods, and fully explore the correlation relationship between data, so that power data of different types can complement each other, providing data reference for subsequent power planning and design.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions: A method for processing power data based on system learning comprises the following steps: S1, receiving scattered data, the scattered data including first structured data and unstructured data, and processing the unstructured data to generate second structured data; S2, classifying the first structure data and the second structure data according to the power data elements to form a plurality of data sets to be processed, and performing duplication screening on the data sets to be processed to generate third structure data; S3, performing association analysis on the third structure data, and generating first relationship data and second relationship data according to the timestamp and the power data element; S4, generating a three-dimensional data group according to the structural data and the relationship data, performing a first digital-to-image conversion and splicing process on the three-dimensional data group to generate an overall image, and the overall image is sent and restored for query.

[0006] In the technical solution, firstly, the scattered data is received by the corresponding receiving device, and the unstructured data in the scattered data is processed to obtain the second structured data. The first structured data existing in the scattered data itself and the second structured data obtained by the conversion process are classified according to the power data elements to obtain a number of data sets to be processed. The above data sets are screened for duplication to form the third structured data. After obtaining the third structured data, the first relationship data and the second relationship data are generated from two dimensions. These two relationship data can further reflect the function and data meaning of the third structured data. Finally, the three-dimensional data group is converted into a digital image to obtain an overall image, and the above data is transmitted to the user end by image transmission, so as to be used in the subsequent power planning and design, and provide data reference for the subsequent power planning and design process.

[0007] The present invention is further configured as follows: the step of processing the unstructured data to generate the second structured data comprises: S11, inputting all the unstructured data into the deep learning neural network system model, parsing and outputting the corresponding model output structure data; S12, filtering and removing all format data from the model output structure data, and extracting characteristic structure data; S13, reassemble the characteristic structure data in the order of timestamps to obtain second structure data.

[0008] In the present technical solution, for the conversion process of the above-mentioned unstructured data, it mainly utilizes a pre-trained neural network system model to perform parsing output. The system has a deep learning function. The input of the model is unstructured data, and its output is the corresponding model output structure data. The model output structure data is structured data at this time, but there is a large amount of irrelevant data in the model output structure data. Therefore, after the model outputs the structure data, it is screened and extracted in turn to obtain feature structure data. Since the feature structure data is relatively scattered, it is reassembled according to the timestamp order to obtain the final second structure data.

[0009] The present invention is further configured as follows: Step S2 comprises: S21, classifying and obtaining a number of power data elements according to actual usage of the power data; S22, judging the specific type of power data element to which the first structure data and the second structure data belong based on the key information, and obtaining a plurality of data sets to be processed of different power data elements; S23, deleting duplicate data in the data set to be processed to obtain third structure data.

[0010] In the present technical solution, the specific classification of the power data element is first determined. After the specific classification is determined, the corresponding key information is directly extracted from the first structure data and the second structure data obtained in step S1, and the type of power data element it belongs to is judged based on the key information. After the type of power data element is determined, the duplicate data in the data set to be processed is deleted, and finally the third structure data is obtained.

[0011] The present invention is further configured as follows: deleting duplicate data in the data set to be processed includes: Traverse the data set to be processed of the same power data element and determine the key features in the data to be processed; Extract the data bytes with the same key features and count the number of repeated bytes of the data bytes; if the number of repeated bytes is greater than the repetition judgment range, the data byte is a repeated byte; If the key features are all repeated bytes, the data to be processed is repeated data.

[0012] In the present technical scheme, in order to better perform subsequent data clustering, duplicate data in the data set to be processed is eliminated. First, several key features in the data to be processed are determined. These key features are composed of several data bytes. Statistical analysis is directly performed on these data bytes. If there are two identical data bytes, the two data bytes are duplicate bytes. Specifically, the judgment standard for the two identical data bytes only needs to meet the requirement that the number of duplicate bytes is greater than the duplication judgment range. Subsequently, if all data bytes in the key features are duplicate bytes, the two data to be processed are duplicate data, and one of them can be deleted.

[0013] The present invention is further configured as follows: the performing relationship analysis on the third structure data includes: Extract a third structure data of the same power data element, analyze the relationship between the data and the third structure data corresponding to the adjacent timestamps according to the order of timestamps, and determine whether there is related data. If so, generate the first relationship data according to the current timestamp until all the same power data elements are traversed and a set of first relationship data is generated.

[0014] In the present technical solution, the third structure data under the same power data element determines whether to generate the first relationship data based on whether there is a relationship between it and the third structure data corresponding to the adjacent timestamp. If the third structure data can be associated with the third structure data corresponding to the adjacent timestamp and can be combined to generate new associated data, then the corresponding first relationship data is generated, and finally a set of first relationship data can be generated.

[0015] The present invention is further configured as follows: the performing relationship analysis on the third structure data further includes: Extract several third structure data with the same timestamp but different power data elements, and determine whether there is related data between the several third structure data. If so, generate second relationship data according to the current timestamp until all the third structure data are traversed to generate a set of second relationship data.

[0016] In the technical solution, the third structure data under the same timestamp and different power data elements also have the possibility of data association at the same time point. If there is an association relationship and there can be associated data, the second relationship data is directly generated, and finally a set of second relationship data is generated.

[0017] The present invention is further configured as follows: Step S4 comprises: S41, extracting the third structure data, the first relationship data and the second relationship data, performing normalization processing on them respectively, and combining them to generate a three-dimensional data group; S42, converting the three-dimensional data group into an image-converted data group, using the image-converted data group as the RGB value of the converted image, converting the image-converted data group into a first image, and splicing all the first images to generate an overall image; S43, sending the entire image to the receiving end for restoration and query.

[0018] In the present technical solution, firstly, a corresponding three-dimensional data group is formed according to the structural data and relational data formed in the process before step S4. After the three-dimensional data group is scaled proportionally, an image conversion data group is generated. The three data in the image conversion data group are respectively represented as RGB values ​​of the converted image, thereby forming a number of first images with the same pixel size. Finally, these first images are spliced ​​as a whole, and the corresponding splicing order is recorded to obtain an overall image that can be transmitted. The overall image can be sent to a client with query requirements. After being sent to the client, the above-mentioned overall image is restored and can then be queried to ensure data security during the transmission process.

[0019] The present invention is further configured as follows: the splicing processing includes sequential splicing and irregular splicing, the sequential splicing is specifically splicing according to the order of the data groups, the irregular splicing is specifically splicing according to the random order of the data groups, the random order is generated according to a random function, and the irregular splicing needs to save and record its order.

[0020] In this technical solution, irregular splicing has higher security, and its random order can be generated by a random function. The order of irregular splicing must be saved for use in subsequent data restoration processing. The order of irregular splicing can be transmitted using a separate transmission method.

[0021] The present invention is further configured as follows: the basis for judging whether there is related data is an empirical formula or an AI big data model, and the AI ​​big data model performs AI prediction based on the characteristics of the related data obtained historically.

[0022] In this technical solution, most of the empirical formulas are obtained by combining manual testing, and the AI ​​big data model is generated with the help of AI technology, and its accuracy is higher than that of the empirical formula.

[0023] A power data processing system based on system learning, using the above-mentioned power data processing method based on system learning, comprises: A non-structured data processing unit converts the non-structured data into second structured data; A classification and screening unit, which classifies the structural data according to the power data elements and screens the data sets to be processed for duplication; A correlation analysis unit performs correlation analysis on the structure data and generates first relationship data and second relationship data in sequence; The digital-to-image conversion unit performs digital-to-image conversion on the three-dimensional data set to generate an overall image.

[0024] A power data processing system based on system learning in the technical solution mainly includes a non-structured data processing unit, a classification and screening unit, a correlation analysis unit and a digital-to-image conversion unit. The non-structured data processing unit is connected to the classification and screening unit, the classification and screening unit is connected to the correlation analysis unit, and the correlation analysis unit is connected to the digital-to-image conversion unit. The non-structured data processing unit can convert the non-structured data to generate corresponding second structured data; the classification and screening unit can classify the structured data according to the power data elements and screen the data to be processed; the correlation analysis unit can perform correlation analysis on the third structured data and generate corresponding relationship data, so as to search and analyze the corresponding similar data; the digital-to-image conversion unit can convert the above data to generate a relatively reliable and secure overall graph for data transmission, ensuring the security and tamper-proof of the data.

[0025] The present invention can bring the following beneficial effects: 1. The present invention relates to a power data processing method based on system learning, which can comprehensively process structured data and unstructured data with the help of the system learning processing method, and fully explore the correlation between the data, so that different types of power data can complement each other and provide data reference for subsequent power planning and design; 2. The present invention relates to a power data processing system based on system learning, which fully utilizes the correlation between multi-source data through the sequential and coordinated action of a non-structured data processing unit, a classification and screening unit, a correlation analysis unit, and a digital-to-graphic conversion unit to obtain a more accurate data processing result. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is an overall flow chart of a power data processing method based on system learning in this application.

[0027] Figure 2 This is a flowchart of step S3 in a power data processing method based on system learning in the present application. DETAILED DESCRIPTION

[0028] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation method described herein is only an optimal embodiment of the present invention, which is only used to explain the present invention and does not limit the scope of protection of the present invention. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0029] Example 1 This embodiment proposes a power data processing method based on system learning, referring to Figure 1 , which includes the following steps.

[0030] Step S1, first receive scattered data, specifically, the scattered data is received and acquired by a corresponding receiving device; the above-mentioned scattered data mainly includes first structured data and unstructured data, and the unstructured data needs to be converted and processed to be converted into second structured data.

[0031] In this embodiment, the specific receiving device is determined according to the type of power data actually collected and acquired, and is not limited here.

[0032] In step S2, the first structure data and the second structure data are classified according to the power data element, and the first structure data and the second structure data are classified according to the power data element to obtain a plurality of data sets to be processed, and the power data elements of these data sets to be processed are different; then, the data sets to be processed are screened for duplication to obtain third structure data.

[0033] The above-mentioned power data elements are generally divided according to the purpose and function of the data. Common power data elements are voltage, power and power consumption information. Of course, other data types related to power data are also included, which will not be described one by one here.

[0034] Step S3, after completing step S2, performing association analysis on the third structure data generated in step S2, and obtaining first relationship data and second relationship data based on the timestamp and power data elements.

[0035] Specifically, the first relational data is associated data formed when there is an association relationship between the current third structure data and the third structure data with a close timestamp.

[0036] The second relational data is associated data formed when a plurality of third structure data under different power data elements but with the same timestamp are associated with each other.

[0037] Step S4, using the above structural data and relational data to form a three-dimensional data group, and then performing a first digital-to-image conversion and splicing process on the three-dimensional data group to generate an overall image, which is sent and restored for subsequent query.

[0038] In the above technical scheme, firstly, the scattered data is received by the corresponding receiving device, and the unstructured data in the scattered data is processed to obtain the second structured data. The first structured data existing in the scattered data itself and the second structured data obtained by the conversion process are classified according to the power data elements to obtain several data sets to be processed. The above data sets are screened for duplication to form the third structured data. After obtaining the third structured data, the first relationship data and the second relationship data are generated from two dimensions. These two relationship data can further reflect the function and data meaning of the third structured data. Finally, the three-dimensional data group is converted into a digital image to obtain an overall image, and the above data is transmitted to the user end by image transmission, so as to prepare for the use of subsequent power planning and design, and provide data reference for the subsequent power planning and design process.

[0039] Example 2 This embodiment describes the steps of embodiment 1 in detail.

[0040] Regarding the step S1 of processing the non-structured data to generate the second structured data, the specific process includes the following several sub-steps.

[0041] Step S11, input the above-mentioned unstructured data into the deep learning neural network system model, and then parse the output to obtain the corresponding model output structure data.

[0042] In this embodiment, the unstructured data is initially converted through a deep learning neural network system model, which has higher accuracy.

[0043] Step S12, in the model output structure data, all format data are screened and removed, and then characteristic structure data is extracted.

[0044] Step S13: reassemble the characteristic structure data in the order of timestamps to obtain second structure data.

[0045] In this embodiment, since the re-extracted feature structure data is not strictly in chronological order, it is necessary to re-order and reassemble them.

[0046] In the present technical solution, for the conversion process of the above-mentioned unstructured data, it mainly utilizes a pre-trained neural network system model to perform parsing output. The system has a deep learning function. The input of the model is unstructured data, and its output is the corresponding model output structure data. The model output structure data is structured data at this time, but there is a large amount of irrelevant data in the model output structure data. Therefore, after the model outputs the structure data, it is screened and extracted in turn to obtain feature structure data. Since the feature structure data is relatively scattered, it is reassembled according to the timestamp order to obtain the final second structure data.

[0047] Unstructured data refers to data with irregular or incomplete structure, no predefined data model, and is not convenient to be represented by a two-dimensional logical table in a database. It mainly includes office documents of all formats, text, pictures, HTML, various reports, images, and audio or video information. In the field of intelligent power design, most of it is power video information data.

[0048] The first structure data is the same as the second structure data, and also includes corresponding timestamp data and other characteristic data.

[0049] After completing the above steps S11 to S13, the generated second structure data will be stored in the storage space to which the first structure data belongs, so as to facilitate subsequent unified classification and processing.

[0050] The neural network system model is formed through pre-training, specifically, by collecting a large amount of unstructured data in a historical period and outputting structured data through calculation or processing.

[0051] More specifically, these model output structure data are obtained through empirical formulas or manual processing, and because they are data from a historical period, they have been actually verified, so the large amount of unstructured data from the above historical period and the calculated or processed model output structure data are placed in the corresponding training set and validation set, which has a strong predictive analysis effect. The optimal parameters of the model are determined through the above training set and validation set.

[0052] In addition, the trained neural network system model has deep learning capabilities and can periodically self-update and self-check. For self-checking, it mainly uses actual data to compare with input and output data to determine whether the error range is greater than the preset standard range. If it is greater than the standard range, the input and output data are replaced by actual data, and the actual data is added to the training set for retraining, thereby re-determining the optimal parameters of the model to prepare for more accurate generation of structural data in the future.

[0053] For step S2, in more detail, it mainly includes the following sub-steps.

[0054] In step S21, firstly, a plurality of power data elements are classified according to the actual use of the power data.

[0055] Of course, the power data may be directly classified according to its function to generate multiple power data elements. For example, the power data elements may be voltage, power, and power consumption information.

[0056] Step S22, extracting key information from the first structure data and the second structure data, and determining the specific type of power data element to which the key information belongs based on the extracted key information, to obtain a plurality of data sets to be processed of different power data elements.

[0057] In this embodiment, the key information can also be extracted automatically using the trained neural network system model.

[0058] Step S23, deleting duplicate data in the data set to be processed to obtain third structure data.

[0059] In the above technical solution, the specific classification of the power data element is first determined. After the specific classification is determined, the corresponding key information is directly extracted from the first structure data and the second structure data obtained in step S1, and the type of power data element it belongs to is judged according to the key information. After the type of power data element is determined, the duplicate data in the data set to be processed is deleted, and finally the third structure data is obtained.

[0060] The third structure data can combine the first structure data and the second structure data, and classify the first structure data and the second structure data according to the power data elements. Some repeated parts in the first structure data and the second structure data can also be screened and removed, which facilitates subsequent data processing and analysis.

[0061] For the specific process of deleting duplicate data in the data set to be processed, refer to the following steps.

[0062] The data sets to be processed of the same power data element are traversed to determine the key features in the data to be processed.

[0063] In this embodiment, the key feature here is the sum of all the above key information.

[0064] Extract data bytes with the same key features and count the number of repeated bytes of the data bytes; if the number of repeated bytes is greater than the repetition judgment range, then the data byte is a repeated byte; if the number of repeated bytes is less than or equal to the repetition judgment range, then the data byte is not a repeated byte.

[0065] If all key features are repeated bytes, then the data to be processed is repeated data; if any key feature is not a repeated byte, then the data to be processed is not repeated data.

[0066] In the above technical solution, in order to better perform subsequent data clustering, duplicate data in the data set to be processed is eliminated. First, several key features in the data to be processed are determined. These key features are composed of several data bytes. These data bytes are directly statistically analyzed. If there are two identical data bytes, the two data bytes are duplicate bytes. Specifically, the judgment standard for the two identical data bytes only needs to meet the requirement that the number of duplicate bytes is greater than the duplication judgment range; then, if all data bytes in the key features are duplicate bytes, the two data to be processed are duplicate data, and one of them can be deleted.

[0067] For the repetition judgment range, it is generally satisfied that only the maximum value is set, that is, as long as the number of repeated bytes exceeds this maximum value, the data byte is determined to be a repeated byte. For example, in a data byte with 10 bytes, the maximum value is set to 8, that is, after the number of repeated bytes exceeds 8, the entire data byte is a repeated byte; because in the same data byte, there are generally 2-3 bits of irrelevant data, so generally only the following formula needs to be satisfied: N = number of repeated bytes / total number of data bytes Wherein, N is the repetition rate, which is generally between 75% and 85%. In the above example of this embodiment, N is 80%. Of course, it can also be adjusted within the range of 75% to 85% according to the actual power data, and is not limited here. Of course, there may be some extreme data where N is not within the above range of 75% to 85%.

[0068] The above key features generally include a number of data bytes. If these data bytes are all repeated bytes, the corresponding data to be processed is determined to be repeated data.

[0069] The process of performing association analysis on the third structure data in step S3 mainly includes the process of generating the first relationship data and the process of generating the second relationship data. Figure 1 and Figure 2 , including the following processes.

[0070] The generation process of the first relationship data is specifically as follows: first, a third structure data of the same power data element is extracted, and then the relationship between the data and the third structure data corresponding to the adjacent timestamps is analyzed according to the order of timestamps to determine whether there is related data. If so, the first relationship data is generated according to the current timestamp until all the third structure data of the same power data element are traversed to generate a collection of the first relationship data.

[0071] In the above technical solution, the third structure data under the same power data element determines whether to generate the first relationship data based on whether it has a relationship with the third structure data corresponding to the adjacent timestamp. If the third structure data can be associated with the third structure data corresponding to the adjacent timestamp and can be combined to generate new associated data, then the corresponding first relationship data is generated, and finally a set of first relationship data can be generated.

[0072] For the above-mentioned adjacent timestamps, the size range of the timestamp is generally determined according to actual conditions. For example, the time is 4 p.m. on a certain day of a certain month of a certain year, and the third structure data is voltage data. The range of the temporary timestamp is set to 1 hour before and after. Then the voltage data can be searched for a corresponding relationship with all data from 3 p.m. to 5 p.m. If at 5 p.m., the voltage data suddenly changes and exceeds the preset range, it is deemed that there is a correlation between 5 p.m. and 4 p.m., that is, the voltage data at 5 p.m. is the correlated data, and the abnormal voltage change can be analyzed and generated based on the correlated data, which is the above-mentioned first relationship data; the first relationship data is recorded according to the timestamp of 4 p.m., that is, the abnormal voltage change at 4 p.m. is recorded.

[0073] According to actual needs, determine which power data elements' third structure data need to generate first relationship data. The process of generating first relationship data and sets of first relationship data from the third structure under other power data elements is the same as above and will not be summarized here.

[0074] The generation process of the second relationship data is specifically as follows: extracting several third structure data with the same timestamp but different power data elements, and determining whether there is related data between the several third structure data. If so, the second relationship data is generated according to the current timestamp until all the third structure data are traversed to generate a set of second relationship data.

[0075] In the above technical solution, the third structure data under the same timestamp and different power data elements also have the possibility of data association because they are at the same time point. If there is an association relationship and there can be associated data, the second relationship data is directly generated, and finally a set of second relationship data is generated.

[0076] In this embodiment, for example, there are two electricity data elements, namely, electricity consumption and real-time electricity price, that is, there is a correlation between the two, and related data can be generated. Because real-time electricity price information can be obtained based on electricity consumption and real-time electricity price, this real-time electricity price information can be regarded as second relationship data.

[0077] Step S4 specifically includes the following sub-steps.

[0078] Step S41 , extracting the third structure data, the first relationship data and the second relationship data, normalizing them respectively, and finally combining them to obtain a three-dimensional data group.

[0079] In this embodiment, the order of the three-dimensional data groups is not limited.

[0080] Step S42, processing and converting the three-dimensional data group generated in step S41 to obtain an image conversion data group, and using the image conversion data group as the RGB value of the converted image, converting the image conversion data group to obtain a first image, and finally splicing all the first images to obtain the overall image.

[0081] The specific method of processing and converting the three-dimensional data set to obtain the image conversion data set is to multiply the three-dimensional data set by a corresponding scaling factor.

[0082] Step S43, based on step S42, the entire image is sent to the receiving end, and after receiving the corresponding image data, the receiving end returns the image for subsequent query and analysis.

[0083] In the above technical solution, firstly, a corresponding three-dimensional data group is formed according to the structural data and relational data formed in the process before step S4. After the three-dimensional data group is scaled proportionally, an image conversion data group is generated. The three data in the image conversion data group are respectively represented as RGB values ​​of the converted image, thereby forming a number of first images with the same pixel size. Finally, these first images are spliced ​​as a whole, and the corresponding splicing order is recorded to obtain an overall image that can be transmitted. The overall image can be sent to a client with query requirements. After being sent to the client, the above-mentioned overall image is restored and can then be queried to ensure data security during the transmission process.

[0084] For example, after normalization, the obtained three-dimensional data group is represented as (0.5, 0.2, 1). This three-dimensional data group is scaled proportionally. In order to facilitate conversion according to RGB values, this three-dimensional data group is multiplied by the same scaling factor, which is 255. Therefore, the image-converted data group (127.5, 51, 255) is obtained. Subsequently, 127.5, 51, 255 are respectively used as the R value, G value and B value of the converted image to map to form a first image. According to the above rules, all the first images are spliced ​​to finally obtain the overall image.

[0085] In the above-mentioned splicing process, the splicing processing methods used mainly include sequential splicing and irregular splicing.

[0086] For sequential splicing, it is mainly spliced ​​according to the order of data groups, and this splicing method has relatively low security.

[0087] For irregular splicing, it is mainly spliced ​​according to the random order of the data group. The random order is generated according to a random function. The irregular splicing needs to save and record its order.

[0088] In the above technical solution, irregular splicing has higher security, and its random order can be generated by a random function. The order of irregular splicing must be saved for use in subsequent data restoration processing. The order of irregular splicing can be transmitted using a separate transmission method.

[0089] In the above process of determining whether there is related data, the determination is based on an AI big data model or an empirical formula. The AI ​​big data model performs AI prediction based on the characteristics of historical related data, and most of the empirical formulas are obtained in combination with manual detection.

[0090] In the above technical solution, the AI ​​big data model is generated with the help of AI technology, and its accuracy is higher than that of the empirical formula.

[0091] Similarly, the AI ​​big data model also has deep learning capabilities, which can continuously learn and update to ensure that the entire power data processing system keeps pace with the times and ensure the accuracy of data processing.

[0092] Example 3 On the basis of Example 1 or Example 2, this embodiment further proposes a power data processing system based on system learning, which mainly includes a non-structured data processing unit, a classification and screening unit, a correlation analysis unit and a digital-to-image conversion unit.

[0093] The non-structured data processing unit corresponds to step S1 in the first embodiment, and can convert the non-structured data obtained by summarizing the scattered data into the second structured data.

[0094] The classification and screening unit corresponds to step S2 in Example 1, and can classify the structural data according to the power data elements to obtain the data set to be processed, and screen the data set to be processed for duplication.

[0095] The association analysis unit corresponds to step S3 in Embodiment 1. Specifically, it can perform association analysis on the third structure data and generate first relationship data and second relationship data in sequence.

[0096] The digital-to-image conversion unit mainly performs the digital-to-image conversion on the three-dimensional data set and finally generates an overall image. For more detailed information, please refer to the content of step S4 in embodiment 1 or the content of sub-steps of step S4 in embodiment 2.

[0097] In a power data processing system based on system learning in the present technical solution, the non-structured data processing unit is connected to the classification and screening unit, the classification and screening unit is connected to the correlation analysis unit, and the correlation analysis unit is connected to the digital-to-image conversion unit.

[0098] The non-structured data processing unit can convert the non-structured data to generate corresponding second structured data; the classification and screening unit can classify the structured data according to the power data elements and screen the data to be processed; the association analysis unit can perform association analysis on the third structured data to generate corresponding relationship data, so as to search and analyze the corresponding similar data; the data-to-graph conversion unit can convert the above data to generate a relatively reliable and secure overall graph to transmit the data, thereby ensuring the security and tamper-proofing of the data.

[0099] In addition, the non-structured data processing unit is also connected to a data receiving unit, and the data receiving unit can obtain scattered data including the first structured data and the non-structured data.

[0100] The association analysis unit mainly includes a first relationship data generating part and a second relationship data generating part, and the two can operate independently.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention is described in detail with reference to the embodiments, it should be understood by those skilled in the art that any modification or equivalent replacement of the technical solutions of the present invention does not depart from the spirit and scope of the technical solutions of the present invention and should be included in the scope of the claims of the present invention.

Claims

1. A method for processing power data based on system learning, characterized in that: The following steps are involved: S1, receiving scattered data, the scattered data including first structured data and unstructured data, and processing the unstructured data to generate second structured data; S2, classifying the first structure data and the second structure data according to the power data elements to form a plurality of data sets to be processed, and performing duplication screening on the data sets to be processed to generate third structure data; S3, performing association analysis on the third structure data, and generating first relationship data and second relationship data according to the timestamp and the power data element; S4, generating a three-dimensional data group according to the structural data and the relationship data, performing a first digital-to-image conversion and splicing process on the three-dimensional data group to generate an overall image, and the overall image is sent and restored for query.

2. A method for processing power data based on system learning according to claim 1, characterized in that: The step of processing the non-structured data to generate the second structured data includes: S11, inputting all the unstructured data into the deep learning neural network system model, parsing and outputting the corresponding model output structure data; S12, filtering and removing all format data from the model output structure data, and extracting characteristic structure data; S13, reassemble the characteristic structure data in the order of timestamps to obtain second structure data.

3. The power data processing method based on system learning according to claim 1 is characterized in that: The step S2 comprises: S21, classifying and obtaining a number of power data elements according to actual usage of the power data; S22, judging the specific type of power data element to which the first structure data and the second structure data belong based on the key information, and obtaining a plurality of data sets to be processed of different power data elements; S23, deleting duplicate data in the data set to be processed to obtain third structure data.

4. The method for processing power data based on system learning according to claim 3, characterized in that: Deleting duplicate data in the data set to be processed includes: Traverse the data set to be processed of the same power data element and determine the key features in the data to be processed; Extract the data bytes with the same key features and count the number of repeated bytes of the data bytes; if the number of repeated bytes is greater than the repetition judgment range, the data byte is a repeated byte; If the key features are all repeated bytes, the data to be processed is repeated data.

5. The method for processing power data based on system learning according to claim 1, characterized in that: The performing relationship analysis on the third structure data includes: Extract a third structure data of the same power data element, analyze the relationship between the data and the third structure data corresponding to the adjacent timestamps according to the order of timestamps, and determine whether there is related data. If so, generate the first relationship data according to the current timestamp until all the same power data elements are traversed and a set of first relationship data is generated.

6. The method for processing power data based on system learning according to claim 5, characterized in that: The performing relationship analysis on the third structure data further includes: Extract several third structure data with the same timestamp but different power data elements, and determine whether there is related data between the several third structure data. If so, generate second relationship data according to the current timestamp until all the third structure data are traversed to generate a set of second relationship data.

7. A method for processing power data based on system learning according to any one of claims 1 to 6, characterized in that: The step S4 comprises: S41, extracting the third structure data, the first relationship data and the second relationship data, performing normalization processing on them respectively, and combining them to generate a three-dimensional data group; S42, converting the three-dimensional data group into an image-converted data group, using the image-converted data group as the RGB value of the converted image, converting the image-converted data group into a first image, and splicing all the first images to generate an overall image; S43, sending the entire image to the receiving end for restoration and query.

8. A method for processing power data based on system learning according to any one of claims 1 to 6, characterized in that: The splicing process includes sequential splicing and irregular splicing. The sequential splicing is specifically splicing according to the order of the data groups, and the irregular splicing is specifically splicing according to the random order of the data groups. The random order is generated according to a random function, and the irregular splicing needs to save and record its order.

9. The method for processing power data based on system learning according to claim 5, characterized in that: The basis for judging whether there is related data is an empirical formula or an AI big data model, and the AI ​​big data model performs AI prediction based on the characteristics of the related data obtained historically.

10. A power data processing system based on system learning, adopting a power data processing method based on system learning according to any one of claims 1 to 9, characterized in that: include: A non-structured data processing unit converts the non-structured data into second structured data; A classification and screening unit, which classifies the structural data according to the power data elements and screens the data sets to be processed for duplication; A correlation analysis unit performs correlation analysis on the structure data and generates first relationship data and second relationship data in sequence; The digital-to-image conversion unit performs digital-to-image conversion on the three-dimensional data set to generate an overall image.

Images