Power grid monitoring data processing method and device, equipment and storage medium

Abstract

The application provides a power grid monitoring data processing method and device, equipment and a storage medium, first, the first structured data and unstructured data are acquired; then the unstructured data is converted into semi-structured data, and the first correlation coefficient of the first structured data and the semi-structured data is determined according to the first knowledge graph; then the first structured data with the first correlation coefficient greater than the first preset threshold is selected to form the first data group; then the semi-structured data is all converted into the second structured data; finally, the repetitive data is determined and removed according to the second structured data and the first data group. By comparing the correlation between the characteristic information of the semi-structured data and the first structured data, the semi-structured data and the first structured data that may be repetitive are associated, and finally the repetitive data is screened out after the semi-structured data is converted into structured data, so that the waste of storage resources and processing resources caused by data repetition is avoided.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of data processing, and particularly relates to a power grid monitoring data processing method and device, equipment and a storage medium. BACKGROUND

[0002] The power system has a very large amount of information in the running process, which can be divided into structured data and unstructured data. In order to better master the information of the power system, the running state needs to be monitored in real time, and the generated structured data and unstructured data need to be analyzed. Due to the complex characteristics of unstructured data, it is difficult to store, process and use, so it is generally necessary to process the unstructured data.

[0003] In the prior art, unstructured data and structured data are usually processed separately, but since structured data and unstructured data are closely related and have strong correlation, separate processing will produce a large amount of repetitive data, causing waste of storage resources and processing resources. SUMMARY

[0004] Therefore, the present application provides a power grid monitoring data processing method, device, equipment and storage medium, aiming to solve the problem of a large amount of repetitive data in the prior art when processing power grid monitoring data.

[0005] The first aspect of the embodiment of the present application provides a power grid monitoring data processing method, comprising:

[0006] obtaining power grid monitoring data; wherein the power grid monitoring data comprises first structured data and unstructured data;

[0007] transforming the unstructured data into semi-structured data, and determining a first correlation coefficient between the first structured data and the semi-structured data according to a first knowledge graph established in advance;

[0008] For each semi-structured data, selecting first structured data with a first correlation coefficient greater than a first preset threshold value between the semi-structured data to form a first data group of the semi-structured data;

[0009] transforming all semi-structured data into second structured data;

[0010] determining and removing repetitive data according to the second structured data and the first structured data in the first data group to complete data processing.

[0011] The second aspect of the embodiment of the present application provides a power grid monitoring data processing device, comprising:

[0012] The acquisition module is configured to acquire power grid monitoring data; wherein the power grid monitoring data comprises first structured data and unstructured data;

[0013] The determination module is configured to convert the unstructured data into semi-structured data, and determine a first correlation coefficient between the first structured data and the semi-structured data according to a first knowledge graph established in advance;

[0014] The grouping module is configured to select, for each semi-structured data, first structured data having a first correlation coefficient greater than a first preset threshold with the semi-structured data, to form a first data group of the semi-structured data;

[0015] The conversion module is configured to convert all the semi-structured data into second structured data;

[0016] The deduplication module is configured to determine and remove repetitive data according to the second structured data and the first structured data in the first data group, to complete data processing.

[0017] The third aspect of the embodiment of the present application provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the power grid monitoring data processing method of the first aspect when executing the computer program.

[0018] The fourth aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the steps of the power grid monitoring data processing method of the first aspect.

[0019] The power grid monitoring data processing method, device, equipment and storage medium provided by the embodiment of the present application firstly acquire power grid monitoring data; wherein the power grid monitoring data comprises first structured data and unstructured data; then the unstructured data is converted into semi-structured data, and a first correlation coefficient between the first structured data and the semi-structured data is determined according to a first knowledge graph established in advance; then for each semi-structured data, first structured data having a first correlation coefficient greater than a first preset threshold with the semi-structured data is selected to form a first data group of the semi-structured data; then all the semi-structured data is converted into second structured data; finally, repetitive data is determined and removed according to the second structured data and the first structured data in the first data group, to complete data processing. By comparing the correlation between the characteristic information of the semi-structured data and the first structured data, the semi-structured data and the first structured data that may be repetitive are associated, and finally the repetitive data is screened out after the semi-structured data is converted into structured data, so that the waste of storage resources and processing resources caused by data repetition is avoided. BRIEF DESCRIPTION OF DRAWINGS

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is an application scenario diagram of the power grid monitoring data processing method provided in the embodiments of the present invention;

[0022] Figure 2 This is a flowchart illustrating the implementation of the power grid monitoring data processing method provided in this embodiment of the invention.

[0023] Figure 3 This is a schematic diagram of the structure of the power grid monitoring data processing device provided in an embodiment of the present invention;

[0024] Figure 4 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0025] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0026] Figure 1 This is an application scenario diagram of the power grid monitoring data processing method provided in an embodiment of the present invention. The application scenarios of the power grid monitoring data processing method provided by the present invention may include, but are not limited to, this system. For example... Figure 1 The system includes at least one power grid monitoring terminal 11 and electronic equipment 12.

[0027] The power grid monitoring terminal 11 is used to record the operation data of the power system in its area and report it to the electronic device 12. The electronic device 12 can be a terminal or a server. The terminal can be a computer, laptop, etc., and the server can be a physical server or a cloud server, which is not limited here. The power grid monitoring terminal 11 and the electronic device 12 can be the same device or different devices, which is not limited here.

[0028] In the prior art, unstructured data and structured data are usually processed separately, but since the structured data and the unstructured data are closely related and have strong correlation, separate processing will generate a large amount of repetitive data, causing waste of storage resources and processing resources. The processing process of the unstructured data usually first processes the unstructured data into semi-structured data, and then further processes the semi-structured data into structured data.

[0029] In the present application, the way to remove repetitive data can be to first convert the unstructured data into structured data B, and then compare the structured data B with the structured data A in the original power grid monitoring data one by one, so as to determine and remove the repetitive data.

[0030] However, since the data amount of the power grid monitoring data is large, and the order of magnitude of the data after the unstructured data is converted into two-dimensional structured data becomes larger (for example, one video file can be converted into a large amount of two-dimensional text data), the above-mentioned comparison method has a huge amount of calculation and low data processing efficiency. Therefore, further improvement is needed.

[0031] Figure 2 is the implementation flowchart of the power grid monitoring data processing method provided by the embodiment of the present application. As shown in Figure 2 , in some embodiments, the power grid monitoring data processing method is applied to the electronic device 12 shown in Figure 1 , and the method comprises the following steps.

[0032] S210, obtaining power grid monitoring data; wherein the power grid monitoring data comprises first structured data and unstructured data.

[0033] In the embodiment of the present application, the first structured data is stored in a database and can be logically expressed and implemented by a two-dimensional table structure. The unstructured data can include but is not limited to at least one of the following: video, picture, voice, and electronic text, without limitation.

[0034] S220, converting the unstructured data into semi-structured data, and determining a first correlation coefficient between the first structured data and the semi-structured data according to a first knowledge graph established in advance.

[0035] In the embodiment of the present application, the feature information of the unstructured data needs to be extracted in the process of converting the unstructured data into semi-structured data. The semantic triple in the first knowledge graph established in advance is: feature information of semi-structured data-correlation-structured data.

[0036] S230, for each semi-structured data, selecting the first structured data having a first correlation coefficient greater than a first preset threshold value with the semi-structured data to form a first data group of the semi-structured data.

[0037] In the embodiment of the present application, the first preset threshold value can be any value between 0 and 1, which is determined according to actual needs. The greater the first preset threshold value is, the smaller the data amount of the screened first data group is, the smaller the calculation amount of the repetitive comparison is, but the removal rate of the repetitive data will be reduced.

[0038] S240, converting all the semi-structured data into second structured data.

[0039] S250, determining and removing repetitive data according to the second structured data and the first structured data in the first data group, to complete data processing.

[0040] In the embodiment of the present application, for semi-structured data M, the corresponding first structured data n can be retrieved according to the first knowledge graph i , to form a first data group N. When the semi-structured data M is converted into second structured data m j , n i and m j are compared, all m j converted from the first data group N and the semi-structured data M are traversed, so as to determine the repetitive data.

[0041] In the embodiment of the present application, by comparing the correlation between the characteristic information of the semi-structured data and the first structured data, the semi-structured data that may be repetitive is associated with the first structured data, and finally the associated data is screened out after the semi-structured data is converted into structured data, so as to avoid the waste of storage resources and processing resources caused by data repetition.

[0042] In some embodiments, S250 can include: comparing the second structured data corresponding to each semi-structured data with the first structured data in the first data group corresponding to the semi-structured data, to determine and remove repetitive data.

[0043] In the embodiment of the present application, for each semi-structured data, after the semi-structured data is converted into second structured data, the second structured data converted from the semi-structured data is placed into a second data group, and then the second data group is compared with the corresponding first data group to remove repetitive data. Compared with the way of comparing all data, the present application can effectively reduce the calculation amount of data processing while ensuring the removal of repetitive data, by comparing only the second data group and the first data group with high correlation.

[0044] In some embodiments, after S250, the method further comprises: calculating the group repetition rate of each first data group according to the number of second structured data, the number of first structured data, and the number of repetitive data; determining the processing effectiveness of the power grid monitoring data according to the group repetition rate of each first data group and a preset repetition rate threshold; when the processing effectiveness of the power grid monitoring data is low, comparing each second structured data with each first structured data again to obtain a comparison result; and updating the first knowledge graph according to the comparison result.

[0045] In the embodiments of the present application, only the data with high correlation is compared, which can effectively reduce the data processing time, but also increases the risk that the repetitive data cannot be effectively processed. Therefore, after each processing is completed, the group repetition rate can be counted, and when the average of the group repetition rate is less than the preset repetition rate threshold, it indicates that the processing effectiveness of the power grid monitoring data is low, and then the data is traversed in the traditional way.

[0046] wherein the group repetition rate corresponding to each semi-structured data = c / (a+b-c), a is the total number of first structured data in the first data group, b is the total number of second structured data in the second data group, and c is the number of repetitive data.

[0047] The average of the group repetition rate corresponding to each semi-structured data is calculated and compared with a preset repetition rate threshold, so as to determine the processing effectiveness of the power grid monitoring data.

[0048] In some embodiments, after S230, the method further comprises: determining the second correlation coefficient between each semi-structured data according to the pre-established second knowledge graph; and for each semi-structured data, when the second correlation coefficient between another semi-structured data and the semi-structured data is greater than a preset threshold, marking the first data group of the other semi-structured data as the adjacent first data group of the semi-structured data.

[0049] Correspondingly, S250 can comprise: comparing the second structured data corresponding to each semi-structured data with the first structured data in the first data group corresponding to the semi-structured data and the first structured data in the adjacent first data group of the semi-structured data, respectively, to determine and remove repetitive data.

[0050] In the embodiment of the present application, the structure of the semantic triple of the second knowledge graph is the feature information of the semi-structured data A-relevance-feature information of the semi-structured data B. Since the semi-structured data is converted from the unstructured data, the types of the unstructured data are various, and different unstructured data are likely to be similar (for example, the recorded contents of the video and the voice are the same or similar), and the corresponding first data groups are also likely to be similar. Therefore, for the semi-structured data A and the semi-structured data B, the first data group corresponding to the semi-structured data A and the first data group corresponding to the semi-structured data B can be marked as adjacent first data groups, and in the comparison of the repetitive data, the semi-structured data A is not only compared with the first data group corresponding to itself, but also compared with the adjacent first data group, so as to improve the screening accuracy of the repetitive data.

[0051] In some embodiments, S220 can include: obtaining the type of the unstructured data; extracting the feature information of the unstructured data according to the type of the unstructured data and the identification model corresponding to each type; and converting the unstructured data into semi-structured data according to the feature information of the unstructured data.

[0052] In the embodiment of the present application, the feature information of the unstructured data is first extracted, and then the unstructured data is converted into semi-structured data, such as an XML format file, according to the feature information and according to certain rules.

[0053] In some embodiments, the type of the unstructured data includes at least one of the following: video, picture, voice, and electronic text; and the identification model includes a first neural network model and a second neural network model. Accordingly, the feature information of the unstructured data is extracted according to the type of the unstructured data and the identification model corresponding to each type, including: when the type of the unstructured data is video or picture, the feature information of the unstructured data is extracted according to the first neural network model; and when the type of the unstructured data is voice or electronic text, the feature information of the unstructured data is extracted according to the second neural network model.

[0054] In the embodiment of the present application, when the type of the unstructured data is video, the video can be divided into multiple images frame by frame, and then feature extraction is performed on each frame of image according to the first neural network model. When the type of the unstructured data is picture, if the picture is large, the picture is segmented into multiple images, and then feature extraction is performed on the segmented multiple images according to the first neural network model, if the picture is small, feature extraction is directly performed on the picture according to the first neural network model. When the type of the unstructured data is voice, the voice is first recognized as text in a preset format, and then feature extraction is performed on the text in the preset format according to the second neural network model. When the type of the unstructured data is electronic text, the electronic text is first converted into text in a preset format, and then feature extraction is performed on the text in the preset format according to the second neural network model.

[0055] In some embodiments, S240 can include: performing parallel processing on the semi-structured data; and converting the semi-structured data after parallel processing into second structured data by using an XML technology.

[0056] In the embodiment of the present application, MapReduce parallel processing is a conventional XML processing method, which can conveniently convert a file into an XML document. Users can also analyze the content and structure of the file in the field according to their own needs to write a corresponding program to output a suitable XML document.

[0057] Since the unstructured data has multiple types, the conversion into semi-structured XML files also has multiple types, and as the amount of data increases, the number of XML files also becomes larger and larger. Because the XML file belongs to semi-structured data, these factors make the structured relational database not suitable for query processing of the XML file. Therefore, before the XML file is converted into structured data, the MapReduce is used to perform parallel processing on the XML file.

[0058] In summary, the beneficial effects of the present application are as follows: by comparing the correlation between the characteristic information of the semi-structured data and the first structured data, the possibly repeated semi-structured data is associated with the first structured data, and finally after the semi-structured data is converted into structured data, the associated data is screened out for repetition, thereby avoiding the waste of storage resources and processing resources caused by data repetition.

[0059] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0060] Figure 3 is a structural schematic diagram of the power grid monitoring data processing device provided by the embodiment of the present application. As shown in Figure 3As shown, in some embodiments, the power grid monitoring data processing apparatus 3 comprises:

[0061] The acquisition module 310 is configured to acquire power grid monitoring data, wherein the power grid monitoring data comprises first structured data and unstructured data.

[0062] The determination module 320 is configured to convert the unstructured data into semi-structured data, and determine a first correlation coefficient between the first structured data and the semi-structured data according to a first knowledge graph established in advance.

[0063] The grouping module 330 is configured to, for each semi-structured data, select first structured data having a first correlation coefficient greater than a first preset threshold value with the semi-structured data, and form a first data group of the semi-structured data.

[0064] The conversion module 340 is configured to convert all the semi-structured data into second structured data.

[0065] The deduplication module 350 is configured to determine and remove repetitive data according to the second structured data and the first structured data in the first data group, so as to complete data processing.

[0066] Optionally, the deduplication module 350 is specifically configured to compare the second structured data corresponding to each semi-structured data with the first structured data in the first data group corresponding to the semi-structured data, determine and remove repetitive data.

[0067] Optionally, the power grid monitoring data processing apparatus 3 further comprises an updating module configured to calculate an intra-group repetition rate of each first data group according to the number of second structured data, the number of first structured data, and the number of repetitive data; determine the processing effectiveness of the power grid monitoring data according to the intra-group repetition rate of each first data group and a preset repetition rate threshold; when the processing effectiveness of the power grid monitoring data is low, compare each second structured data with each first structured data again to obtain a comparison result; and update the first knowledge graph according to the comparison result.

[0068] Optionally, the power grid monitoring data processing apparatus 3 further comprises an adjacent group calculation module configured to determine a second correlation coefficient between each semi-structured data according to a second knowledge graph established in advance; and for each semi-structured data, when another semi-structured data has a second correlation coefficient greater than a preset threshold value with the semi-structured data, mark the first data group of the another semi-structured data as an adjacent first data group of the semi-structured data.

[0069] Correspondingly, the deduplication module 350 is specifically configured to compare the second structured data corresponding to each semi-structured data with the first structured data in the first data group corresponding to the semi-structured data and the first structured data in the adjacent first data group of the semi-structured data respectively, and determine and remove repetitive data.

[0070] Optionally, the determination module 320 is specifically configured to obtain the type of the unstructured data, extract feature information of the unstructured data according to the type of the unstructured data and an identification model corresponding to each type, and convert the unstructured data into semi-structured data according to the feature information of the unstructured data.

[0071] Optionally, the type of the unstructured data includes at least one of the following: video, picture, voice and electronic text; and the identification model includes a first neural network model and a second neural network model. Correspondingly, the determination module 320 is specifically configured to extract the feature information of the unstructured data according to the first neural network model when the type of the unstructured data is video or picture, and extract the feature information of the unstructured data according to the second neural network model when the type of the unstructured data is voice or electronic text.

[0072] Optionally, the conversion module 340 is specifically configured to perform parallel processing on the semi-structured data, and convert the semi-structured data after the parallel processing into second structured data by using an XML technology.

[0073] The power grid monitoring data processing apparatus provided in the embodiment can be used to execute the method embodiments, and has similar implementation principles and technical effects, which will not be described here again.

[0074] Figure 4 is a structural schematic diagram of an electronic device provided in the embodiment of the present application. As shown in Figure 4 The electronic device 4 provided in the embodiment of the present application includes a processor 40, a memory 41, and a computer program 42 stored in the memory 41 and executable on the processor 40. The processor 40 implements the steps in each of the power grid monitoring data processing method embodiments when executing the computer program 42, such as Figure 2 S210 to S250 shown in . Alternatively, the processor 40 implements the functions of each module / unit in each of the system embodiments when executing the computer program 42, such as Figure 3 the functions of the modules 310 to 350 shown in .

[0075] For example, the computer program 42 can be divided into one or more modules / units, one or more modules / units are stored in the memory 41 and executed by the processor 40 to complete the present application. One or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program 42 in the electronic device 4.

[0076] The electronic device 4 can be a terminal or a server, where the terminal can be a mobile phone, MCU, ECU, etc., which is not limited here, and the server can be a physical server, a cloud server, etc., which is not limited here. The electronic device 4 can include, but is not limited to, the processor 40 and the memory 41. Those skilled in the art can understand that the electronic device 4 can include more or fewer components than those shown, or combine certain components, or include different components, for example, the terminal can also include an input / output device, a network access device, a bus, etc. Figure 4 The electronic device 4 is only an example and does not constitute a limitation on the electronic device 4, and can include more or fewer components than those shown, or combine certain components, or different components, for example, the terminal can also include an input / output device, a network access device, a bus, etc.

[0077] The processor 40 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0078] The memory 41 can be an internal storage unit of the electronic device 4, such as a hard disk or memory of the electronic device 4. The memory 41 can also be an external storage device of the electronic device 4, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 4. Further, the memory 41 can include both the internal storage unit and the external storage device of the electronic device 4. The memory 41 is used to store computer programs and other programs and data required by the terminal. The memory 41 can also be used to temporarily store data that has been output or will be output.

[0079] The embodiment of the present application provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps in the power grid monitoring data processing method embodiment.

[0080] The computer readable storage medium stores a computer program 42, the computer program 42 includes program instructions, the program instructions are executed by the processor 40 to realize all or part of the processes in the above embodiment method, and the related hardware can also be completed by the computer program 42, the computer program 42 can be stored in a computer readable storage medium, and the computer program 42 can realize the steps of each method embodiment when being executed by the processor 40. The computer program 42 includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium can include any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier wave signal, telecommunication signal and software distribution medium.

[0081] The computer readable storage medium can be an internal storage unit of the terminal in any of the foregoing embodiments, for example, a hard disk or a memory of the terminal. The computer readable storage medium can also be an external storage device of the terminal, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card or the like. Further, the computer readable storage medium can include both the internal storage unit and the external storage device of the terminal. The computer readable storage medium is used to store the computer program and other programs and data required by the terminal. The computer readable storage medium can also be used to temporarily store data that has been output or will be output.

[0082] It should be understood that the size of the serial number of each step in the above embodiment does not mean the execution order, the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0083] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be realized in the form of hardware or software function unit. In addition, the specific name of each functional unit and module is only for the convenience of mutual distinction, and does not limit the protection scope of the present application. The specific working process of the unit and module in the above system can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

[0084] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0085] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0086] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / terminal and method can be implemented by other ways. For example, the above-mentioned apparatus / terminal embodiments are only schematic, and the division of the modules or units is only a logical function division, and there can be another division way in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0087] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0088] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0089] If the integrated module / unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of each method embodiment described above can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0090] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit it; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for processing power grid monitoring data, characterized in that, include: Acquire power grid monitoring data; wherein, the power grid monitoring data includes a first type of structured data and unstructured data; Unstructured data is transformed into semi-structured data, and a first correlation coefficient between the first structured data and the semi-structured data is determined based on a pre-established first knowledge graph; wherein, the semantic triples in the pre-established first knowledge graph are the feature information-relevance-structured data of the semi-structured data. For each semi-structured data, select the first structured data with a first correlation coefficient greater than a first preset threshold to form the first data group of the semi-structured data; Convert all semi-structured data into second-structured data; Based on the second structured data and the first structured data in the first data group, duplicate data is identified and removed to complete the data processing. The step of determining and removing duplicate data based on the second structured data and the first structured data in the first data group includes: Compare the second structured data corresponding to each semi-structured data with the first structured data in the first data group corresponding to that semi-structured data to identify and remove duplicate data; After comparing the second structured data corresponding to each semi-structured data with the first structured data in the first data group corresponding to that semi-structured data, and identifying and removing duplicate data, the method further includes: Calculate the intra-group repetition rate for each first data group based on the number of second structured data, the number of first structured data, and the number of repetitive data. The processing effectiveness of the power grid monitoring data is determined based on the intra-group repetition rate of each first data group and the preset repetition rate threshold. When the processing validity of the power grid monitoring data is low, each second structured data is compared with each first structured data again to obtain the comparison result; The first knowledge graph is updated based on the comparison results.

2. The power grid monitoring data processing method according to claim 1, characterized in that, After selecting first structured data with a first correlation coefficient greater than a first preset threshold for each semi-structured data set, and forming a first data group for that semi-structured data set, the method further includes: Based on the pre-established second knowledge graph, determine the second correlation coefficient between each semi-structured data; For each semi-structured data, when the second correlation coefficient between another semi-structured data and the semi-structured data is greater than a preset threshold, the first data group of the other semi-structured data is marked as the adjacent first data group of the semi-structured data. The step of determining and removing duplicate data based on the second structured data and the first structured data in the first data group includes: The second structured data corresponding to each semi-structured data is compared with the first structured data in the first data group corresponding to the semi-structured data, as well as the first structured data in the adjacent first data group of the semi-structured data, to identify and remove duplicate data.

3. The power grid monitoring data processing method according to claim 1, characterized in that, The process of converting unstructured data into semi-structured data includes: Obtain the type of the unstructured data; Based on the type of unstructured data and the recognition model corresponding to each type, extract the feature information of the unstructured data; Based on the characteristic information of the unstructured data, the unstructured data is transformed into semi-structured data.

4. The power grid monitoring data processing method according to claim 3, characterized in that, The types of unstructured data include at least one of the following: video, image, audio, and electronic text; the recognition model includes a first neural network model and a second neural network model. Based on the type of unstructured data and the corresponding recognition model for each type, feature information of the unstructured data is extracted, including: When the type of unstructured data is video or image, feature information of the unstructured data is extracted according to the first neural network model; When the type of unstructured data is speech or electronic text, the feature information of the unstructured data is extracted according to the second neural network model.

5. The power grid monitoring data processing method according to any one of claims 1-4, characterized in that, Transform all semi-structured data into second-structured data, including: The semi-structured data is processed in parallel; The semi-structured data after parallel processing is transformed into the second structured data using XML technology.

6. A power grid monitoring data processing device, characterized in that, include: An acquisition module is used to acquire power grid monitoring data; wherein, the power grid monitoring data includes a first type of structured data and unstructured data; The determination module is used to convert unstructured data into semi-structured data and determine the first correlation coefficient between the first structured data and the semi-structured data based on the pre-established first knowledge graph; wherein, the semantic triples in the pre-established first knowledge graph are the feature information-correlation-structured data of the semi-structured data; The grouping module is used to select, for each semi-structured data, first structured data with a first correlation coefficient greater than a first preset threshold, and form a first data group of the semi-structured data. The transformation module is used to convert all semi-structured data into second-structured data; The deduplication module is used to compare the second structured data corresponding to each semi-structured data with the first structured data in the first data group corresponding to the semi-structured data, identify and remove duplicate data, so as to complete the data processing. The deduplication module is further configured to calculate the intra-group duplication rate of each first data group based on the number of second structured data, the number of first structured data, and the number of duplicate data; determine the processing validity of the power grid monitoring data based on the intra-group duplication rate of each first data group and a preset duplication rate threshold; when the processing validity of the power grid monitoring data is low, compare each second structured data with each first structured data again to obtain a comparison result; and update the first knowledge graph based on the comparison result.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the power grid monitoring data processing method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the power grid monitoring data processing method as described in any one of claims 1 to 5.

Images