Semantic-based power data quality detection method and system
By using an NLP-based power data quality inspection method and leveraging the BERT model for keyword extraction and semantic recognition to generate a branch structure for power data, the problem of low reliability in power data quality inspection results is solved, and more accurate data quality inspection is achieved.
Patent Information
- Application Number
- CN202411000631.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-07-25
AI Technical Summary
Existing technologies lack sufficient textual understanding of power data, resulting in low reliability of power data quality testing results.
A power data quality inspection method based on NLP technology is adopted. By acquiring the original power database, calling the BERT model to extract keywords and recognize semantics, generating semantic meta-branches, establishing the power data branch structure, monitoring the data missing rate and equivalence, and outputting the data quality inspection results.
By delving deeper into the semantic relationships of power failure events, the accuracy of data quality detection has been improved.
Smart Images

Figure CN118568084B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data processing, and in particular to a semantic-based power data quality detection method and system. Background Art
[0002] In the power system, since various equipment, lines and control systems are involved, when a power failure occurs, the failure may cause equipment damage, power outages or even accidents. Therefore, timely and accurate identification and processing of fault data is crucial to ensure the safety and stable operation of the power grid. The quality of power data has a key impact on the accurate identification and processing of faults.
[0003] Currently, when conducting quality inspections on power data, insufficient understanding of the text leads to inaccurate identification of missing levels in the power data, resulting in low reliability of inspection results. Summary of the invention
[0004] The present application provides a semantic-based power data quality detection method and system, which are used to solve the technical problem in the prior art that the reliability of the detection results is low due to insufficient understanding of the text when performing power data quality detection.
[0005] In view of the above problems, the present application provides a semantic-based power data quality detection method and system.
[0006] In a first aspect of the present application, a semantic-based power data quality detection method is provided, the method comprising:
[0007] Acquire an original power database, wherein the original power database includes data of power failure events;
[0008] Using NLP technology to extract keywords from each power failure event in the original power database, and output multiple key elements of the event;
[0009] Performing semantic recognition according to the multiple event key elements, determining the semantic order of the multiple event key elements, connecting the multiple event key elements according to the semantic order, and generating multiple semantic meta-branches;
[0010] Establishing a power data branch structure with the multiple semantic meta-branches;
[0011] Monitoring the power data branch structure, and outputting a data missing rate of a single semantic meta-branch among the multiple semantic meta-branches and data equivalence between the multiple semantic meta-branches;
[0012] Data quality detection is performed based on the data missing rate of the single semantic meta-branch and the data equivalence between multiple semantic meta-branches, and the data quality detection result is output.
[0013] A second aspect of the present application provides a semantic-based power data quality detection system, the system comprising:
[0014] A power database acquisition module, used to acquire an original power database, wherein the original power database includes data of power failure events;
[0015] A key element output module is used to extract keywords from each power failure event in the original power database by calling NLP technology, and output multiple key elements of the event;
[0016] A semantic meta-branch generation module, used to perform semantic recognition according to the multiple event key elements, determine the semantic order of the multiple event key elements, connect the multiple event key elements according to the semantic order, and generate multiple semantic meta-branches;
[0017] A branch structure building module, used to establish a power data branch structure with the multiple semantic element branches;
[0018] A data equivalence output module, used to monitor the power data branch structure, and output the data missing rate of a single semantic meta-branch among the multiple semantic meta-branches and the data equivalence between the multiple semantic meta-branches;
[0019] The quality detection result output module is used to perform data quality detection based on the data missing rate of the single semantic meta-branch and the data equivalence between multiple semantic meta-branches, and output the data quality detection result.
[0020] One or more technical solutions provided in this application have at least the following technical effects or advantages:
[0021] This application obtains the original power database, wherein the original power database includes the data of power failure events, and then uses NLP technology to extract keywords from each power failure event in the original power database, outputs multiple event key elements, and then performs semantic recognition according to multiple event key elements, determines the semantic order of multiple event key elements, connects multiple event key elements according to the semantic order, generates multiple semantic meta-branches, and then establishes a power data branch structure with multiple semantic meta-branches, and outputs the data missing rate of a single semantic meta-branch in multiple semantic meta-branches and the data equivalence between multiple semantic meta-branches by monitoring the power data branch structure, and performs data quality detection according to the data missing rate of a single semantic meta-branch and the data equivalence between multiple semantic meta-branches, and outputs the data quality detection result. The application achieves the technical effect of deeply excavating the semantic relationship of each event key element in the power failure event and improving the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0023] Figure 1 A flowchart of a semantic-based power data quality detection method provided in an embodiment of the present application;
[0024] Figure 2 A schematic diagram of the structure of a semantic-based power data quality detection system provided in an embodiment of the present application.
[0025] Explanation of the accompanying drawings: power database acquisition module 11, key element output module 12, semantic meta-branch generation module 13, branch structure construction module 14, data equivalence output module 15, quality detection result output module 16. DETAILED DESCRIPTION
[0026] The present application provides a semantic-based power data quality detection method and system, which is used to solve the technical problem in the prior art that the reliability of the detection result is low due to insufficient understanding of the text when performing power data quality detection.
[0027] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0028] It should be noted that the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or modules that are not explicitly listed or inherent to these processes, methods, products or devices.
[0029] Embodiment 1
[0030] like Figure 1 As shown, the present application provides a semantic-based power data quality detection method, wherein the method comprises:
[0031] S100: Acquire an original power database, wherein the original power database includes data of power failure events;
[0032] In an embodiment of the present application, the original power database is a database that needs to be tested for power data quality, and is used to store data within the power system, including data on power failure events, wherein the data on power failure events include repair data, fault type, faulty equipment, maintenance records, and other data corresponding to the power failure event. During the operation of the power system, when repair data appears, the repair data is sent to the fault maintenance department, which then processes the repair information, performs power failure analysis at the repair location, and repairs the power failure. A maintenance record form is filled in according to the maintenance process, and the data in the maintenance record form is uploaded to the original power database. The original power database is obtained to provide data support for the subsequent semantic analysis of power data quality.
[0033] S200: Using NLP technology to extract keywords from each power failure event in the original power database, and output multiple event key elements;
[0034] Further, NLP technology is used to extract keywords from each power failure event in the original power database. Step S200 of the embodiment of the present application further includes:
[0035] Use NLP technology to split the word vectors of power failure event samples and output word vector samples;
[0036] Training the word vector samples to generate a BERT model, the BERT model comprising a masked language network layer and a continuous recognition network layer, wherein the masked language network layer is used to perform prediction training on the word vector samples, and the continuous recognition network layer is used to predict whether two word vectors are continuous texts;
[0037] Keywords are extracted from each power failure event in the original power database according to the BERT model.
[0038] Furthermore, the training data of the masked language network layer predicts a preset proportion of word vector samples and outputs the predicted word vector;
[0039] The preset proportion of word vector samples includes mask replacement, random word replacement and no processing, wherein the processing proportion of mask replacement is greater than the processing proportion of random word replacement and no processing;
[0040] When the predicted word vector accuracy reaches the preset accuracy, the masked language network layer is converged and output.
[0041] In a possible embodiment, the NLP (Natural Language Processing) technology is a technology that uses artificial intelligence to enable a computer to understand, interpret, operate and generate natural language text. By using the NLP technology to extract keywords from the data corresponding to each power failure event in the original power data, the keywords that are more important in describing each power failure event are determined, thereby obtaining the multiple event key elements.
[0042] Optionally, the multiple event key elements are used to identify the more important points in the entire process of a power failure event, including event type, occurrence time, multiple faulty devices and other elements.
[0043] In a possible embodiment, the power failure event samples are multiple standard samples when the power failure event is fully recorded after the power failure occurs. The power failure event samples are subjected to text preprocessing, and the text preprocessing includes operations such as removing special characters, converting to lowercase, and word segmentation. The power failure event samples are thereby converted into a format that can be understood and processed by a computer. Furthermore, a word vector model is used to convert the multiple words after word segmentation into dense vectors, wherein words with similar meanings are closer in the vector space. Optionally, the word vector model includes Word2Vec, GloVe, FastText, etc.
[0044] Then, by adding the obtained dense vectors, the vector representation of the entire sentence is obtained, and then, the NLP technology is called to split the obtained sentence vector into word-level word vectors, thereby obtaining the word vector sample. The word vector sample is used as training data to generate the BERT model. Among them, the BERT (Bidirectional Encoder Representations from Transformer) model is a pre-trained language representation model.
[0045] The BERT model includes a masked language network layer and a continuous recognition network layer, wherein the masked language network layer is used to perform prediction training on the word vector samples, and the continuous recognition network layer is used to predict whether two word vectors are continuous texts. In other words, the masked language network layer learns the word vector samples, and then uses the masked language network layer after learning to configure the training data (network parameters of the masked language network layer) to predict the word vectors that are masked or replaced in the word vector samples, thereby learning the word vector sequence distribution in the word vector samples.
[0046] Optionally, the masked language network layer has two sublayers, namely a self-attention mechanism and a feedforward neural network. The self-attention mechanism enables the masked language network layer to focus on the information entropy of other word vectors in the sequence when processing each word vector in the input word vector sample, and calculate the correlation between each word vector and the position of other word vectors, thereby learning the relationship between different word vectors in the word vector sample. The feedforward neural network includes two fully connected layers. After the self-attention mechanism, the hidden identifier of each word vector will be sent to two fully connected layers in a feedforward neural network for processing, so that the sequence distribution of the word vector is learned.
[0047] Then, the word vector samples are subjected to mask replacement and random word replacement to obtain word vector samples of a preset ratio. Exemplarily, the word vector samples of the preset ratio include 200 word vector samples, wherein a number of word vectors in the word vector samples are randomly selected for masking, and the number of masked word vectors obtained after replacing the word vectors with special tags is 120, and the processing ratio is 3 / 5; a number of word vectors in the word vector samples are randomly selected for random word replacement, and the number of word vectors obtained after replacing the word vectors with other random word vectors is 40, and the processing ratio is 1 / 5; the number of word vectors not processed is 40, and the processing ratio is 1 / 5. The processing ratio of mask replacement is greater than the processing ratio of random word replacement and non-processing.
[0048] Furthermore, the learned masked language network layer is used to predict the word vectors that are masked and replaced by random words in the preset proportion of word vector samples, and the ratio of the number of successfully predicted word vectors to the number of word vectors that are masked and replaced by random words is calculated to obtain the accuracy of the predicted word vectors.
[0049] The predicted word vector accuracy is compared with the preset accuracy (85%, 95%, etc.) set by the technicians in this field. When the predicted word vector accuracy is less than or equal to the preset accuracy, the network parameters of the masked language network layer are updated and adjusted, and then the updated and adjusted masked language network layer predicts a preset proportion of word vector samples until the predicted word vector accuracy is greater than the preset accuracy. The model converges and outputs the masked language network layer.
[0050] In one embodiment, two word vectors of continuous text in the word vector sample are marked to obtain a continuously marked word vector sample, and then the continuously marked word vector sample is used as construction data. The continuous recognition network layer learns the continuously marked word vectors in the continuously marked word vector sample.
[0051] Furthermore, several word vectors in the word vector sample are input into the continuous recognition network layer in sequence, and the continuous recognition network layer is used to recognize the continuous relationship between the input several word vectors, and the recognition result is output. The ratio of successful matching between the recognition result and the continuously identified word vector sample is counted as the recognition accuracy of the continuous recognition network layer. When the recognition accuracy meets the preset recognition accuracy, the model converges and outputs the trained continuous recognition network layer.
[0052] The mask language network layer and the continuous recognition network layer constitute the BERT model. The BERT model is used to perform keyword vector recognition on the data corresponding to each power failure event in the original power database, and it is determined whether there is a continuous text between two keyword vectors, so as to obtain the multiple event key elements.
[0053] Since the BERT model is built based on the power failure event samples and has strong text understanding and reasoning performance, and has a deep level of learning of the semantic relationship between word vectors, even if there are differences between the data of power failure events in the original power database and the standard samples, keywords can be extracted for each power failure event to determine multiple event key elements corresponding to each power failure event.
[0054] S300: performing semantic recognition according to the multiple event key elements, determining the semantic order of the multiple event key elements, connecting the multiple event key elements according to the semantic order, and generating multiple semantic meta-branches;
[0055] In one embodiment, the semantic order of the multiple event key elements is determined by distributing the word vectors of the multiple event key elements in the semantic understanding according to the sequence order, and then the multiple event key elements are sequentially connected based on the semantic order to obtain the multiple semantic meta-branches. The multiple semantic meta-branches are respectively used to store the data corresponding to the multiple event key elements, and each semantic meta-branch corresponds to one event key element.
[0056] S400: Establishing a power data branch structure using the multiple semantic element branches;
[0057] Furthermore, step S400 in the embodiment of the present application further includes:
[0058] Establishing a synonymous power database of the original power database, wherein the synonymous power database is obtained through a synonymous conversion module;
[0059] The near-sense conversion module performs near-sense conversion on the multiple event key elements, and outputs multiple near-sense key elements of each event key element, wherein the multiple event key elements and the multiple near-sense key elements are in a many-to-many mapping relationship;
[0060] The data content of each semantic element branch in the power data branch structure is optimized according to the synonymous power database to obtain an optimized power data branch structure.
[0061] Furthermore, the synonym conversion module performs synonym conversion on the multiple event key elements, and step S400 of the embodiment of the present application further includes:
[0062] Determining a semantic order of the plurality of event key elements;
[0063] Performing a synonymous conversion on the first event key element according to the synonymous conversion module, and outputting a plurality of synonymous key elements corresponding to the first event key element;
[0064] Performing context prediction based on the multiple proximate key elements corresponding to the first event key element, generating multiple predicted proximate key elements corresponding to the remaining event key elements respectively;
[0065] The multiple predicted proximate key elements corresponding to the remaining event key elements are used as the initial proximate power database.
[0066] Furthermore, after obtaining the initial synonymous power database, step S400 of the embodiment of the present application further includes:
[0067] Performing a synonymous conversion on the second event key element according to the synonymous conversion module, and outputting a plurality of synonymous key elements corresponding to the second event key element;
[0068] Optimize the multiple proximate key elements corresponding to the second event key element and the multiple predicted proximate key elements corresponding to the second event key element in the initial proximate power database, output the multiple optimized proximate key elements corresponding to the second event key element, and so on, output the multiple optimized proximate key elements that correspond one to one to the multiple event key elements.
[0069] In a possible embodiment, the power data branch structure is generated by connecting the plurality of semantic element branches in a sequential order, wherein the power data branch structure is used to perform structured storage of data corresponding to a power failure event in the original power database.
[0070] In an embodiment of the present application, in order to expand the adaptability of the power data branch structure to similar power events, the original power database is subjected to event derivation to obtain the synonymous power database. The synonymous power database is obtained by performing synonymous conversion on multiple key event elements corresponding to the power failure event in the original power database.
[0071] Optionally, the synonym conversion module is a functional module for performing synonym analysis and expansion on event key elements. By using the synonym conversion module to perform synonym conversion on the multiple event key elements respectively, multiple synonymous key elements of each event key element are obtained. Among them, the multiple event key elements and the multiple synonymous key elements are in a many-to-many mapping relationship. In other words, one event key element corresponds to multiple synonymous key elements.
[0072] Furthermore, based on the multiple synonymous key elements in the synonymous power database, and the many-to-many mapping relationship between the multiple event key elements and the multiple synonymous key elements, the multiple synonymous key elements are used to supplement the data of the semantic meta-branches of the corresponding event key elements in the mapping relationship, and after supplementing and optimizing the data content in the multiple semantic meta-branches, an optimized power data branch structure is obtained.
[0073] In one embodiment, the semantic order of the multiple event key elements is determined, and optionally, the semantic order is the time order of occurrence of the multiple event key elements. Then, the near-sense conversion module is used to perform near-sense conversion on the first event key element, and multiple near-sense key elements corresponding to the first event key element are output. Context prediction is performed based on the multiple near-sense key elements corresponding to the first event key element.
[0074] Optionally, multiple key element sequences are obtained from the original power database in historical time, each key element sequence is the failure sequence of multiple faulty devices when a power failure event occurs. The multiple key element sequences are used as training data to perform supervised training on a prediction module built based on a convolutional neural network model, and the prediction module is used to learn the context sequence of key elements in the multiple key element sequences until the model output reaches convergence, thereby obtaining the trained prediction module.
[0075] Then, the multiple synonymous key elements corresponding to the multiple first event key elements are input into the prediction module respectively, and multiple predicted synonymous key element sets of the context positions of the multiple synonymous key elements are obtained. Then, the predicted synonymous key elements of the corresponding positions in the multiple predicted synonymous key element sets are extracted with the positions of the remaining event key elements in the semantic order as indexes, so as to obtain multiple predicted synonymous key elements corresponding to the remaining event key elements. The multiple synonymous key elements corresponding to the first event key element and the multiple predicted synonymous key elements corresponding to the remaining event key elements are used as the initial synonymous power database.
[0076] Based on the same principle of obtaining multiple synonymous key elements corresponding to the first event key element, the synonymous conversion module is used to perform synonymous conversion on the second event key element, and multiple synonymous key elements corresponding to the second event key element are output. According to the multiple synonymous key elements corresponding to the second event key element, the multiple predicted synonymous key elements corresponding to the second event key element in the initial synonymous power database are unioned to obtain multiple optimized synonymous key elements corresponding to the second event key element, and so on, multiple optimized synonymous key elements corresponding to the multiple event key elements are output one by one. The goal of expanding and optimizing the initial synonymous power database is achieved by combining the multiple predicted synonymous key elements corresponding to the multiple event key elements predicted by the first event key element and the multiple synonymous key elements after the multiple event key elements are subjected to synonymous conversion.
[0077] S500: monitoring the power data branch structure, and outputting a data missing rate of a single semantic meta-branch among the multiple semantic meta-branches and data equivalence between the multiple semantic meta-branches;
[0078] S600: Perform data quality detection based on the data missing rate of the single semantic meta-branch and the data equivalence between multiple semantic meta-branches, and output the data quality detection result.
[0079] In one possible embodiment, data monitoring is performed on the power data branch structure. Optionally, the data missing values in each semantic meta-branch are counted by using a statistical function (such as count()), and the statistical result is compared with the total amount of data in each semantic meta-branch, so as to obtain the data missing rate of a single semantic meta-branch among the multiple semantic meta-branches. The data missing rate reflects the degree of missing records of key elements of events in a single semantic meta-branch. The lower the data missing rate, the better the quality of the data stored in the corresponding single semantic meta-branch.
[0080] By performing variance calculation on the data missing rates of multiple semantic meta-branches, the consistency of data missing between different semantic meta-branches is determined, and the inverse of the variance calculation result is used as the data equivalence between the multiple semantic meta-branches. The data equivalence reflects the consistency of data missing between multiple semantic meta-branches. The higher the data equivalence, the higher the consistency of data missing between multiple semantic meta-branches and the better the data quality.
[0081] Optionally, the data missing rates of the multiple semantic meta-branches are weightedly calculated according to the weights pre-set by a person skilled in the art to obtain a first data quality detection result, and the data equivalence between the multiple semantic meta-branches is used as a second data quality detection result. The first data quality detection result and the second data quality detection result are used as the data quality detection result. The data quality detection result reflects the quality of the power data stored in the original power database.
[0082] Furthermore, step S100 in the embodiment of the present application further includes:
[0083] Classifying the power failure events and outputting multiple types of power failure events;
[0084] According to each type of power failure event, multiple power data branch structures are established;
[0085] Then, data quality detection is performed on the multiple power data branch structures respectively, multiple data quality detection results are output, and the multiple data quality detection results are integrated and output.
[0086] In a possible embodiment, the fault type of each power fault event in the original power database is extracted, and multiple power fault events corresponding to the same fault type, such as short circuit, tripping, etc., are clustered into one category, thereby obtaining the multiple categories of power fault events. The goal of clustering the power fault events according to their types is achieved.
[0087] Then, based on each type of power failure event, based on the same construction principle as the power data branch structure, multiple power data branch structures corresponding to multiple types of power failure events are obtained. Then, data quality detection is performed on the data missing rate of a single semantic element branch and the data equivalence between multiple semantic element branches in the multiple power data branch structures, thereby obtaining multiple data quality detection results. By performing mean calculation on the multiple data quality detection results, the goal of fusing the multiple data quality detection results is achieved, and the data quality detection result of the original power database as a whole is obtained.
[0088] In summary, the embodiments of the present application have at least the following technical effects:
[0089] This application obtains the original power database, calls NLP technology to extract keywords from each power failure event in the original power database, outputs multiple event key elements, and then determines the semantic order of multiple event key elements, generates multiple semantic meta-branches, and establishes a power data branch structure to achieve the goal of structured processing of data in the original power database, monitor the power data branch structure, output the data missing rate of a single semantic meta-branch in multiple semantic meta-branches and the data equivalence between multiple semantic meta-branches, and achieve the goal of detecting the data quality of the power data branch structure from two dimensions. According to the data missing rate of the single semantic meta-branch and the data equivalence between multiple semantic meta-branches, data quality detection is performed, and data quality detection results are output. The technical effect of deeply excavating the semantic relationship of each key element of the event in the power failure event and improving the detection accuracy is achieved.
[0090] Embodiment 2
[0091] Based on the same inventive concept as the semantic-based power data quality detection method in the aforementioned embodiment, Figure 2 As shown, the present application provides a semantic-based power data quality detection system, and the system and method embodiments in the present application are based on the same inventive concept. The system includes:
[0092] The power database acquisition module 11 is used to acquire an original power database, wherein the original power database includes data of power failure events;
[0093] A key element output module 12 is used to extract keywords from each power failure event in the original power database by calling NLP technology, and output multiple event key elements;
[0094] A semantic meta-branch generating module 13, configured to perform semantic recognition according to the plurality of event key elements, determine the semantic order of the plurality of event key elements, connect the plurality of event key elements according to the semantic order, and generate a plurality of semantic meta-branches;
[0095] A branch structure building module 14, used to establish a power data branch structure with the multiple semantic element branches;
[0096] A data equivalence output module 15, configured to monitor the power data branch structure, and output a data missing rate of a single semantic meta-branch among the multiple semantic meta-branches and data equivalence between the multiple semantic meta-branches;
[0097] The quality detection result output module 16 is used to perform data quality detection based on the data missing rate of the single semantic meta-branch and the data equivalence between multiple semantic meta-branches, and output the data quality detection result.
[0098] Furthermore, the branch structure building module 14 is used to perform the following steps:
[0099] Establishing a synonymous power database of the original power database, wherein the synonymous power database is obtained through a synonymous conversion module;
[0100] The near-sense conversion module performs near-sense conversion on the multiple event key elements, and outputs multiple near-sense key elements of each event key element, wherein the multiple event key elements and the multiple near-sense key elements are in a many-to-many mapping relationship;
[0101] The data content of each semantic element branch in the power data branch structure is optimized according to the synonymous power database to obtain an optimized power data branch structure.
[0102] Furthermore, the branch structure building module 14 is used to perform the following steps:
[0103] Determining a semantic order of the plurality of event key elements;
[0104] Performing a synonymous conversion on the first event key element according to the synonymous conversion module, and outputting a plurality of synonymous key elements corresponding to the first event key element;
[0105] Performing context prediction based on the multiple proximate key elements corresponding to the first event key element, generating multiple predicted proximate key elements corresponding to the remaining event key elements respectively;
[0106] The multiple predicted proximate key elements corresponding to the remaining event key elements are used as the initial proximate power database.
[0107] Furthermore, the branch structure building module 14 is used to perform the following steps:
[0108] Performing a synonymous conversion on the second event key element according to the synonymous conversion module, and outputting a plurality of synonymous key elements corresponding to the second event key element;
[0109] Optimize the multiple proximate key elements corresponding to the second event key element and the multiple predicted proximate key elements corresponding to the second event key element in the initial proximate power database, output the multiple optimized proximate key elements corresponding to the second event key element, and so on, output the multiple optimized proximate key elements that correspond one to one to the multiple event key elements.
[0110] Furthermore, the key element output module 12 is used to perform the following steps:
[0111] Use NLP technology to split the word vectors of power failure event samples and output word vector samples;
[0112] Training the word vector samples to generate a BERT model, the BERT model comprising a masked language network layer and a continuous recognition network layer, wherein the masked language network layer is used to perform prediction training on the word vector samples, and the continuous recognition network layer is used to predict whether two word vectors are continuous texts;
[0113] Keywords are extracted from each power failure event in the original power database according to the BERT model.
[0114] Furthermore, the training data of the masked language network layer predicts a preset proportion of word vector samples and outputs the predicted word vector;
[0115] The preset proportion of word vector samples includes mask replacement, random word replacement and no processing, wherein the processing proportion of mask replacement is greater than the processing proportion of random word replacement and no processing;
[0116] When the predicted word vector accuracy reaches the preset accuracy, the masked language network layer is converged and output.
[0117] Furthermore, the power database acquisition module 11 is used to perform the following steps:
[0118] Classifying the power failure events and outputting multiple types of power failure events;
[0119] According to each type of power failure event, multiple power data branch structures are established;
[0120] Then, data quality detection is performed on the multiple power data branch structures respectively, multiple data quality detection results are output, and the multiple data quality detection results are integrated and output.
[0121] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. And the above-mentioned specific embodiments of this specification are described. Other embodiments are within the scope of the attached claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0122] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.
[0123] This specification and drawings are merely exemplary illustrations of the present application and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, a person skilled in the art may make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application intends to include these modifications and variations.
Claims
1. A semantic-based power data quality detection method, characterized in that: The method comprises: Acquire an original power database, wherein the original power database includes data of power failure events; Using NLP technology to extract keywords from each power failure event in the original power database, and output multiple key elements of the event; Performing semantic recognition according to the multiple event key elements, determining the semantic order of the multiple event key elements, connecting the multiple event key elements according to the semantic order, and generating multiple semantic meta-branches; Establishing a power data branch structure with the multiple semantic meta-branches; Monitoring the power data branch structure, and outputting a data missing rate of a single semantic meta-branch among the multiple semantic meta-branches and data equivalence between the multiple semantic meta-branches; Performing data quality detection based on the data missing rate of the single semantic meta-branch and the data equivalence between the multiple semantic meta-branches, and outputting the data quality detection result; Among them, a statistical function is used to count the data missing values in each semantic meta-branch, and the statistical result is compared with the total data amount in each semantic meta-branch to obtain the data missing rate of a single semantic meta-branch among the multiple semantic meta-branches; the variance of the data missing rates of multiple semantic meta-branches is calculated to determine the consistency of data missing between different semantic meta-branches, and the inverse of the variance calculation result is used as the data equivalence between the multiple semantic meta-branches.
2. The method according to claim 1, characterized in that The method further comprises: Establishing a synonymous power database of the original power database, wherein the synonymous power database is obtained through a synonymous conversion module; The near-sense conversion module performs near-sense conversion on the multiple event key elements, and outputs multiple near-sense key elements of each event key element, wherein the multiple event key elements and the multiple near-sense key elements are in a many-to-many mapping relationship; The data content of each semantic element branch in the power data branch structure is optimized according to the synonymous power database to obtain an optimized power data branch structure.
3. The method according to claim 2, characterized in that The near meaning conversion module performs near meaning conversion on the multiple event key elements, including: Determining a semantic order of the plurality of event key elements; Performing a synonymous conversion on the first event key element according to the synonymous conversion module, and outputting a plurality of synonymous key elements corresponding to the first event key element; Performing context prediction based on the multiple proximate key elements corresponding to the first event key element, generating multiple predicted proximate key elements corresponding to the remaining event key elements respectively; The multiple predicted proximate key elements corresponding to the remaining event key elements are used as the initial proximate power database.
4. The method according to claim 3, characterized in that After obtaining the initial proximate power database, the method further includes: Performing a synonymous conversion on the second event key element according to the synonymous conversion module, and outputting a plurality of synonymous key elements corresponding to the second event key element; Optimize the multiple proximate key elements corresponding to the second event key element and the multiple predicted proximate key elements corresponding to the second event key element in the initial proximate power database, output the multiple optimized proximate key elements corresponding to the second event key element, and so on, output the multiple optimized proximate key elements that correspond one to one to the multiple event key elements.
5. The method according to claim 1, characterized in that Calling NLP technology to extract keywords from each power failure event in the original power database, the method includes: Use NLP technology to split the word vectors of power failure event samples and output word vector samples; Training the word vector samples to generate a BERT model, the BERT model comprising a masked language network layer and a continuous recognition network layer, wherein the masked language network layer is used to perform prediction training on the word vector samples, and the continuous recognition network layer is used to predict whether two word vectors are continuous texts; Keywords are extracted from each power failure event in the original power database according to the BERT model.
6. The method according to claim 5, characterized in that The training data of the masked language network layer predicts a preset proportion of word vector samples and outputs the predicted word vector; The preset proportion of word vector samples includes mask replacement, random word replacement and no processing, wherein the processing proportion of mask replacement is greater than the processing proportion of random word replacement and no processing; When the predicted word vector accuracy reaches the preset accuracy, the masked language network layer is converged and output.
7. The method according to claim 1, characterized in that The method further comprises: Classifying the power failure events and outputting multiple types of power failure events; According to each type of power failure event, multiple power data branch structures are established; Then, data quality detection is performed on the multiple power data branch structures respectively, multiple data quality detection results are output, and the multiple data quality detection results are integrated and output.
8. The semantic-based power data quality detection system is characterized by: The system comprises: A power database acquisition module, used to acquire an original power database, wherein the original power database includes data of power failure events; A key element output module is used to extract keywords from each power failure event in the original power database by calling NLP technology, and output multiple key elements of the event; A semantic meta-branch generation module, used to perform semantic recognition according to the multiple event key elements, determine the semantic order of the multiple event key elements, connect the multiple event key elements according to the semantic order, and generate multiple semantic meta-branches; A branch structure building module, used to establish a power data branch structure with the multiple semantic element branches; A data equivalence output module, used to monitor the power data branch structure, and output the data missing rate of a single semantic meta-branch among the multiple semantic meta-branches and the data equivalence between the multiple semantic meta-branches; A quality detection result output module is used to perform data quality detection based on the data missing rate of the single semantic meta-branch and the data equivalence between multiple semantic meta-branches, and output the data quality detection result; Among them, a statistical function is used to count the data missing values in each semantic meta-branch, and the statistical result is compared with the total data amount in each semantic meta-branch to obtain the data missing rate of a single semantic meta-branch among the multiple semantic meta-branches; the variance of the data missing rates of multiple semantic meta-branches is calculated to determine the consistency of data missing between different semantic meta-branches, and the inverse of the variance calculation result is used as the data equivalence between the multiple semantic meta-branches.
