Point table data checking method and system based on knowledge vector library and large model

Abstract

The invention provides a point table data checking method and system based on a knowledge vector library and a large model, and relates to the technical field of point table data checking, and the method comprises the steps: constructing a knowledge vector library storing a plurality of historical point table data standardized vectors, and adding a checking rule for each standardized vector based on domain knowledge; performing structured analysis on the to-be-processed point table data to obtain a target field and an unstructured field, performing word segmentation processing and entity recognition on the unstructured field by using a preset BERT-CRF model to obtain a processing result, integrating the target field and the processing result into a multi-tuple, and generating a target vector according to the multi-tuple; and calculating the similarity between the target vector and each standardized vector in sequence, recording the checking rule corresponding to the similarity greater than a preset threshold value as a target rule, calling the target rule to check the to-be-processed point table data and outputting a result, and the application can process unstructured data in the point table data.

Description

Technical Field

[0001] The present application relates to the technical field of verifying point table data, and in particular to a point table data verification method and system based on a knowledge vector library and a large model. Background Art

[0002] In the operation and management of State Grid's power systems, point-meter data carries a vast amount of information about power grid equipment. The standardization of its naming directly impacts the accuracy and efficiency of power system dispatching, operation and maintenance, and troubleshooting. State Grid's standardized naming system for typical information sets strict requirements for the integrity, formality, and use of numerals and Chinese characters for names of equipment such as lines and switches. This ensures data consistency and recognizability throughout the entire business process, preventing misinterpretation and decision-making errors caused by confusing naming. However, in practice, point table data often exhibits incomplete line and switch names, uses informal abbreviations, and mixes numbers and Chinese characters with serial numbers, inconsistent with the typical State Grid information naming standards. Traditional processing methods often rely on manual verification and correction against standards, which is not only labor-intensive but also prone to omissions and misjudgments due to the numerous details of the standards and differences in human understanding. While some automated verification tools can handle simple rules, they struggle to accurately identify and correct naming irregularities with complex semantics.

[0003] The Chinese invention patent with application publication number CN115168688A provides a method for automatic adaptation of device point tables based on absorption learning. The patent combines the device type and the values of each physical quantity in the point table into training samples and generates a training model, and establishes a mapping relationship between the device type and the point table, so that the device type can be matched according to the feature vector composed of physical quantities in the message, and then the device point table can be reversely matched.

[0004] However, the above patents mainly focus on the processing of structured data, and their ability to identify and process unstructured data is relatively limited. Summary of the Invention

[0005] In order to identify and process unstructured data in point table data, the present application provides a point table data verification method and system based on a knowledge vector library and a large model.

[0006] In the first aspect, this application provides a point table data verification method based on a knowledge vector library and a large model, using the following technical solutions: The point table data verification method based on the knowledge vector library and the large model includes the following steps: Database construction: Build a knowledge vector database that stores standardized vectors of multiple historical point table data. Based on domain knowledge, formulate verification rules for each historical point table data and associate the verification rules with the corresponding standardized vectors. Data processing: Perform structured analysis on the point table data to be processed to obtain target fields and unstructured fields. Use the preset BERT-CRF model to parse the unstructured fields to obtain the processing results of the unstructured fields. Combine the target fields and processing results into a tuple, and generate a target vector based on the tuple. Calculate similarity: Calculate the similarity between the target vector and each standardized vector in turn, record the corresponding calibration rule when the similarity is greater than the preset similarity threshold as the target rule, call the target rule to calibrate the point table data to be processed, and output the calibration result.

[0007] This application integrates and standardizes the scattered historical point table data by constructing a knowledge vector library that stores standardized vectors of multiple historical point table data. This application realizes the comparison and calculation of historical data from different sources and formats through standardized vectors. Subsequently, this application formulates verification rules for each historical point table data based on domain knowledge, combining domain expertise with point table data. Subsequently, this application performs structured parsing on the point table data to be processed to obtain target fields and unstructured fields. Structured parsing can adopt different processing methods for different types of fields to improve the efficiency and pertinence of data processing. Subsequently, this application uses a preset BERT-CRF model to parse the unstructured fields and obtain the processing results of the unstructured fields. The BERT-CRF model combines the semantic understanding ability of BERT and the sequence labeling advantages of CRF, and can effectively extract valuable information from unstructured text and convert it into structured entities and relationships.

[0008] Subsequently, the present application integrates the target field and the processing result into a tuple, and generates a target vector based on the tuple. The generation of the tuple organically combines different types of data, and the construction of the target vector provides a unified data representation for subsequent similarity calculations. By converting the data into vector form, the present application can use these vectors to efficiently perform similarity calculations and comparisons. Subsequently, the present application calculates the similarity between the target vector and each standardized vector in turn, and records the verification rule when the similarity is greater than the preset similarity threshold as the target rule. The similarity calculation can measure the degree of similarity between the data to be processed and the historical data. By setting the similarity threshold, the historical point table data and its corresponding verification rules that are most relevant to the data to be processed can be screened out. By adopting the above scheme, the present application can automatically and quickly determine the verification rules applicable to the current data, reducing the tediousness and subjectivity of manual rule selection.

[0009] Subsequently, this application applies the target rules to the pending point table data and outputs the verification results. Using the selected target rules to verify the pending data ensures that the verification process complies with the standards and requirements of the field, improving the accuracy and consistency of the data. The output verification results provide a reliable basis for subsequent data use and analysis, helping to promptly detect and correct errors and anomalies in the data.

[0010] Optionally, the method further includes: Correction: When the similarity is not greater than a preset similarity threshold, the abnormal fields in the point table data to be processed are extracted. If the abnormal fields are data categories, the abnormal fields are mapped to structured rules based on domain knowledge, and a correction strategy for the structured rules is obtained based on the verification rules in the knowledge vector library. The abnormal fields in the point table data to be processed are modified using the correction strategy to obtain the modified point table data to be processed, and the data processing step is performed; In the data processing step, the modified point table data to be processed is subjected to structured analysis.

[0011] When the similarity is not greater than the preset threshold, this application will extract the abnormal fields in the point table data to be processed, and further process these abnormal fields (especially the abnormal fields of the data class). Specifically, this application can map the abnormal fields into structured rules based on domain knowledge, and obtain the correction strategy from the verification rules in the knowledge vector library. By adopting the above scheme, this application can combine professional field knowledge and existing verification rules to make the correction strategy more scientific and reasonable. This application uses the verification rules in the knowledge vector library to automatically obtain the correction strategy, and uses the strategy to modify the abnormal fields in the point table data to be processed, thereby realizing the automated processing of abnormal fields. Automated processing can not only improve processing speed, but also reduce labor costs and reduce the impact of human factors on data processing results.

[0012] Optionally, the correction step further includes: If the abnormal field is not a data category, the preset BERT-CRF model is used to parse the abnormal field to obtain the processing result of the abnormal field. The API of the large model is called to input the processing result of the abnormal field into the large model to obtain the inference result of the large model on the abnormal field. The inference result is used to replace the abnormal field in the point table data to be processed, and new point table data to be processed is obtained. Then, the data processing steps are performed; In the data processing step, the new point table data to be processed is subjected to structured analysis.

[0013] For abnormal fields that are not part of the data category, this application uses a preset BERT-CRF model to parse them, obtain the processing results of the abnormal fields, and then input the processing results of the abnormal fields into the large model to obtain the inference results of the abnormal fields. The large model has a massive data training foundation and a complex network structure. It can comprehensively consider multiple factors and contextual relationships to conduct more in-depth analysis and reasoning on abnormal fields. The large model can combine broader knowledge and complex logical relationships to generate more accurate and practical correction suggestions, further improving the accuracy of processing abnormal fields that are not part of the data category.

[0014] This application distinguishes between abnormal fields of data category and non-data category, and adopts different processing methods. For abnormal fields of non-data category, they are processed through a combination of BERT-CRF model and large model, so that this application can adapt to various types of unstructured or semi-structured abnormal data, such as text descriptions, free-format fields, etc. The large model has strong generalization ability and can handle unseen abnormal situations. When encountering new and complex non-data category abnormal fields, the large model can give reasonable reasoning results based on its extensive knowledge and training experience, so that this application can still maintain a good processing effect when facing constantly changing data, and enhance the adaptability of the entire data processing process.

[0015] This application improves data quality by correcting non-data category abnormal fields to obtain new point table data to be processed. High-quality data can improve the reliability of analysis results, reduce processing errors and abnormal situations caused by abnormal fields, and thus improve the standardization of point table data.

[0016] Optionally, the step of calculating the similarity further includes: When there are multiple target rules, each target rule is encoded to obtain quantum bits, and the quantum state of each target rule is set based on the quantum bits. The entanglement relationship between the target rules is simulated through the quantum state of each target rule and quantum gate operations, and the entangled quantum state is output. The entangled quantum state is projected and measured to obtain the collapse result. The target rules are deleted based on the collapse result, and the remaining target rules are called to check the data of the point table to be processed.

[0017] This application encodes each target rule into a quantum bit and sets a quantum state, making full use of the characteristics of quantum bits. A quantum bit can be in a superposition of multiple states at the same time, which means that multiple target rules can be represented and processed simultaneously at the quantum level. Compared with traditional binary coding, this application can represent rule information in a more compact way, providing a larger information processing space for subsequent processing.

[0018] Subsequently, this application simulates the entangled relationship between target rules through quantum states and quantum gate operations. Entanglement is a unique resource in quantum computing. The entangled relationship makes the target rules no longer independent of each other. The state change of one rule will immediately affect the other rules entangled with it. By adopting the above scheme, this application can capture the complex and subtle connections between target rules, which helps to discover rule combination patterns that are difficult to detect with traditional methods, thereby more comprehensively understanding the internal connections between target rules and improving the ability to handle complex rule relationships.

[0019] When there are multiple target rules, traditional methods may face problems such as rule combination explosion and high computational complexity. However, this application uses quantum computing to process multiple target rules, which can more efficiently explore the interactions between rules and provide a new approach to solving complex problems involving multiple rules. For example, in the verification of power system point meter data, different rules may involve multiple aspects such as voltage, current, and equipment status and are interrelated. This application can better handle these complex rule relationships.

[0020] Subsequently, the present application performs projection measurement on the entangled quantum state to obtain the collapse result. This process can be regarded as a mechanism for extracting and screening target rule information. Projection measurement collapses the quantum state to the classical state. In this process, by analyzing the measurement results, it is possible to identify which target rules are more critical for the verification of the current point table data to be processed, and which rules may be redundant or less relevant. By deleting target rules based on the collapse results, unnecessary rules can be removed, reducing the amount of calculation and complexity in the subsequent verification process, and improving the verification efficiency.

[0021] Different point table data to be processed may have different characteristics and requirements. This application can dynamically adjust the target rule set based on the specific point table data. Through quantum computing and projection measurement, this application can automatically screen out the rules that are most suitable for the current data, making the use of rules more flexible and accurate, and adapting to the data verification needs in different scenarios. Subsequently, this application calls the remaining, filtered target rules to verify the point table data to be processed. Since redundant rules are removed, the remaining target rules are more focused on the key aspects related to the current data, which can more accurately identify problems in the data and improve the accuracy and quality of verification.

[0022] Optionally, the step of calculating the similarity further includes: The number of calls of each verification rule in the knowledge vector library is counted to obtain statistical results, the number of calls of each target rule is obtained based on the statistical results, the weight of the quantum state of each target rule is set according to the number of calls of each target rule, the superposition state of each target rule is calculated based on the quantum state of each target rule and the weight, and the superposition state is used as the new quantum state.

[0023] This application counts the number of times each verification rule in the knowledge vector library is called, and uses this as a basis to set the quantum state weight for each target rule. This process can intuitively reflect the importance and frequency of use of different target rules in practical applications. Verification rules with a high number of calls are often the more commonly used and more critical verification rules in the data verification process. By giving them higher weights, these important verification rules can play a greater role in the subsequent similarity calculation, thereby improving the accuracy and effectiveness of data verification.

[0024] In different data scenarios and business needs, the requirements for verification rules will also vary. The statistical results of the number of calls will change dynamically with actual usage, so the weights set based on the number of calls can also be adjusted accordingly. By adopting the above solution, this application can automatically optimize the use of rules according to the actual application scenario, and better meet the data verification needs in different business scenarios.

[0025] Subsequently, this application incorporates the weight set according to the number of calls into the quantum state of each target rule, calculates the superposition state of each target rule, and uses the superposition state as a new quantum state. This new quantum state not only contains the characteristic information of the rule itself, but also contains the number of times the rule is called in actual application, thereby more comprehensively characterizing the target rule. When calculating similarity, the new quantum state takes into account the weight factor of the rule, which can more reasonably measure the importance of different target rules, and thus more reasonably delete the target rules.

[0026] Optionally, the step of calculating the similarity further includes: The calling time of each verification rule in the knowledge vector library is counted, and a verification rule in the knowledge vector library is used as a node. The knowledge graph is formulated with the order of the calling time of each verification rule as the directed edge. The remaining target rules are input into the knowledge graph in sequence, and the verification order of the remaining target rules is output. The remaining target rules are then called to verify the data in the processing point table according to the verification order.

[0027] This application counts the calling time of each verification rule in the knowledge vector library and constructs a knowledge graph with the order of calling time as the directed edge, which can reflect the usage logic and sequential dependency of the verification rules in actual applications. Some rules may need to be called after other rules to achieve better results, or the calling results of certain rules will affect the applicability of subsequent rules. The knowledge graph constructed based on the calling time can make these implicit logical relationships explicit, thereby providing a basis for determining a reasonable verification order. By adopting the above scheme, this application can reduce the problem of repeated verification caused by improper rule calling order. For example, first calling the rule for preliminary screening of the data range, and then calling the rule for detailed analysis based on the screened data, you can reduce unnecessary detailed analysis of data that does not meet the range and improve verification efficiency.

[0028] Calling the remaining target rules in the verification order output by the knowledge graph to verify the point table data to be processed can make the entire verification process smoother and more efficient, reduce waiting time and waste of computing resources caused by unreasonable rule calling order, and speed up data processing.

[0029] In many business scenarios, the order in which verification rules are called often follows specific business rules and processes. By building a knowledge graph based on call time to determine the verification order, we can better meet actual business needs and make the verification process more consistent with business logic. An unreasonable order in which rules are called can cause errors to propagate and amplify during the verification process. If a rule with high data accuracy requirements is called first, and the data has not yet undergone the necessary preprocessing and screening, a large number of erroneous results may be generated, affecting the verification of subsequent rules. Performing verification in a reasonable verification order can reduce the possibility of such error propagation and improve the reliability of the verification results.

[0030] Optionally, before performing the data processing step, the method further includes: Completeness check: Set completeness indicators, including field completeness rate, field value range coverage, logical association between fields, and collection time continuity; check the target fields and unstructured field processing results in the point table data to be processed one by one, and calculate the score of each completeness indicator; Score judgment: judge whether the score of each integrity indicator is greater than the preset score threshold in turn. If so, execute the data processing step; if not, issue an alarm signal, which includes the integrity indicator below the preset score threshold.

[0031] The integrity indicators set in this application cover multiple dimensions, including field integrity rate, field value range coverage, logical correlation between fields, and collection time continuity. They can comprehensively evaluate the integrity of the point table data to be processed from different angles. The field integrity rate can confirm whether each field of the point table data to be processed has corresponding data; the field value range coverage can check whether the field values of the point table data to be processed are within a reasonable value range; the logical correlation between fields can confirm that the data between different fields of the point table data to be processed conforms to business logic and internal connections; the collection time continuity focuses on the coherence of the point table data to be processed in the time dimension, which is particularly important for time series data. Through the comprehensive evaluation of these indicators, integrity issues in the data can be discovered more accurately.

[0032] Through integrity verification, this application can determine the completeness of the point table data to be processed. Subsequently, this application sequentially determines whether the score of each integrity indicator is greater than a preset score threshold. If the score of a certain indicator is lower than the threshold, this application will issue an alarm signal and clearly indicate the integrity indicator that is below the threshold. By adopting this solution, data management personnel can quickly locate integrity issues in the data, understand where the problem lies, and then carry out targeted treatment. Compared with traditional data quality inspection methods, the use of integrity indicator score judgment and alarm signals can reduce the time and cost of problem investigation.

[0033] Optionally, before issuing the alarm signal, the method further includes: Call the big model API, input the point table data to be processed into the big model, infer the missing fields in the point table data to be processed, obtain the inferred missing fields, use the inferred missing fields to fill the point table data to be processed, obtain the filled point table data to be processed, recalculate the score of the integrity index of the filled point table data to be processed, and re-execute the score judgment step based on the score of the integrity index of the filled point table data to be processed.

[0034] After discovering that the data integrity index score did not reach the pre-examination score threshold, this application analyzed the historical point table data through a large model, and then predicted and filled the missing fields in the point table data to be processed. This application proactively solved the problem of missing data and improved the automated data processing capabilities.

[0035] Secondly, this application provides a point table data verification system based on a knowledge vector library and a large model, which adopts the following technical solutions: The point table data verification system based on the knowledge vector library and large model includes: processor and memory, The memory stores program code; When the processor calls the program code in the memory, the steps of the method described in the first aspect are executed.

[0036] In summary, this application includes at least one of the following beneficial technical effects: 1. This application uses the pre-defined BERT-CRF model to parse unstructured fields and obtain processing results for unstructured fields. The BERT-CRF model combines the semantic understanding capabilities of BERT with the sequence annotation advantages of CRF. It can effectively extract valuable information from unstructured text and convert it into structured entities and relationships.

[0037] 2. This application calls the target rules to check the data of the point table to be processed and outputs the verification results. Using the filtered target rules to check the data to be processed can make the verification process meet the standards and requirements in the field and improve the accuracy and consistency of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flow chart of Example 1 of the present application; Figure 2 This is a flowchart of Example 3 of the present application. DETAILED DESCRIPTION

[0039] The following combination Figure 1 and Figure 2 This application is described in further detail.

[0040] Example 1: This example discloses a point table data verification method based on a knowledge vector library and a large model, referring to Figure 1 The method includes: S11 database construction, S12 data processing, and S13 similarity calculation. First, a knowledge vector library is constructed to store standardized vectors of multiple historical point table data, and verification rules are added to each standardized vector based on domain knowledge; then, the point table data to be processed is structured parsed to obtain target fields and unstructured fields, and a preset BERT-CRF model is used to perform word segmentation and entity recognition on the unstructured fields to obtain processing results, and then the target field and the processing results are integrated into a multi-tuple and a target vector is generated accordingly; finally, the similarity between the target vector and each standardized vector is calculated in turn, and the verification rule corresponding to the similarity greater than a preset threshold is recorded as the target rule, and the target rule is called to verify the point table data to be processed and output the result. The execution process of this embodiment is as follows: S11 builds a database, constructing a knowledge vector database that stores standardized vectors of multiple historical point table data, formulating verification rules for each historical point table data based on domain knowledge, and associating the verification rules with the corresponding standardized vectors.

[0041] This embodiment first constructs a knowledge vector library, the main function of which is to store standardized vectors corresponding to multiple historical point table data.

[0042] For point table data from different sources, there may be differences in field naming and data types. The process of constructing standardized vectors will unify field names, standardize data types, and encode the content of historical point table data according to certain rules to generate standardized vectors.

[0043] Domain knowledge is the professional experience and rules accumulated in a specific field. When constructing a knowledge vector library, corresponding verification rules are formulated for each historical point table data based on domain knowledge in accordance with the naming specifications of the power system. The formulation of verification rules is to improve the accuracy and consistency of the data. This application converts domain knowledge into verification rules and associates them with corresponding standardized vectors. When these data are subsequently used for processing and analysis, the point table data to be processed can be verified according to the verification rules.

[0044] S12 data processing: perform structured analysis on the point table data to be processed, obtain target fields and unstructured fields, use the preset BERT-CRF model to parse the unstructured fields, obtain the processing results of the unstructured fields, integrate the target fields and processing results into a tuple, and generate a target vector based on the tuple.

[0045] In this embodiment, the target vector in the S12 data processing and the standardized vector in the S11 database construction use a unified encoding method and template, that is, the corresponding positions of the target vector and the standardized vector express the same meaning. For example, bits 2-4 of the target vector and bits 2-4 of the standardized vector both represent switch names, etc.

[0046] The data in the pending point table contains multiple fields with different data types and meanings. By performing structured parsing on the data in the pending point table, the data can be split by field and the specific content of each field can be clarified.

[0047] Structured parsing refers to the process of breaking down, analyzing, and understanding data with a specific format or pattern according to its inherent structure and rules. It aims to extract meaningful information units that conform to certain logical relationships from complex data. During the structured parsing process, target fields and unstructured fields can be identified. Target fields are those with clear meanings that require focus and use in subsequent processing. For example, in ledger information, line names, switch names, current values, voltage values, etc. are target fields. Unstructured fields refer to fields with relatively free content and non-fixed formats, such as descriptive text from staff members. These fields may contain rich information, but further processing is required to extract useful content.

[0048] For unstructured fields, the preset BERT-CRF model is used for processing. The BERT-CRF model consists of a BERT (Bidirectional Encoder Representations from Transformers) sub-model and a CRF (Conditional Random Field) sub-model. The BERT sub-model is a powerful pre-trained language model that can understand the semantic information of text; the CRF sub-model is a commonly used sequence labeling model that can label each word in the text and identify the entities within it.

[0049] After inputting unstructured fields into the BERT-CRF model, the BERT sub-model first performs word segmentation on the unstructured fields, breaking the continuous text into individual words or sub-word units. Each word is then encoded to obtain its semantic representation. Next, the CRF sub-model uses these semantic representations and predefined annotation rules to annotate each word with entities and entity relationships, for example, identifying entities such as line names and switch names, or content such as closed switches. This process yields the processed results for the unstructured fields, namely, the extracted entities and entity relationships.

[0050] Combine the processing results of the target field and the unstructured field into a tuple. A tuple is a data structure that groups multiple related data elements. For example, if the target field includes a switch name and a line name, and the processing results of the unstructured field extract whether the switch is included in the line, this information can be combined into a tuple such as (line name, switch, included or not included).

[0051] A target vector is generated based on the above-mentioned tuple. To generate the target vector, each element in the tuple needs to be encoded or digitized so that the tuple is converted into a vector form. The target vector will represent the key information of the point table data to be processed and will be used in S13 to calculate the similarity with the standardized vector in the knowledge vector library.

[0052] S13 calculates the similarity, calculates the similarity between the target vector and each standardized vector in turn, records the calibration rule when the similarity is greater than the preset similarity threshold as the target rule, calls the target rule to calibrate the point table data to be processed, and outputs the calibration result.

[0053] The similarity between the target vector and each standardized vector in the knowledge vector library is calculated in turn. Similarity is an indicator to measure the degree of similarity between two vectors. Commonly used similarity calculation methods include cosine similarity and Euclidean distance.

[0054] Taking cosine similarity as an example, it measures the similarity between two vectors by calculating the cosine of the angle between them. The cosine value ranges from -1 to 1. Values closer to 1 indicate closer orientations and higher similarity; values closer to -1 indicate opposite orientations and lower similarity; and a value of 0 indicates orthogonal orientations and no similarity. By calculating the cosine similarity between the target vector and each standardized vector, a series of similarity values can be obtained, reflecting the degree of similarity between the data being processed and the historical point table data.

[0055] The verification rule for a similarity greater than a preset similarity threshold is recorded as the target rule. Once a normalized vector with a similarity greater than the threshold is found, the verification rule corresponding to that normalized vector is obtained and used as the target rule. These target rules are then applied to the point table data to be processed. The verification process involves checking the data for compliance with specifications and requirements based on the target rules. For example, this involves checking whether the data is within a reasonable range and whether the data format is correct.

[0056] After verification is complete, the verification results are output. These results include information such as whether the data passed verification, any data issues, and corresponding correction suggestions. For example, if verification reveals that a data field's value is outside the acceptable range, the verification result will identify the issue and provide a reasonable value range as a correction suggestion. Outputting verification results can help data managers promptly identify and correct data errors, improving data quality and reliability.

[0057] The present embodiment is described below with reference to specific cases.

[0058] S11 builds a database, collecting historical point table data from the power system over a period of time, such as the ledger information of different substations. This information includes line names, switch names, current values, voltage values, etc. Following the State Grid naming standards, field names and data types are standardized. The data is encoded according to certain rules, and this historical point table data is converted into standardized vectors to build a knowledge vector library.

[0059] As shown in Table 1, this is the ledger information of different substations described in this step, that is, the historical point table data.

[0060] Table 1 Historical point table data Substation name Substation A Substation B Substation C Line Name 110kV Chengnan Line 220kV Chengbei Line 35kV City Center Line Switch Name 000 201 switch 301 switch Current value (A) 110 130 080 Voltage value (kV) 110 220 035 Switch status none disconnect closure Temperature (℃) 20 35 60 The line name is encoded using one-hot encoding. However, since there may be many line names and they have specific meanings, this embodiment exemplarily combines the voltage level and line identifier in the line name into a string and assigns a unique numerical ID. For example: "110kV Chengnan Line" is coded as 1 "220kV Chengbei Line" is coded as 2 The code for "35kV City Center Line" is 3 The switch name uses its serial number as its own encoding result, that is: "000" means transformer A does not include a switch "201 switch" is coded as 201 "301 switch" is coded as 301 Switch status coding: "Closed" is coded as 1, "Open" is coded as 0, and no switch is coded as 2.

[0061] Current, voltage, and temperature: Keep the values unchanged.

[0062] The standardized vectors of the three substations are: 1000110110020, 2201130220035, 3301080035160.

[0063] Based on expertise in the power system field, verification rules are added to each standardized vector in the knowledge vector library. For example, for current values, the current value range is specified based on the rated capacity and operating requirements of the line. For switch states, only "closed" or "open" are specified, and descriptions such as "open" and "off" are not standardized.

[0064] S12 data processing obtains the substation ledger point table data to be processed, which contains multiple fields. Through structured parsing, target fields and unstructured fields are identified. Target fields include line names such as "220kV Chengbei Line," switch names such as "220kV Chengbei Line 201 Switch," current values such as "120A," and voltage values such as "220kV." Unstructured fields include descriptive text from staff members such as "220kV Chengbei Line 201 Switch is closed, and the temperature is normal."

[0065] The unstructured fields are input into the preset BERT-CRF model. The BERT-CRF model first performs word segmentation on the unstructured fields. For example, the word segmentation of "220kV Chengbei Line 201 switch is closed, the temperature is normal" is "220kV Chengbei Line", "201 switch", "closed", "temperature", "normal", ",". Then, the BERT sub-model encodes each word to obtain its semantic representation. The CRF sub-model annotates each word with entities and entity relationships based on the semantic representation and pre-defined annotation rules. For example, it identifies "220kV Chengbei Line" as the line name entity, "201 switch" as the switch name entity, and "closed" as the switch state entity, as well as the relationship between them. For example, "220kV Chengbei Line" contains "201 switch" and the switch state is "closed".

[0066] Combine into tuples and generate a target vector: Combine the target fields (line name "220kV Chengbei Line", switch name "220kV Chengbei Line, 201 Switch", current value "120A", voltage value "220kV") and the unstructured field processing results (the relationship between the switch and the line, and the switch status) into a tuple, such as ("220kV Chengbei Line", "201 Switch", "Closed", "Temperature", "Normal"). Then, encode the elements in these tuples to generate a target vector, which represents the key information of the point table data to be processed.

[0067] S13 calculates similarity, sequentially calculating the similarity between the target vector and each standardized vector in the knowledge vector library. In this embodiment, cosine similarity is used as the calculation method. The similarity between the target vector and each standardized vector is obtained by calculating the cosine value of the angle between the target vector and each standardized vector.

[0068] Assuming that the preset similarity threshold is 0.8, when the calculated cosine similarity between the target vector and a normalized vector is 0.85, which is greater than the preset threshold, the verification rule corresponding to the normalized vector is obtained as the target rule.

[0069] For example, the target rules stipulate that the line current value should be between 100A and 150A, and the switch state can only be "closed" or "open." These target rules are then applied to the data in the pending point table. The verification reveals that the current value of "120A" is within the reasonable range, and the switch state of "closed" also meets the requirements.

[0070] After the calibration is complete, the calibration results are output. Since the data in this example all meet the target rules, the calibration result indicates that the data has passed the calibration. If there are any violations, such as current values outside the range, the calibration result will indicate the issue and recommend corrections based on the range of values specified in the calibration rules.

[0071] By adopting the above solution, this embodiment can process unstructured data in the point table data to be processed.

[0072] In other embodiments, the method further comprises: S14 Correction: During the data processing process, after executing S13 to calculate the similarity, if it is found that the similarity between the target vector and all the standardized vectors in the knowledge vector library is not greater than the preset similarity threshold, this means that there is a large difference between the current point table data to be processed and the historical data, and there may be an abnormality. At this time, it is necessary to start the correction process. The specific steps are as follows: Abnormal fields are extracted from the point table data to be processed. Abnormal fields refer to fields that do not match the format of historical point table data, do not comply with the naming specifications of the State Grid, or may contain errors.

[0073] If the extracted abnormal fields are data categories, they need to be mapped into structured rules based on domain knowledge. Domain knowledge includes professional information about data classification, value ranges, business logic, and other aspects of the power system. Data categories include equipment parameters (such as line resistance and capacitance), operating status (such as current, voltage, and power), and equipment information (such as equipment model and manufacturer).

[0074] Based on the verification rules added to each standardized vector in the knowledge vector library, the structured rules are compared with the verification rules, and the parts of the structured rules that are inconsistent with the verification rules are replaced with the content in the verification rules, that is, the correction strategy. Using the obtained correction strategy, the abnormal fields in the point table data to be processed are modified to obtain the modified point table data to be processed. After the modification is completed, S12 data processing is re-executed, and the modified data is re-performed with operations such as structured parsing, unstructured field processing, field integration, and target vector generation, so that similarity calculation and verification can be performed again later to improve the quality of the point table data to be processed. When S13 is executed again to calculate the similarity, if the similarity is still not greater than the preset similarity threshold, the unmatched verification rule is output.

[0075] If the outlier fields are not data categories, they may contain text information and are considered unstructured data. In this case, the pre-set BERT-CRF model is used to parse the outlier fields. The BERT sub-model segments the continuous text into meaningful words or word units; the CRF sub-model then performs entity or entity relationship annotation on each word based on semantic information and pre-defined annotation rules, obtaining the processing results for the outlier fields.

[0076] The large model API is called to input the processing results of the abnormal field into the large model. The large model possesses powerful language understanding and reasoning capabilities, enabling it to conduct in-depth analysis and reasoning on the abnormal field based on the input entity information and context. For example, given an abnormal text describing a switch failure, the large model can infer standard information about the fault. The large model returns standard information (i.e., the inference result of the abnormal field). This inference result is a replacement scheme for the abnormal field in the processed point table data that complies with power system naming standards. The inference result of the large model is then used to replace the abnormal field in the processed point table data, generating new processed point table data. Subsequently, S12 data processing is executed again, performing structured parsing and other operations on the new processed point table data. This prepares for subsequent similarity calculation and verification to improve the accuracy and consistency of the processed point table data. If the similarity calculation in S13 is still not greater than the preset similarity threshold, a "no match" verification rule is output.

[0077] By adopting the above solution, this embodiment can take corresponding processing measures for abnormal fields of different types (data category and non-data category), thereby improving the quality of the point table data to be processed and making it more consistent with historical data patterns and business specifications.

[0078] Example 2: This example differs from Example 1 in that the similarity calculation process in step S13 is as follows: The similarity between the target vector and each standardized vector is calculated sequentially. Once the similarity is calculated, it is compared with a preset similarity threshold. This threshold is a standard set based on actual business needs and data characteristics. It is used to select the verification rules corresponding to the historical data most similar to the data being processed. If the similarity between a standardized vector and the target vector exceeds the threshold, the verification rule corresponding to that standardized vector is recorded as the target rule.

[0079] Determine whether multiple target rules exist. If only one target rule exists, directly apply that target rule to the data in the pending point table. The verification process checks whether the data meets specifications and requirements based on the target rule. For example, it checks whether the data is within a reasonable range, whether the data format is correct, and whether the naming method is standardized. After the verification is completed, the verification results are output. These results may include information such as whether the data passed the verification, any problems found in the data, and any correction suggestions.

[0080] If there are multiple target rules, it is necessary to perform S21 to filter the target rules, so as to select the rule combination that is most suitable for verifying the current point table data to be processed from the multiple target rules.

[0081] S21 selects target rules and encodes each target rule to obtain a qubit. A qubit is the fundamental unit in quantum computing, capable of being in a state of 0, 1, or a superposition of both. The quantum state of each target rule is set based on the qubit. This process represents the target rule's information in a quantum state for subsequent processing using quantum computing methods.

[0082] For example, there are four target rules R1, R2, and R3, which respectively represent: R1 is used to check whether the switch naming complies with the specifications, R2 is used to check whether the switch status is closed, and R3 is used to check whether the switch naming is open.

[0083] The target rules R1, R2, and R3 are encoded as quantum bits q1, q2, and q3 respectively. A quantum bit is the basic unit in quantum computing. It can be in a state of 0, 1, or a superposition of the two. The quantum state of R1 after information encoding is , the quantum state of R2 after information encoding is , the quantum state of R3 after information encoding is .

[0084] The number of calls to each verification rule in the knowledge vector library is counted to obtain statistical results. This statistical result can be used to determine the number of calls to each target rule. The number of calls reflects the frequency with which the verification rule has been used in previous data processing processes. Generally, a verification rule with higher frequency of use is more versatile and effective. A quantum state weight is set for each target rule based on its number of calls. The more times a target rule is called, the greater its quantum state weight, indicating its greater importance in subsequent processing.

[0085] For example, the number of calls to R1 is 50, the number of calls to R2 is 30, and the number of calls to R3 is 20. Then the weight of R1 is 0.5, the weight of R2 is 0.3, and the weight of R3 is 0.2.

[0086] Based on the quantum state of each target rule and the weights set, the superposition state of each target rule is calculated. A superposition state represents a linear combination of multiple states that a quantum system can be in simultaneously. This superposition state is used as a new quantum state to subsequently simulate the entanglement relationship between the target rules.

[0087] For example, new quantum states The calculation method is, .

[0088] The entanglement between target rules is simulated through the use of a new quantum state for each target rule and quantum gate operations. Quantum gate operations transform the quantum state, simulating the interactions and connections between the rules. The entangled quantum state is then output, and projection measurement is performed on this entangled quantum state to obtain a collapse result. Projection measurement is an operation in quantum computing that causes the quantum state to collapse to a specific eigenstate. Based on the collapse result, the target rules are deleted to obtain the remaining target rules. The most relevant and important rules can then be selected from the multiple target rules, reducing the complexity and computational effort of subsequent verification.

[0089] Taking R2 and R3 as an example, a CONT gate operation is used to establish an entanglement between them. That is, when target rules R2 and R3 are triggered simultaneously, target rule R3 is forcibly disabled. The second qubit in R2 (indexed from left to right, starting at 1) and the third qubit in R3 are used to construct a CNOT gate operation, using the second qubit in R2 as the control bit and the third qubit in R3 as the target bit.

[0090] The rule of the CNOT gate is: when the control bit is |1>, the state of the target bit will flip; when the control bit is |0>, the state of the target bit remains unchanged.

[0091] Since the control bit is 1, according to the CNOT gate rule, the state of the second 1 of the target bit will be flipped. The target bit is initially in the state of 1 and becomes 0 after flipping.

[0092] .

[0093] That is, after the CNOT gate operation, when the target rule R2 is triggered, the target rule R3 is forced to close. At this time, the entangled quantum state of the target rule R2 and the target rule R3 for: .

[0094] Projection measurement is performed on the entangled quantum state to calculate the probability of all target rule combinations. The calculation model is as follows: ; Assume the input rules are {target rule R1, target rule R2, target rule R3}. After quantum gate operation and entanglement, the calculated joint probability distribution is P(target rule R1 ∧ target rule R2) = 0.1, P(target rule R1 ∧ target rule R3) = 0.7, and P(target rule R2 ∧ target rule R3) = 0.2. The optimal combination is target rule R1 + target rule R3, as their joint probability is the highest. This completes the selection of target rules.

[0095] The running code is as follows: # Initial state R2 = [0, 1, 0] # Quantum state corresponding to rule R2 R3 = [0, 0, 1] # Quantum state corresponding to rule R3 # Extract the control bits and target bits (assuming that the second bit of R2 and the third bit of R3 participate in the CNOT operation) control_bit_R2 = R2[1] # The second bit of R2 (index starts from 0) target_bit_R3 = R3[2] # The third bit of R3 (index starts from 0) # Apply CNOT gate rules if control_bit_R2 == 1: target_bit_R3 = 1 - target_bit_R3 # Flip target bit # Update the status of R3 (update only the bits involved in the operation) R3_new = R3.copy() R3_new[2] = target_bit_R3 print("Before operation:") print("Rule R2 status:", R2) print("Rule R3 status:", R3) print("After the operation:") print("Rule R2 status:", R2) # The status of R2 has not changed print("Rule R3 status:", R3_new) # The bits involved in the operation in the status of R3 are updated The call time of each verification rule in the knowledge vector library is counted. Each verification rule in the knowledge vector library is used as a node, and a knowledge graph is constructed with the order of each verification rule's call time as a directed edge. A knowledge graph is a graphical structure used to represent entities and their relationships. This method clearly demonstrates the call order relationship between verification rules. The remaining target rules are sequentially input into the knowledge graph, and the verification order of the remaining target rules is output. The verification order is determined based on the historical call time of the rules.

[0096] The remaining target rules are applied to the data in the target point table in the order in which they are checked. The check process is similar to that of a single target rule, but each rule is applied sequentially in the specified order. This ensures a more organized and efficient check process, improving the accuracy and reliability of the data.

[0097] By adopting the above scheme, the present application can screen out the most suitable rule combination from multiple target rules and determine a reasonable verification order, thereby performing more accurate and effective verification of the point table data to be processed, improving data quality, and providing a reliable foundation for subsequent data analysis and application.

[0098] Example 3: Reference Figure 2 The difference between this embodiment and embodiment 1 is that, before executing S12 data processing, the method further includes: S31 integrity check: setting integrity indicators, which include field integrity rate, field value range coverage, logical correlation between fields, and collection time continuity; checking the target fields and unstructured field processing results in the point table data to be processed one by one, and calculating the score of each integrity indicator.

[0099] S32 score judgment, judge in turn whether the score of each integrity indicator is greater than the preset score threshold, if so, execute S12 data processing; if not, issue an alarm signal, the alarm signal includes the integrity indicator below the preset score threshold.

[0100] The integrity index is a key criterion for measuring the completeness of the point table data to be processed, and mainly includes the following four aspects: Field completeness rate: The field completeness rate reflects whether each field in the point table data to be processed has corresponding content.

[0101] Field Range Coverage: Field range coverage measures whether a field's values cover the expected range of values for that field. For example, for a "switch" field, its range should be "closed" and "open." If the data only contains the value "closed," then the field range coverage is low, indicating that the data is incomplete in terms of the values for that field.

[0102] Logical correlation between fields: There is often a certain logical relationship between different fields. For example, the "line" and "switch" fields should have a containment and being contained relationship. If the logical relationship between the two does not hold, it means that there is a problem with the logical correlation between the fields.

[0103] Collection time continuity: For data with time series characteristics, collection time continuity is crucial. If the point table data being processed has collection intervals that are too large or data at certain time points is missing, this continuity will be compromised. For example, in point tables recording equipment operating data, data should normally be collected at fixed intervals. If there are long periods of missing data, this continuity will be compromised.

[0104] Each target field and unstructured field processing result in the processed point table data is verified one by one. For each integrity indicator, a score is calculated according to the corresponding calculation rules. For example, for field completeness, the score can be calculated based on the ratio of the number of fields with valid values to the total number of fields. For field range coverage, the score can be calculated based on the ratio of the actual number of field values (for range-based data, the range needs to be discretized according to certain intervals (such as 10, 1, 0.1, 0.01, etc.) to the expected number of values. For logical correlation between fields, the data is checked for compliance with pre-set logical rules and the proportion of data that complies with the rules is calculated. For collection time continuity, the theoretical number of data points that should be collected is calculated based on the pre-set time interval and the start and end times of the data collection. The actual number of data points collected is then counted, and the ratio of the actual number of data points to the theoretical number of data points is used as the score for the indicator.

[0105] It is determined in turn whether the score of each integrity indicator is greater than a preset score threshold. If the score of each integrity indicator is greater than the preset score threshold, it means that the point table data to be processed meets the integrity requirements and S12 data processing can be performed.

[0106] If the score of a particular integrity indicator is below the preset threshold, it indicates that there is a problem with the data in the point table to be processed, and an alarm signal is issued. The alarm signal clearly indicates the integrity indicator that is below the preset threshold, so that data management personnel can promptly understand the data problems and take appropriate measures to address them.

[0107] In other embodiments, before issuing an alarm signal based on the score determination in step S32, the method further includes: Call the big model API, input the point table data to be processed into the big model, infer the missing fields in the point table data to be processed, obtain the inferred missing fields, use the inferred missing fields to fill the point table data to be processed, obtain the filled point table data to be processed, recalculate the score of the integrity index of the filled point table data to be processed, and re-execute the S32 score judgment based on the score of the integrity index of the filled point table data to be processed. When the S32 score judgment is re-executed, if the score of the integrity index is still less than the preset score threshold after the S32 score judgment is re-executed, an alarm signal is issued.

[0108] In other embodiments, the data in the point table to be processed is input into a large model. The large model outputs a prediction result for missing fields in the point table data to be processed based on patterns in historical data and known fields in the point table data to be processed. The predicted missing fields are then used to fill in the gaps in the point table data to obtain the filled point table data to be processed.

[0109] Recalculate the integrity score of the populated data in the pending point table and re-execute the score judgment step based on the calculation result. This can check whether the populated data meets the integrity requirements. If it does, the subsequent data processing steps can be continued; if it still does not meet the requirements, an alarm signal will be issued again.

[0110] By adopting the above solution, the present application can comprehensively check and process the integrity of the point table data to be processed, improve the quality of the data, and enhance the accuracy and reliability of processing and analysis.

[0111] Example 4: This example discloses a point table data verification system based on a knowledge vector library and a large model, the system comprising: a processor and a memory, The memory stores program code; When the processor calls the program code in the memory, the processor executes the steps of the point table data verification method based on the knowledge vector library and the large model.

[0112] The above are all preferred embodiments of the present application, and are not intended to limit the scope of protection of the present application. Therefore, any equivalent changes made based on the structure, shape, and principle of the present application should be included in the scope of protection of the present application.

Claims

1. A point table data verification method based on a knowledge vector library and a large model, characterized by: include: Database construction: Build a knowledge vector database that stores standardized vectors of multiple historical point table data. Based on domain knowledge, formulate verification rules for each historical point table data and associate the verification rules with the corresponding standardized vectors. Data processing: Perform structured analysis on the point table data to be processed to obtain target fields and unstructured fields. Use the preset BERT-CRF model to parse the unstructured fields to obtain the processing results of the unstructured fields. Combine the target fields and processing results into a tuple, and generate a target vector based on the tuple. Calculate similarity: Calculate the similarity between the target vector and each standardized vector in turn, record the corresponding calibration rule when the similarity is greater than the preset similarity threshold as the target rule, call the target rule to calibrate the point table data to be processed, and output the calibration result.

2. The point table data verification method based on the knowledge vector library and the large model according to claim 1 is characterized in that: The method further comprises: Correction: When the similarity is not greater than a preset similarity threshold, the abnormal fields in the point table data to be processed are extracted. If the abnormal fields are data categories, the abnormal fields are mapped to structured rules based on domain knowledge, and a correction strategy for the structured rules is obtained based on the verification rules in the knowledge vector library. The abnormal fields in the point table data to be processed are modified using the correction strategy to obtain the modified point table data to be processed, and the data processing step is performed; In the data processing step, the modified point table data to be processed is subjected to structured analysis.

3. The point table data verification method based on the knowledge vector library and the large model according to claim 2 is characterized in that: The correction step further includes: If the abnormal field is not a data category, the preset BERT-CRF model is used to parse the abnormal field to obtain the processing result of the abnormal field. The API of the large model is called to input the processing result of the abnormal field into the large model to obtain the inference result of the large model on the abnormal field. The inference result is used to replace the abnormal field in the point table data to be processed, and new point table data to be processed is obtained. Then, the data processing steps are performed; In the data processing step, the new point table data to be processed is subjected to structured analysis.

4. The point table data verification method based on a knowledge vector library and a large model according to any one of claims 1 to 3, characterized in that: The step of calculating the similarity also includes: When there are multiple target rules, each target rule is encoded to obtain quantum bits, and the quantum state of each target rule is set based on the quantum bits. The entanglement relationship between the target rules is simulated through the quantum state of each target rule and quantum gate operations, and the entangled quantum state is output. The entangled quantum state is projected and measured to obtain the collapse result. The target rules are deleted based on the collapse result, and the remaining target rules are called to check the data of the point table to be processed.

5. The point table data verification method based on the knowledge vector library and the large model according to claim 4 is characterized in that: The step of calculating the similarity also includes: The number of calls of each verification rule in the knowledge vector library is counted to obtain statistical results, the number of calls of each target rule is obtained based on the statistical results, the weight of the quantum state of each target rule is set according to the number of calls of each target rule, the superposition state of each target rule is calculated based on the quantum state of each target rule and the weight, and the superposition state is used as the new quantum state.

6. The point table data verification method based on the knowledge vector library and the large model according to claim 5 is characterized in that: The step of calculating the similarity also includes: The calling time of each verification rule in the knowledge vector library is counted, and a verification rule in the knowledge vector library is used as a node. The knowledge graph is formulated with the order of the calling time of each verification rule as the directed edge. The remaining target rules are input into the knowledge graph in sequence, and the verification order of the remaining target rules is output. The remaining target rules are then called to verify the data in the processing point table according to the verification order.

7. The point table data verification method based on a knowledge vector library and a large model according to any one of claims 1 to 3, characterized in that: Before performing the data processing step, the method further includes: Completeness check: Set completeness indicators, including field completeness rate, field value range coverage, logical association between fields, and collection time continuity; check the target fields and unstructured field processing results in the point table data to be processed one by one, and calculate the score of each completeness indicator; Score judgment: judge whether the score of each integrity indicator is greater than the preset score threshold in turn. If so, execute the data processing step; if not, issue an alarm signal, which includes the integrity indicator below the preset score threshold.

8. The point table data verification method based on the knowledge vector library and the large model according to claim 7 is characterized in that: The method further comprises: Call the big model API, input the point table data to be processed into the big model, infer the missing fields in the point table data to be processed, obtain the inferred missing fields, use the inferred missing fields to fill the point table data to be processed, obtain the filled point table data to be processed, recalculate the score of the integrity index of the filled point table data to be processed, and re-execute the score judgment step based on the score of the integrity index of the filled point table data to be processed.

9. The point table data verification system based on the knowledge vector library and the large model is characterized by: include: processor and memory, The memory stores program code; When the processor calls the program code in the memory, the steps of the method according to any one of claims 1 to 8 are executed.

Images