Method, device and electronic equipment for processing power system text data

Abstract

The application provides a power system text data processing method, device and electronic equipment. The processing method comprises: acquiring text data corresponding to each document; decomposing the text data into a plurality of character sequences, and constructing a feature vector corresponding to each document according to the plurality of character sequences; segmenting the feature vector to obtain a plurality of sub-feature vectors corresponding to the feature vector; establishing an inverted index of the plurality of sub-feature vectors; regarding text data corresponding to sub-feature vectors with the same value after the inverted index as a combination; calculating the similarity of two text data in a similar text data pair, determining that the two text data in the similar text data pair are duplicate data when the similarity is greater than a preset threshold, deleting one duplicate data, and performing quality evaluation and data screening on the deduplicated text data. The problem of low efficiency in the prior art that duplicate checking of power system Chinese text or language data is usually performed manually is solved.

Description

Technical Field

[0001] This application relates to the field of power system text data processing, and more specifically, to a method for processing power system text data, a device for processing power system text data, a computer-readable storage medium, and an electronic device. Background Technology

[0002] With the rapid development of artificial intelligence technology, especially in the field of Natural Language Processing (NLP), large-scale pre-trained models have demonstrated their superior capabilities in numerous application scenarios. While many existing general-purpose models perform well across a wide range of tasks, their performance in specific domains, such as the expertise and regulations of the power industry, has fallen short of expectations. To improve model performance in specific domains, further training using domain-specific data is necessary. In the power industry, regulations, including laws and regulations, industry standards, management rules, operating procedures, and technical specifications, play a crucial role in the normal operation of the entire power company. With the acceleration of digital transformation, management needs are becoming increasingly refined, and the complexity of various regulations is also increasing. The enterprise internal policy question-answering robot project, focusing on large-scale language models, faces several challenges. First, enterprise internal policy documents are vast and complex, potentially containing different formats, technical terminology, and regulatory clauses. Therefore, significant effort is required in data preparation and cleaning to ensure the quality and consistency of training data. Second, large-scale language models are typically pre-trained on extensive datasets. To adapt to the specific domain of enterprise internal policies, it is necessary to combine policy data with large models.

[0003] Existing technical solutions primarily focus on constructing and training large language models to improve model performance and generalization ability. However, these solutions have limitations in data deduplication and quality assessment. Current deduplication methods typically rely on manual identification and processing of duplicate data, resulting in low efficiency. Simultaneously, existing data quality assessment methods are heavily subjective and lack automation and scalability. Regarding model processing capabilities, the inconsistent quality of existing institutional data limits the generalization ability of models trained on this data, making it difficult to adapt to the complexity of power industry regulations.

[0004] Therefore, a method is needed to improve the processing efficiency of language processing models. Summary of the Invention

[0005] The main objective of this application is to provide a method, apparatus, computer-readable storage medium, and electronic device for processing text data in power systems, so as to at least solve the problem that in the prior art, the deduplication of text or language data in power systems is usually carried out manually, which is inefficient.

[0006] To achieve the above objectives, according to one aspect of this application, a method for processing text data in a power system is provided, comprising: acquiring text data corresponding to each document, wherein the text data is text-formatted data in the power system; decomposing the text data into multiple character sequences, constructing a feature vector corresponding to each document based on the multiple character sequences, segmenting the feature vector to obtain multiple sub-feature vectors corresponding to the feature vector, establishing an inverted index of the multiple sub-feature vectors, and combining the text data corresponding to the sub-feature vectors with the same value after the inverted index to obtain similar text data pairs of documents, wherein the inverted index includes the sub-feature vectors and the text data corresponding to each sub-feature vector; calculating the similarity between two text data in the similar text data pair, and determining that the two text data in the similar text data pair are duplicate data if the similarity is greater than a preset threshold, deleting one of the duplicate data to obtain deduplicated text data, and performing quality assessment and data filtering on the deduplicated text data to obtain high-quality text data corresponding to the document.

[0007] Optionally, constructing a feature vector for each document based on multiple character sequences includes: constructing a two-dimensional word library from multiple character sequences, encoding each dimension of the two-dimensional word library to obtain an initial feature vector for each document, wherein the value of the initial feature vector is 0 or 1; determining the step: generating a random sequence, and sorting the initial feature vectors for multiple documents according to the random sequence, determining the row number corresponding to the first element with a value of 1 in each column as the minimum hash value of the document; repeating the determination step a target number of times to obtain the feature vector for the document, wherein the dimension of the feature vector is the target number of times.

[0008] Optionally, an inverted index of multiple sub-feature vectors is established, including: using the hash value corresponding to each sub-feature vector as the key and the text data corresponding to each sub-feature vector as the numerical value corresponding to the key, to obtain the inverted index of each sub-feature vector.

[0009] Optionally, calculating the similarity between two text data in a similar text data pair includes: according to the formula Calculate the similarity between two text data pairs, where N represents the dimension of the similar text data pair, and I(x) = N(x). n ==y n ) represents text data x n and text data y n same.

[0010] Optionally, the deduplicated text data is subjected to quality assessment and data filtering to obtain high-quality text data corresponding to the document, including: through formulas Calculate the accuracy index, where D q R represents the number of text data that violate the accuracy rules.a C represents the total amount of text information in the text data. a C represents the total number of numerical information in the text data. s S1 represents the number of missing numerical information items, and S1 represents the accuracy index; (using the formula...) Calculate the integrity index, where D s R represents the number of empty data points. s S represents the total number of missing text messages. 21 S represents the proportion of empty data. 22 S1 represents the proportion of accurate data, and S2 represents the integrity index; the number of text data that violates the consistency principle is obtained through the formula. Calculate the degree of satisfaction of the consistency index, where S3 represents the degree of satisfaction, and R... c This indicates the number of text data that violate the consistency principle, which includes at least the requirement that the text data format is the same as the predetermined format. The quality assessment value of the deduplicated text data is calculated using a formula. If the quality assessment value is greater than a preset threshold, the deduplicated text data is determined to be high-quality text data.

[0011] Optionally, after obtaining the high-quality text data corresponding to the document, the method further includes: dividing the high-quality text data into a training set and a validation set; training the initial language model using the training set and updating the parameters of the initial language model using gradient descent to obtain the language model; and validating the language model using the validation set.

[0012] Optionally, after obtaining the text data corresponding to each document, the method further includes: cleaning the text data to obtain valid text data; and standardizing the valid text data to obtain standardized text data.

[0013] According to another aspect of this application, a power system text data processing apparatus is provided, comprising: an acquisition unit for acquiring text data corresponding to each document, wherein the text data is text-formatted data in the power system; a determination unit for decomposing the text data into multiple character sequences, constructing a feature vector corresponding to each document based on the multiple character sequences, segmenting the feature vector to obtain multiple sub-feature vectors corresponding to the feature vector, establishing an inverted index of the multiple sub-feature vectors, and combining the text data corresponding to the sub-feature vectors with the same value after the inverted index as a group to obtain similar text data pairs of documents, wherein the inverted index includes sub-feature vectors and text data corresponding to each sub-feature vector; and a filtering unit for calculating the similarity between two text data in a similar text data pair, determining that two text data in the similar text data pair are duplicate data if the similarity is greater than a preset threshold, deleting one of the duplicate data to obtain deduplicated text data, and performing quality assessment and data filtering on the deduplicated text data to obtain high-quality text data corresponding to the document.

[0014] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein, when the program is running, it controls the device where the computer-readable storage medium is located to perform the above-described method for processing power system text data.

[0015] According to another aspect of this application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include a method for processing the above-described power system text data.

[0016] Applying the technical solution of this application, the method for processing text data in a power system includes: acquiring text data corresponding to each document, wherein the text data is text-formatted data within the power system; decomposing the text data into multiple character sequences, constructing a feature vector corresponding to each document based on the multiple character sequences, segmenting the feature vector to obtain multiple sub-feature vectors corresponding to the feature vector, establishing an inverted index for the multiple sub-feature vectors, and combining the text data corresponding to the sub-feature vectors with the same value after the inverted index to obtain similar text data pairs for the document, wherein the inverted index includes the sub-feature vectors and the text data corresponding to each sub-feature vector; calculating the similarity between two text data pairs in the similar text data pairs, and determining that the two text data pairs are duplicates if the similarity is greater than a preset threshold, deleting one of the duplicate data to obtain deduplicated text data, and performing quality assessment and data filtering on the deduplicated text data to obtain high-quality text data corresponding to the document. By decomposing the text data and determining the feature vectors, and then calculating the similarity, duplicate text data can be automatically identified, achieving automatic deduplication and improving the efficiency of language text processing. Attached Figure Description

[0017] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 A hardware structure block diagram of a mobile terminal for performing a method for processing text data in a power system, according to an embodiment of this application, is shown.

[0019] Figure 2 A flowchart illustrating an embodiment of this application provides a method for processing text data in a power system.

[0020] Figure 3 A flowchart illustrating an embodiment of this application provides a method for processing text data in a power system;

[0021] Figure 4 A structural block diagram of a power system text data processing device provided in an embodiment of this application is shown.

[0022] The above figures include the following reference numerals:

[0023] 102. Processor; 104. Memory; 106. Transmission device; 108. Input / output device. Detailed Implementation

[0024] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] As described in the background section, the complexity of various systems in the power industry is gradually increasing. Internal enterprise policy documents are vast and complex, potentially containing different formats, technical terms, and regulatory clauses. Significant effort is required for data preparation and cleaning to ensure the quality and consistency of training data. However, existing data deduplication methods typically rely on manual identification and processing of duplicate data, which is inefficient and cannot achieve automated or scalable processing. Therefore, to address the inefficiency of manually identifying and processing duplicate text or language data in power systems, embodiments of this application provide a method for processing text data in power systems, a device for processing text data in power systems, a computer-readable storage medium, and an electronic device.

[0028] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0029] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for a method of processing text data in a power system according to an embodiment of the present invention. Figure 1As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0030] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the power system text data processing method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned networks may include wireless networks provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0031] This embodiment provides a method for processing text data in a power system that runs on a mobile terminal, computer terminal, or similar computing device. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0032] Figure 2 This is a flowchart of a method for processing text data in a power system according to an embodiment of this application. For example... Figure 2 As shown, the method includes the following steps:

[0033] Step S101: Obtain the text data corresponding to each document, wherein the text data is text-formatted data in the power system;

[0034] Specifically, text-formatted data in the power system includes text, reports, and policy documents.

[0035] Step S101: Obtaining the text data corresponding to each document can be achieved as follows:

[0036] Step S1011: After obtaining the text data corresponding to each document, perform data cleaning on the text data to obtain valid text data;

[0037] Step S1012: Standardize the valid text data to obtain standardized text data. This removes irrelevant information and incomplete data, and divides the preprocessed dataset into training and validation sets according to a set ratio (e.g., 8:2 or 7:3).

[0038] Step S102: Decompose the text data into multiple character sequences, construct a feature vector corresponding to each document based on the multiple character sequences, segment the feature vector to obtain multiple sub-feature vectors corresponding to the feature vector, establish an inverted index of multiple sub-feature vectors, and combine the text data corresponding to the sub-feature vectors with the same value after the inverted index to obtain similar text data pairs of documents. The inverted index includes the sub-feature vectors and the text data corresponding to each sub-feature vector.

[0039] Specifically, the MinHash algorithm is used to remove duplicates from the preprocessed dataset. Each text is represented as a set of n-grams, and the MinHash signature of this text set is calculated and stored in a hash table. An n-gram is a text feature representation method that decomposes text into a continuous sequence of n words or characters.

[0040] n-gram set representation: For a given value of n, the text is broken down into n consecutive words or character sequences. For example, text = "The quick brown fox jumps over the lazy dog."

[0041] 3-grams:['the quick brown','quick brown fox','brown fox jumps','foxjumps over','jumps over the','over the lazy','the lazy dog'].

[0042] Statistical Feature Vector: The distinct K-shingle values ​​in the n-gram set of all documents constitute a phrase library of size M. Each document is encoded using a one-hot encoding method, resulting in a feature vector of length M. When the m-th K-shingle in the phrase library appears in the document, the m-th element of the document's feature vector is 1; otherwise, it is 0. For example, 3-grams: ['the quick brown','quick brown fox','brown fox jumps','fox jumpsover','jumps over the','over the lazy','the lazy dog']. After deduplication, no duplicates are found, so the length M is the length of the 3-gram. Each document refers to a vector of M dimensions.

[0043] Calculate the Minhash feature vector: Generate a random permutation by rearranging the document matrix: if each column represents a document, then rearrange the rows; assuming each column represents a document, after rearranging the rows, take the row number corresponding to the first element with a value of 1 in each column as the minimum hash value of the document in that column after this random permutation. Repeat the above process N times to generate an N-dimensional Minhash feature vector for each document.

[0044] One-hot encoding of K-shingle distinct values ​​results in a format like [1,0,0,0,0,0,0], indicating the presence of the first K-shingle distinct value. Therefore, if a document contains the corresponding K-shingle distinct value, a 1 is set at the corresponding position.

[0045] Specifically, a segmented index is established to extract potentially similar text pairs: After segmenting the N-dimensional feature vector, an inverted index is created. When the number of segments is sufficiently large, two similar texts have a high probability of having the same value in a certain segment. Texts with the same index value are paired up as potentially similar text pairs.

[0046] The segmentation method is as follows: Suppose that the N-dimensional Minhash feature vector is divided into B segments, and the length of each segment is C, that is, N = B * C.

[0047] Furthermore, an inverted index refers to constructing segments using hash values ​​as keys and text objects as values. When text within a segment has the same value, it indicates that the two texts are similar within that segment, and an inverted index is constructed.

[0048] Here, it's important to explain that "same index value" means that the feature vector values ​​within this segment are consistent. For example, in a 16-dimensional feature dataset divided into 4 segments, to find values ​​with high similarity, such as those within a Hamming distance of 3, S1 = (1111, 0000, 0101, 1010), a similar segment might be (1111, 0001, 0111, 1011), where only three numbers differ. Therefore, at least one segment is consistent, namely 1111. Thus, their indices within this segment are identical.

[0049] Step S103: Calculate the similarity between two text data in a similar text data pair. If the similarity is greater than a preset threshold, determine that the two text data in the similar text data pair are duplicate data, delete one of the duplicate data to obtain deduplicated text data, and perform quality assessment and data filtering on the deduplicated text data to obtain high-quality text data corresponding to the document.

[0050] Specifically, the chatGLM model is fine-tuned using selected high-quality data. Fine-tuning methods such as LoRa are chosen for small parameter tuning to reduce model parameter updates and improve training efficiency. During fine-tuning, the model is repeatedly trained on high-quality data, and optimization algorithms such as gradient descent are used to update the model parameters.

[0051] In step S103: After obtaining the high-quality text data corresponding to the document, the method further includes: dividing the high-quality text data into a training set and a validation set; training the initial language model using the training set and updating the parameters of the initial language model using gradient descent to obtain the language model; and validating the language model using the validation set.

[0052] In this embodiment, constructing a feature vector corresponding to each document based on multiple character sequences includes: constructing a two-dimensional word library from multiple character sequences, encoding each dimension of the two-dimensional word library to obtain an initial feature vector corresponding to each document, wherein the value of the initial feature vector is 0 or 1; determining the step: generating a random sequence, and sorting the initial feature vectors corresponding to multiple documents according to the random sequence, determining the row number corresponding to the first element with a value of 1 in each column as the minimum hash value of the document; repeating the determination step a target number of times to obtain the feature vector corresponding to the document, wherein the dimension of the feature vector is the target number of times.

[0053] To build an inverted index for multiple sub-feature vectors, the following steps are taken: using the hash value corresponding to each sub-feature vector as the key and the text data corresponding to each sub-feature vector as the numerical value corresponding to the key, thus obtaining the inverted index of each sub-feature vector.

[0054] Calculating the similarity between two text data pairs in a similar text data pair includes: according to the formula Calculate the similarity between two text data sets, where N represents the dimension of the similar text data pair, and I(x) = N(x). n ==y n ) represents text data x n and text data y n The similarity of potentially similar text pairs is calculated as follows: the proportion of corresponding elements in two Minhash vectors that are equal is the similarity between the two texts. When the similarity exceeds a specified threshold, the two texts are considered duplicates. Documents marked as duplicates are then removed from the dataset.

[0055] The deduplicated text data is subjected to quality assessment and data filtering to obtain high-quality text data corresponding to the documents, including: through formulas Calculate the accuracy index, where D q R represents the number of text data that violate the accuracy rules. a C represents the total amount of text information in the text data. a C represents the total number of numerical information in the text data. s S1 represents the number of missing numerical information items, and S1 represents the accuracy index; (using the formula...) Calculate the integrity index, where D s R represents the number of empty data points. s S represents the total number of missing text messages. 21 S represents the proportion of empty data. 22 S1 represents the proportion of accurate data, and S2 represents the integrity index; the number of text data that violates the consistency principle is obtained through the formula. Calculate the degree of satisfaction of the consistency index, where S3 represents the degree of satisfaction, and R... c This indicates the number of text data that violate the consistency principle, which includes at least the requirement that the text data format is the same as the predetermined format. The quality assessment value of the deduplicated text data is calculated using a formula. If the quality assessment value is greater than a preset threshold, the deduplicated text data is determined to be high-quality text data.

[0056] A data quality assessment model is used to evaluate the quality of each document. The quality assessment is based on factors such as the accuracy, completeness, and consistency of the content. Each document is assigned a quality score (S). n This can be achieved by using expert systems for scoring or by using machine learning methods.

[0057] The above embodiments demonstrate how text data can be decomposed and feature vectors determined, and then similarity calculated. This method can automatically identify duplicate text data, achieve automatic deduplication, and improve the efficiency of language and text processing.

[0058] To enable those skilled in the art to better understand the technical solution of this application, the implementation process of the power system text data processing method of this application will be described in detail below with reference to specific embodiments.

[0059] This embodiment relates to a method for processing text data in a power system, such as... Figure 3 As shown, it includes the following steps:

[0060] Step S1: Data Collection and Preprocessing; Collect datasets related to the power industry, including texts, reports, policy documents, etc. Clean the data to remove irrelevant and incomplete information. Divide the preprocessed dataset into training and validation sets according to a set ratio (e.g., 8:2 or 7:3).

[0061] Step S2: Deduplication: Filter power industry policy data and remove duplicate text data;

[0062] Specifically, the MinHash algorithm is used to remove duplicates from the preprocessed dataset. Each text is represented as a set of n-grams, and the MinHash signature of this text set is calculated and stored in a hash table. An n-gram is a text feature representation method that decomposes text into a continuous sequence of n words or characters.

[0063] Statistical Feature Vector: The distinct K-shingle values ​​in the n-gram set of all documents constitute a word library of size M. Each document is encoded using a one-hot encoding method to obtain a feature vector of length M. When the m-th K-shingle in the word library appears in the document, the m-th element of the document's feature vector is 1; otherwise, it is 0.

[0064] Calculate the Minhash feature vector: Generate a random permutation by rearranging the document matrix: if each column represents a document, then rearrange the rows; assuming each column represents a document, after rearranging the rows, take the row number corresponding to the first element with a value of 1 in each column as the minimum hash value of the document in that column after this random permutation. Repeat the above process N times to generate an N-dimensional Minhash feature vector for each document.

[0065] To extract potentially similar text pairs, a segmented index is created by dividing the N-dimensional feature vector into segments and then building an inverted index. When the number of segments is sufficiently large, two similar texts have a high probability of sharing the same value in a certain segment. Texts with the same index value are then paired together as potentially similar text pairs.

[0066] The similarity of potentially similar text pairs is calculated as follows: the proportion of corresponding elements in two Minhash vectors that are equal is the similarity between the two texts. When the similarity exceeds a specified threshold, the two texts are considered duplicate texts.

[0067] Step S3: Data Quality Assessment: Perform quality grading on the deduplicated data to obtain assessment data; use a data quality assessment model to evaluate the quality of each document. The quality assessment is based on factors such as the accuracy, completeness, and consistency of the content.

[0068] Step S4: Data Filtering: Perform high-quality filtering on the evaluation data to obtain high-quality data; based on the data quality assessment results, set a quality score threshold S_M. Filter out high-quality documents that exceed this threshold and retain them in the dataset.

[0069] Step S5: Fine-tuning the chatGLM large model: Fine-tuning the chatGLM large model using the selected high-quality data. Fine-tuning methods such as LoRa are chosen to reduce model parameter updates and improve training efficiency. During fine-tuning, repeated training is performed on high-quality data, using optimization algorithms such as gradient descent to update model parameters. The chatGLM large model is a large language model used to generate natural language text. This model can be used for subsequent knowledge-based question answering, such as building a question-answering robot.

[0070] Step S6: Model Evaluation: Evaluate the fine-tuned model using the validation set. Evaluation metrics include accuracy, recall, and F1 score. Monitor model performance based on the evaluation results, and adjust data quality assessment and continue fine-tuning steps S3-S5 based on performance.

[0071] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0072] This application also provides a power system text data processing apparatus. It should be noted that the processing apparatus of this application can be used to execute the power system text data processing method provided in this application. This apparatus is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0073] The processing apparatus provided in the embodiments of this application will be described below.

[0074] Figure 4 This is a schematic diagram of a processing apparatus according to an embodiment of this application. Figure 4 As shown, the device includes: an acquisition unit 10 for acquiring power system text data; a control unit 20 connected to the acquisition unit 10 for analyzing and processing the text data; and a calculation unit 30 connected to the control unit 20 for repeatedly training the text data.

[0075] The processing device includes a processor and a memory. The aforementioned acquisition units are all stored as program units in the memory, and the processor executes the program units stored in the memory to achieve the corresponding functions. All of the aforementioned modules are located in the same processor; alternatively, the aforementioned modules may be located in different processors in any combination.

[0076] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters can improve the efficiency and quality of training large-scale power industry models, meeting the high standards of intelligent applications required by the power industry's regulatory question-and-answer system, thereby promoting the intelligent and automated process of power industry regulatory management.

[0077] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0078] This invention provides a computer-readable storage medium including a stored program, wherein the program controls the device containing the computer-readable storage medium to perform a method for processing power system text data during runtime.

[0079] This invention provides an electronic device, including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs at least the following steps:

[0080] S101: Obtain the text data corresponding to each document, where the text data is text-formatted data in the power system;

[0081] S102: Decompose the text data into multiple character sequences, construct a feature vector for each document based on the multiple character sequences, segment the feature vector to obtain multiple sub-feature vectors corresponding to the feature vector, establish an inverted index for the multiple sub-feature vectors, and combine the text data corresponding to the sub-feature vectors with the same value after the inverted index to obtain similar text data pairs of documents. The inverted index includes the sub-feature vectors and the text data corresponding to each sub-feature vector.

[0082] S103: Calculate the similarity between two text data in a similar text data pair. If the similarity is greater than a preset threshold, determine that the two text data in the similar text data pair are duplicate data, delete one of the duplicate data to obtain deduplicated text data, and perform quality assessment and data filtering on the deduplicated text data to obtain high-quality text data corresponding to the document.

[0083] The devices mentioned in this article can be servers, PCs, tablets, mobile phones, etc.

[0084] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the methods in various embodiments of this application:

[0085] S101: Obtain the text data corresponding to each document, where the text data is text-formatted data in the power system;

[0086] S102: Decompose the text data into multiple character sequences, construct a feature vector for each document based on the multiple character sequences, segment the feature vector to obtain multiple sub-feature vectors corresponding to the feature vector, establish an inverted index for the multiple sub-feature vectors, and combine the text data corresponding to the sub-feature vectors with the same value after the inverted index to obtain similar text data pairs of documents. The inverted index includes the sub-feature vectors and the text data corresponding to each sub-feature vector.

[0087] S103: Calculate the similarity between two text data in a similar text data pair. If the similarity is greater than a preset threshold, determine that the two text data in the similar text data pair are duplicate data, delete one of the duplicate data to obtain deduplicated text data, and perform quality assessment and data filtering on the deduplicated text data to obtain high-quality text data corresponding to the document.

[0088] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0089] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0090] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0091] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0092] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0093] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0094] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0095] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0096] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0097] As can be seen from the above description, the embodiments of this application achieve the following technical effects:

[0098] 1) The power system text data processing method of this application can automatically identify duplicate text data by decomposing the text data and determining the feature vector, and then calculating the similarity, thereby achieving the purpose of automatic deduplication and improving the efficiency of language text processing.

[0099] 2) The power system text data processing method of this application adopts a MinHash-based deduplication method and a high-quality data repetition training strategy to optimize model training data, thereby significantly improving the model's generalization ability. This invention focuses on the management of power industry regulatory training data, aiming to solve key technical problems such as data deduplication efficiency, objectivity and accuracy of data quality assessment, and enhancement of model generalization ability. By introducing the MinHash algorithm, data deduplication is automated, improving the efficiency and accuracy of data processing; an advanced data quality machine learning assessment model is used to comprehensively and objectively assess power industry regulatory data, ensuring the accuracy of the assessment results; through precise data quality assessment, high-quality data is selected, and a repetition training strategy is adopted to effectively improve the model's generalization ability.

[0100] 3) Use the MinHash algorithm to deduplicate data, thereby improving the purity of power industry regulatory data and the efficiency of model training;

[0101] 4) Utilize existing automated data quality assessment models to conduct a comprehensive quality assessment of each document to ensure the accuracy, completeness, and consistency of power industry regulatory data.

[0102] 5) Based on the data quality assessment results, select high-quality data and repeatedly use it for model training to improve the performance and generalization ability of the power industry institutional question-and-answer model.

[0103] 6) Improved Data Deduplication Efficiency: Given that existing methods for manually identifying and processing duplicate data are time-consuming, labor-intensive, and prone to errors, this invention introduces the MinHash algorithm to automate data deduplication, thereby improving the efficiency and accuracy of data processing.

[0104] 7) Objectivity and Accuracy of Data Quality Assessment: Existing data quality assessment methods that rely on manual evaluation suffer from excessive subjectivity, making automation and scalability difficult. This invention utilizes an advanced data quality machine learning assessment model to conduct a comprehensive and objective assessment of regulatory data in the power industry, ensuring the accuracy of the assessment results.

[0105] 8) Enhanced model generalization ability: In response to the problem of limited model generalization ability caused by inconsistent data quality in existing technologies, this invention selects high-quality data through accurate data quality assessment and adopts a repeated training strategy to effectively improve the model's generalization ability.

[0106] This application can significantly improve the efficiency and quality of large-scale model training in the power industry, meet the high standards required for intelligent applications in the power industry's institutional question-and-answer system, and thus promote the intelligent and automated process of institutional management in the power industry.

[0107] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for processing text data in a power system, characterized in that, include: Obtain the text data corresponding to each document, wherein the text data is text-formatted data in the power system; The text data is decomposed into multiple character sequences, and a feature vector corresponding to each document is constructed based on the multiple character sequences. The feature vector is segmented to obtain multiple sub-feature vectors corresponding to the feature vector. An inverted index of the multiple sub-feature vectors is established. The text data corresponding to the sub-feature vectors with the same value after the inverted index are combined as a group to obtain similar text data pairs of the document. The inverted index includes the sub-feature vectors and the text data corresponding to each sub-feature vector. Calculate the similarity between two text data in the similar text data pair. If the similarity is greater than a preset threshold, determine that the two text data in the similar text data pair are duplicate data, and delete one of the duplicate data to obtain deduplicated text data. Then, perform quality assessment and data filtering on the deduplicated text data to obtain high-quality text data corresponding to the document. Constructing a feature vector corresponding to each document based on multiple character sequences includes: constructing a two-dimensional word library from the multiple character sequences; encoding each dimension of the two-dimensional word library to obtain an initial feature vector corresponding to each document, wherein the value of the initial feature vector is 0 or 1; a determination step: generating a random sequence and sorting the initial feature vectors corresponding to multiple documents according to the random sequence; determining the row number corresponding to the first element with a value of 1 in each column as the minimum hash value of the document; repeating the determination step a target number of times to obtain the feature vector corresponding to the document, wherein the dimension of the feature vector is the target number of times. Establishing an inverted index for multiple sub-feature vectors includes: using the hash value corresponding to each sub-feature vector as a key and the text data corresponding to each sub-feature vector as the numerical value corresponding to the key, to obtain an inverted index for each sub-feature vector.

2. The processing method according to claim 1, characterized in that, Calculating the similarity between two text data in the similar text data pair includes: According to the formula Calculate the similarity between two text data pairs, where N represents the dimension of the similar text data pair. Representing text data and text data same.

3. The processing method according to claim 1, characterized in that, The deduplicated text data is subjected to quality assessment and data filtering to obtain high-quality text data corresponding to the document, including: Through formula Calculate accuracy indicators, among which, This indicates the number of text data items that violate the accuracy rules. This indicates the total number of text messages in the text data. This represents the total number of numerical information in the text data. This indicates the number of missing numerical information items. This indicates the accuracy index; Through formula Calculate the integrity index, where, Indicates the number of empty data. Indicates the total number of missing text messages. Indicates the proportion of empty data. This indicates the exact proportion of data. This represents the integrity index; The number of text data items that violate the consistency principle is obtained using the formula. Calculate the degree of satisfaction of the consistency index, where, Indicates the degree of satisfaction. This indicates the number of text data that violate the consistency principle, which at least includes the text data having the same format as a predetermined format. The quality assessment value of the deduplicated text data is calculated using a formula. If the quality assessment value is greater than a preset threshold, the deduplicated text data is determined to be high-quality text data.

4. The processing method according to claim 1, characterized in that, After obtaining the high-quality text data corresponding to the document, the method further includes: The high-quality text data is divided into a training set and a validation set; The initial language model is trained using the training set, and the parameters of the initial language model are updated using gradient descent to obtain the language model. The language model is validated using the validation set.

5. The processing method according to claim 1, characterized in that, After obtaining the text data corresponding to each document, the method further includes: The text data is cleaned to obtain valid text data; The valid text data is standardized to obtain the standardized text data.

6. A power system text data processing device, characterized in that, include: The acquisition unit is used to acquire the text data corresponding to each document, wherein the text data is text-formatted data in the power system; A determining unit is configured to decompose the text data into multiple character sequences, construct a feature vector corresponding to each document based on the multiple character sequences, segment the feature vector to obtain multiple sub-feature vectors corresponding to the feature vector, establish an inverted index of the multiple sub-feature vectors, and combine the text data corresponding to the sub-feature vectors with the same value after the inverted index as a group to obtain similar text data pairs of the document, wherein the inverted index includes the sub-feature vectors and the text data corresponding to each sub-feature vector; A filtering unit is used to calculate the similarity between two text data in the similar text data pair. If the similarity is greater than a preset threshold, the two text data in the similar text data pair are determined to be duplicate data, and one of the duplicate data is deleted to obtain deduplicated text data. The deduplicated text data is then subjected to quality assessment and data filtering to obtain high-quality text data corresponding to the document. The processing device is further configured to construct a two-dimensional word library from multiple character sequences, encode each dimension of the two-dimensional word library to obtain an initial feature vector corresponding to each document, wherein the value of the initial feature vector is 0 or 1; a determination step: generating a random sequence, and sorting the initial feature vectors corresponding to multiple documents according to the random sequence, determining the row number corresponding to the first element with a value of 1 in each column as the minimum hash value of the document; repeating the determination step a target number of times to obtain the feature vector corresponding to the document, wherein the dimension of the feature vector is the target number of times. The processing device is further configured to use the hash value corresponding to each sub-feature vector as a key value and the text data corresponding to each sub-feature vector as the numerical value corresponding to the key value to obtain an inverted index for each sub-feature vector.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device containing the computer-readable storage medium to perform the power system text data processing method according to any one of claims 1 to 5.

8. An electronic device, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including a method for processing power system text data as described in any one of claims 1 to 5.

Images