An energy data cleaning method and device based on a subword vocabulary

Abstract

This invention discloses an energy data cleaning method and apparatus based on a subword vocabulary. The method includes: after acquiring energy data, converting each text in the energy data into a first text embedding matrix using a preset subword vocabulary, wherein the subword vocabulary is a character-level text list; converting the first text embedding matrix into a comprehensive matrix conforming to a model input format; and inputting the comprehensive matrix into a preset neural network model for cleaning processing to obtain cleaned data. This invention can extract sentences and sequence numbers from energy data using a character-level subword vocabulary and convert them into an input matrix for a model. The input matrix is ​​then input into the model for cleaning processing to obtain cleaned data. Because the subword vocabulary is at the character level, it is small in scale and the conversion process does not lose any information, thus saving the amount of data required for subsequent cleaning calculations, shortening the processing time, and improving the efficiency of cleaning data processing.

Description

Technical Field

[0001] This invention relates to the technical field of data cleaning and processing, and in particular to a method and apparatus for cleaning energy data based on a subword vocabulary. Background Technology

[0002] With the continuous improvement of information technology, industrial technology, and production capacity, the sheer scale, rapid growth, diverse types, and significant structural differences of energy economic data have become a major issue that energy companies must address. How to combine massive amounts of energy economic big data with current high technologies to construct an efficient and green energy economic big data cleaning model has become a crucial issue that energy sectors and enterprises urgently need to solve.

[0003] Due to the large volume and diverse types of energy data, which includes data of varying quality, inconsistencies, incompleteness, and non-compliance, data redundancy and other problems arise. To address these legacy data quality issues, it is necessary to transform the massive amounts of stored data and implement "data cleaning." One common data cleaning method involves: first, acquiring energy data information, establishing a data cleaning rule base, and setting up a data cleaning execution strategy; then, based on the cleaning execution strategy and the rule base, cleaning the energy data to identify erroneous, invalid, and missing data, ensuring data consistency and reliability.

[0004] However, the commonly used methods mentioned above have the following technical problems: As time goes by, the accumulated energy data gradually increases and the types involved become more extensive, which increases the time required to build the rule base and clean the execution strategy, reducing the processing efficiency; moreover, the rule base and execution strategy are huge, and the data called during the calculation also increases further, which will further increase the calculation time and reduce the processing efficiency of the cleansing calculation. Summary of the Invention

[0005] This invention proposes an energy data cleaning method and apparatus based on a subword vocabulary. The method can use a character-level subword vocabulary to transform energy data, and then input the transformation result into a preset model for cleaning to obtain cleaned data. Since the subword vocabulary is at the character level, it is not only small in scale, but the transformation process does not lose the amount of information in the data, thereby saving the amount of data for subsequent cleaning calculations, shortening the processing time, and improving the processing efficiency of the cleaned data.

[0006] A first aspect of this invention provides an energy data cleaning method based on a Subword vocabulary, the method comprising:

[0007] After acquiring energy data, each text in the energy data is converted into a first text embedding matrix using a preset subword vocabulary, where the subword vocabulary is a text list at the character granularity level.

[0008] The first text embedding matrix is ​​converted into a comprehensive matrix that conforms to the model input format;

[0009] The comprehensive matrix is ​​input into a preset neural network model for cleaning to obtain cleaned data.

[0010] In one possible implementation of the first aspect, the energy data consists of N texts, each text containing at least one sentence, where N is greater than or equal to 1.

[0011] The step of converting each text in the energy data into a first text embedding matrix using a preset subword vocabulary includes:

[0012] Identify the sentence corresponding to each text in the energy data and the sequence number contained in each sentence, and use the preset Subword vocabulary to convert the sentence corresponding to each text and the sequence number contained in each sentence into vectors, respectively obtaining the first sentence vector and the first sequence number vector corresponding to each text;

[0013] Add the first sentence vector to the first sequence number vector to obtain the first summed vector for each text.

[0014] The first text vector is obtained by concatenating N of the first summed vectors. The first text vector is then subjected to a matrix embedding operation to obtain the first text embedding matrix.

[0015] In one possible implementation of the first aspect, the matrix embedding operation specifically involves embedding a (1*n format) text vector into an n*m matrix.

[0016] The embedding calculation is shown in the following formula:

[0017] .

[0018] In one possible implementation of the first aspect, the construction method of the preset Subword vocabulary includes:

[0019] Input the first text sample data and perform character-level segmentation on the first text sample data to obtain the first segmented character;

[0020] Count the frequency of each first segmenting character and its adjacent characters to form a character pair, and obtain multiple frequency values ​​of the first character pair;

[0021] Select several target first character pair frequency values ​​that are greater than a preset frequency value from the plurality of first character pair frequency values, and use the character pairs corresponding to each target first character pair frequency value to construct and generate a Subword vocabulary.

[0022] In one possible implementation of the first aspect, the preset neural network model is a BERT model obtained by training the model using sample data about energy information;

[0023] The model training process includes:

[0024] Enter the second text sample data, perform character-level segmentation on the second text sample data to obtain the second segmentation character. The second text sample data consists of M texts, where M is greater than or equal to 1.

[0025] Count the frequency of each second dividing character forming a character pair with its adjacent character to obtain multiple frequency values ​​of second character pairs;

[0026] Select several target second character pair frequency values ​​that are greater than a preset frequency value from the plurality of second character pair frequency values, and use the character pairs corresponding to each target second character pair frequency value to construct and generate a training vocabulary;

[0027] Based on the training vocabulary, extract the second sentence vector and the second sequence number vector corresponding to each text from the second text sample data;

[0028] Add the second sentence vector to the second sequence number vector to obtain the second summed vector for each text.

[0029] The second text vector is obtained by concatenating M of the second summed vectors. The second text vector is then subjected to a matrix embedding operation to obtain the second text embedding matrix.

[0030] The second text embedding matrix is ​​converted into a training matrix that conforms to the model input format, and the training matrix is ​​used to train the BERT model.

[0031] In one possible implementation of the first aspect, the BERT model includes: an encoding module, an olive-shaped feedforward neural network, and a softmax processing module connected in sequence;

[0032] There are n encoding modules, and these n encoding modules are connected in series.

[0033] In one possible implementation of the first aspect, the encoding module includes: a linear layer, a residual connection layer, a normalization layer, and a feedforward neural network;

[0034] The feedforward neural network consists of multiple linear layers plus activation functions, where the activation functions are shown in the following equation:

[0035] ;

[0036] The feedforward neural network is represented as:

[0037] ;

[0038] The internal processing flow of the input data in the encoding module includes:

[0039] The input data is transformed by the linear layer, the residual connection layer, and the normalization layer to obtain the linear layer matrix, the residual connection layer matrix, and the normalization layer matrix, respectively.

[0040] The attention score is calculated based on the linear layer matrix, the residual connection layer, and the normalized layer matrix;

[0041] The attention score is input into the residual connection layer for the first forward and reverse propagation, and the result of the first forward and reverse propagation is input into the normalization layer for the first integration matrix to obtain the first integration result.

[0042] The first integration result is input into the feedforward neural network, and the output of the feedforward neural network is input into the residual connection layer for a second forward and reverse propagation. The second forward and reverse propagation is then input into the normalization layer for a second integration matrix.

[0043] In one possible implementation of the first aspect, the specific calculation formulas for the matrix transformation of the input data by the linear layer, the residual connection layer, and the normalization layer are shown below:

[0044] ;

[0045] Where Q is the linear layer matrix, K is the residual connection layer, and V is the normalized layer matrix. For input data, The weights of the linear layer, The weights of the residual connection layer are... The weights of the normalization layer;

[0046] The formula for calculating the attention score is as follows:

[0047] ;

[0048] Where T represents transpose; It is a constant.

[0049] In one possible implementation of the first aspect, converting the first text embedding matrix into a comprehensive matrix conforming to the model input format includes:

[0050] The first text embedding matrix is ​​added to the preset position embedding matrix to obtain a comprehensive matrix. The preset position embedding matrix is ​​obtained by acquiring the third text sample data, segmenting the third text sample data, inputting it into a preset neural network model for pre-training, and then completing the training. The preset position embedding matrix serves as a learnable parameter of the preset neural network model.

[0051] A second aspect of the present invention provides an energy data cleaning apparatus based on a Subword vocabulary, the apparatus comprising:

[0052] The text conversion module is used to convert each text of the energy data into a first text embedding matrix after acquiring the energy data using a preset Subword vocabulary, wherein the Subword vocabulary is a text list at the character granularity level.

[0053] The matrix generation module is used to convert the first text embedding matrix into a comprehensive matrix that conforms to the model input format;

[0054] The cleaning module is used to input the comprehensive matrix into a preset neural network model for cleaning processing to obtain cleaned data.

[0055] Compared with existing technologies, the energy data cleaning method and apparatus based on a subword vocabulary provided in this invention have the following advantages: This invention can extract energy data sentences and serial numbers using a character-level subword vocabulary, convert the extracted sentences and serial numbers into an input matrix for a model, and then input the input matrix into a preset model for cleaning to obtain cleaned data. Since the subword vocabulary is at the character level, it is not only small in scale, but the conversion process does not lose the amount of information in the data, thereby saving the amount of data for subsequent cleaning calculations, shortening the processing time, and improving the processing efficiency of the cleaned data. Attached Figure Description

[0056] Figure 1 This is a flowchart illustrating an energy data cleaning method based on a Subword vocabulary according to an embodiment of the present invention.

[0057] Figure 2 This is a schematic diagram of the structure of a BERT model provided in an embodiment of the present invention;

[0058] Figure 3 This is a flowchart illustrating the operation of an energy data cleaning method based on a Subword vocabulary, provided by an embodiment of the present invention.

[0059] Figure 4 This is a schematic diagram of the structure of an energy data cleaning device based on a Subword vocabulary provided in an embodiment of the present invention. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] With the continuous improvement of information technology, industrial technology, and production capacity, the sheer scale, rapid growth, diverse types, and significant structural differences of energy economic data have become a major issue that energy companies must address. How to combine massive amounts of energy economic big data with current high technologies to construct an efficient and green energy economic big data cleaning model has become a crucial issue that energy sectors and enterprises urgently need to solve.

[0062] Due to the large volume and diverse types of energy data, which includes data of varying quality, inconsistencies, incompleteness, and non-compliance, data redundancy and other problems have arisen. To address these legacy data quality issues, it is necessary to transform the massive amounts of stored data and implement a "data cleaning" process.

[0063] In the current research field of energy economic big data cleaning models, most researchers focus on database technology, machine learning methods such as clustering, association analysis, and conditional functional dependencies—methods related to statistical machine learning. For example, Meng Xiangping et al. proposed an anomaly data cleaning model based on Hadoop, while Qu Chaoyang et al. proposed an energy big data cleaning model based on the Spark framework. Both of these cleaning models do not deviate from the technical framework of distributed file systems. Lin Nügui et al. proposed a big data cleaning method based on improved K-means, while Zou Wang et al. proposed a clustering-based data cleaning algorithm, introducing a density-based clustering algorithm, which also remains within the scope of clustering algorithms. Huangfu Hancong et al. proposed an automatic repair method for order-dependent power data, extracting features from the order attributes of power data and automatically repairing missing data. Yan Yingjie et al. proposed a big data cleaning method for transmission equipment status based on time series analysis, using iterative testing to fit time series data. Furthermore, a paper by Kuang Junqian et al. published in the Journal of Electronic Information this year proposed an anomaly cleaning algorithm based on deep learning. It uses the Iterative Threshold Shrinkage (ISTA) algorithm to solve the data cleaning problem in the Internet of Things. However, the model data does not consider textual text, but only digital text.

[0064] To process energy data in text format, one common data cleaning method is to first acquire energy data information, establish an energy data cleaning rule base, and set an energy data cleaning execution strategy; based on the cleaning execution strategy and the cleaning rule base, the energy data is cleaned, thereby identifying erroneous, invalid, and missing data, ensuring data consistency and reliability.

[0065] However, the commonly used methods mentioned above have the following technical problems: As time goes by, the accumulated energy data gradually increases and the types involved become more extensive, which increases the time required to build the rule base and clean the execution strategy, reducing the processing efficiency; moreover, the rule base and execution strategy are huge, and the data called during the calculation also increases further, which will further increase the calculation time and reduce the processing efficiency of the cleansing calculation.

[0066] Moreover, the data cleaning methods mentioned above are based, to varying degrees, on traditional file systems or traditional, improved statistical machine learning methods. They fail to consider the rapid growth and changing trends of current energy and economic big data. While past statistical machine learning methods might handle small amounts of energy and economic data, they struggle with today's massive and diverse energy and economic big data, resulting in significantly reduced computational efficiency. Processing the same amount of data is computationally time-consuming and extremely wasteful of resources, making it inefficient for handling massive, complex, multi-source, and heterogeneous energy and economic data. Furthermore, energy and economic data involves numerous technical challenges such as multimodal and multi-tasking characteristics, all of which are difficult for past methods to address.

[0067] To address the aforementioned issues, the following specific embodiments will provide a detailed description and explanation of an energy data cleaning method based on a Subword vocabulary provided in this application.

[0068] Reference Figure 1 The diagram shows a flowchart of an energy data cleaning method based on a subword vocabulary according to an embodiment of the present invention.

[0069] As an example, the energy data cleaning method based on the Subword vocabulary may include:

[0070] S11. After acquiring energy data, each text in the energy data is converted into a first text embedding matrix using a preset Subword vocabulary, wherein the Subword vocabulary is a text list at the character granularity level.

[0071] In one embodiment, energy data can be energy-related text data. This could include data from various equipment of energy companies, energy production capacity figures, and related descriptions of energy content. For example, a sample data entry from the International Energy Agency could be: "Global coal-fired generation reached an all-time high in 2021, pushing CO2 emissions from coal power plants to record levels. Despite increased calls from governments and the private sector to phase down or transition away from coal, it accounted for over one-third of total electricity generation. These trends are not on track with the Net Zero Emissions by 2050 Scenario, which calls for an annual average 9% reduction in unabated coal-fired generation between 2022 and 2030 and a complete phaseout by 2040."

[0072] In one embodiment, the preset subword vocabulary can be a list containing multiple text characters or words. The characters and words in this list can be used to extract relevant or useful information from the text of energy data, converting it into corresponding vectors, and then cleaning the data based on these vectors. Furthermore, the subword vocabulary stores characters at the character-level granularity, ensuring that subsequent information extraction also involves segmenting the energy data at the character-level granularity. This allows subsequent operations to retain as much information as possible from the energy data, avoiding omissions and ensuring no data loss. Simultaneously, the low character-level storage results in a smaller subword vocabulary, saving computational resources during subsequent calculations and significantly reducing the computational load during data cleaning. This greatly improves the efficiency of subsequent data cleaning processing while ensuring accuracy.

[0073] In one embodiment, the construction of the preset subword vocabulary includes the following steps:

[0074] S21. Input the first text sample data and perform character-level segmentation on the first text sample data to obtain the first segmented character.

[0075] In one embodiment, the first text sample data is similar to the energy data described above, and is energy-related text data. Specifically, it could be like the example of the English sample data from the International Energy Agency mentioned above.

[0076] Next, the first text sample data needs to be segmented at the character level. "Character level" here refers to segmentation at the level of each Chinese character, each letter, or each number. Using the aforementioned English sample data from the International Energy Agency as an example, taking "reached" as an example, it needs to be segmented into individual characters: "r", "e", "a", "c", "h", "e", and "d". All other characters are segmented similarly.

[0077] S22. Count the frequency of each first segmentation character and its adjacent character to form a character pair, and obtain multiple first character pair frequency values.

[0078] The frequency values ​​of each segmented character and its adjacent characters can be counted to obtain multiple frequency values ​​of the first character pair.

[0079] S23. Select several target first character pair frequency values ​​that are greater than a preset frequency value from the plurality of first character pair frequency values, and use the character pairs corresponding to each target first character pair frequency value to construct and generate a Subword vocabulary.

[0080] You can filter target first character pair frequency values ​​that are greater than a preset frequency value from multiple first character pair frequency values. The character pairs corresponding to several target first character pair frequency values ​​that are greater than the preset frequency value are the character pairs that appear frequently. These character pairs can be merged to build and generate a Subword vocabulary.

[0081] In an alternative embodiment, when counting frequently occurring character pairs, it may be considered that some words share the same prefix, infix, or suffix.

[0082] In the samples above, "generation," "transition," and "reduction" all share the same suffix "tion," and "reached," "unbated," and "increased" all share the same suffix "ed." This step is used to extract common affixes. The frequencies of the segmented letters and their adjacent letters are screened and counted. Letters are combined to form new affixes, and the frequency of these affixes is counted again until the affixes converge. After extracting common affixes from the words, such as "tion," "ed," and "al," a subword vocabulary is constructed using these affixes.

[0083] When constructing and updating the subword vocabulary, the text is segmented at the character level, and the subword vocabulary is constructed and updated based on the frequency of character pairs formed by characters and their preceding and following characters. Character pairs with high frequency are included, while those with low frequency are not. This makes the subword vocabulary smaller, saving computing resources without losing information, which in turn greatly reduces the amount of computation required for subsequent data cleaning and improves the accuracy of data cleaning.

[0084] In one embodiment, the energy data consists of N texts, each text containing at least one sentence, where N is greater than or equal to 1.

[0085] Taking the English sample data from the International Energy Agency as an example, if the energy data has only one English sample data, then the energy data contains one text; if the energy data has three or three English sample data, then the energy data contains three texts.

[0086] As an example, step S11 may include the following sub-steps:

[0087] S111. Identify the sentence corresponding to each text in the energy data and the sequence number contained in each sentence, and use the preset Subword vocabulary to convert the sentence corresponding to each text and the sequence number contained in each sentence into vectors, so as to obtain the first sentence vector and the first sequence number vector corresponding to each text respectively.

[0088] Taking the English sample data from the International Energy Agency as an example, due to the large number of words, let's take "generation", "transiton", "reached", and "increased" as examples: The constructed subword list is: {"genera": 0, "transi": 1, "reach": 2, "increas": 3, "tion": 4, "ed": 5, "*": 6}, where "*" represents the affixes that need to be cleaned. The text of the above four words, "generation transition reached increased", can be represented as a vector: [0, 4, 1, 4, 2, 5, 3, 5].

[0089] It should be noted that if a word contains an error and needs to be cleaned, the value at the error location needs to be replaced with the value represented by the "*" sign. For example, if the word "generation" is incorrect and becomes "generasion", the word needs to be cleaned, and the original word's [0, 4] vector becomes a [0, 6] vector.

[0090] S112. Add the first sentence vector to the first sequence number vector to obtain the first summed vector corresponding to each text.

[0091] For example, energy data consists of two texts, each containing one sentence. The first sentence vector for each text can be [0, 4, 1, 4] and [2, 5, 3, 5], respectively. The first sequence vector for each sentence is [0, 0, 0, 0] and [1, 1, 1, 1]. The first sentence vector and its corresponding first sequence vector for each text can be added together to obtain the first sum vector for each text. Specifically, the first sum vectors for the two texts are [0, 4, 1, 4] and [3, 6, 4, 6], respectively.

[0092] S113. Concatenate N of the first summed vectors to obtain a first text vector, and perform a matrix embedding operation on the first text vector to obtain a first text embedding matrix.

[0093] Specifically, following the example above, the first summed vectors of the two texts can be concatenated to obtain the integrated first text vector: [0, 4, 1, 4, 3, 6, 4, 6].

[0094] It should be noted that if each text contains multiple sentences, the vector of each first sentence can be added to its corresponding first index vector to obtain the vector for each sentence; these vectors can then be concatenated to form the vector for each text. Finally, the vectors for multiple texts can be concatenated to form the first text vector corresponding to the energy data.

[0095] Alternatively, the above operation method can also concatenate the vectors of multiple sentences for model training. In this way, the model can learn one sentence at a time or multiple sentences at a time, which greatly improves the training efficiency of the model and the efficiency of data cleaning.

[0096] In another optional embodiment, if the energy data contains only one text, and the text contains only the first sentence, then the first sentence vector and the first sequence number vector corresponding to the text can be added together, and the vector after addition can be used as the first text vector.

[0097] Before matrix embedding, segmentation was performed to form encodings (i.e., vectors). Depending on the segmentation process, a sentence can be used as input by constructing a sentence vector and an index vector, or multiple sentences can be concatenated (i.e., sentence vectors are concatenated). In this way, the model can learn one sentence at a time or multiple sentences at a time, which greatly improves the training efficiency of the model and also greatly improves the efficiency of data cleaning.

[0098] After generating the first text vector, a matrix transformation operation can be performed on the first text vector so that the vector can be transformed into a matrix.

[0099] In one implementation, the first text vector can be transformed into a matrix through matrix embedding operations.

[0100] In one embodiment, the matrix embedding operation specifically involves embedding a (1*n format) text vector into an n*m matrix;

[0101] The embedding calculation is shown in the following formula:

[0102] .

[0103] Optionally, a matrix embedding operation can be performed on the text vectors to embed the text vectors of each sentence into a matrix, thereby obtaining the text embedding matrix of each sentence.

[0104] S12. Convert the first text embedding matrix into a comprehensive matrix that conforms to the model input format.

[0105] To facilitate subsequent cleaning processes, the first text embedding matrix can be further processed to make it meet the requirements of the subsequent cleaning process.

[0106] In one embodiment, the processing of the first text embedding matrix may include format conversion, adding, deleting, or modifying the matrix, or adding a new matrix.

[0107] In an optional embodiment, step S12 may include the following sub-steps:

[0108] S121. Add the first text embedding matrix to the preset position embedding matrix to obtain a comprehensive matrix. The preset position embedding matrix is ​​obtained by acquiring the third text sample data, segmenting the third text sample data, inputting it into a preset neural network model for pre-training, and then completing the training. The preset position embedding matrix serves as a learnable parameter of the preset neural network model.

[0109] Specifically, text content similar to energy data or the first text sample data can be entered as the third text sample data. Specifically, the English sample data of the International Energy Agency mentioned above can be referred to. Then, the third text sample data can be segmented at the character level. This operation method can be the same as step S21.

[0110] Next, the characters after segmentation of the third text sample data are directly input into the preset neural network model for pre-training. After training, the matrix of the preset neural network model is the position embedding matrix.

[0111] Adding the first text embedding matrix and the position embedding matrix together yields the comprehensive matrix.

[0112] In one implementation, a ready-made BERT model can be downloaded directly from the internet, as it contains some pre-existing parameters. For the downloaded BERT model, the location embedding matrix needs to be set as a learnable parameter.

[0113] It's important to note that for text data, the positional relationships between different words and between different positions are crucial. A good text model needs to capture accurate positional information. Similarly, when cleaning energy and economic big data, incorporating positional information can improve the cleaning effect of deep learning models; therefore, this step involves positional embedding.

[0114] The positional embeddings of the BERT model are learned during pre-training, not calculated through a manually set encoding method. In other words, the positional embeddings of the BERT model can be learned, rather than being fixed.

[0115] The specific method is to set all the position embedding matrices as learnable parameters. After pre-training, a position embedding matrix can be obtained in the BERT model.

[0116] Pre-training the BERT model yields an initial position embedding matrix. However, this position embedding matrix is ​​not fixed; it remains a learnable parameter during BERT model training. This allows for updating relevant parameters during subsequent BERT model training, greatly increasing the flexibility of the BERT model.

[0117] S13. Input the comprehensive matrix into a preset neural network model for cleaning processing to obtain cleaned data.

[0118] The composite matrix can be input into a preset neural network model, which can then output cleaned data.

[0119] Alternatively, the comprehensive matrix of each sentence can be input into a preset neural network model, so that the model can output cleaned data.

[0120] In one embodiment, the preset neural network model is a BERT model obtained by training the model using sample data about energy information.

[0121] In an optional embodiment, energy data or first text sample data as described in the above steps can be obtained, and then the data can be used to train the model to obtain the BERT model.

[0122] In one embodiment, the model training process includes the following steps:

[0123] S31. Input the second text sample data, and perform character-level segmentation on the second text sample data to obtain the second segmentation character. The second text sample data consists of M texts, where M is greater than or equal to 1.

[0124] Specifically, the second text sample data can be similar to the energy data or the text content of the first text sample data. For example, the English sample data of the International Energy Agency can be referred to above. Then, the second text sample data can be segmented at the character level. This operation method can be the same as step S21, and the parsed content of the above steps can be operated.

[0125] S32. Count the frequency of each second dividing character and its adjacent character to form a character pair, and obtain multiple frequency values ​​of second character pairs.

[0126] In one embodiment, step S32 is the same as step S22. To avoid repetition, it will not be described again here. For details, please refer to the analysis of step S22 above.

[0127] S33. Select several target second character pair frequency values ​​that are greater than a preset frequency value from the plurality of second character pair frequency values, and use the character pairs corresponding to each target second character pair frequency value to construct a training vocabulary.

[0128] In one embodiment, step S33 is the same as step S23. To avoid repetition, it will not be described again here. For details, please refer to the analysis of step S23 above.

[0129] The constructed training vocabulary can also be a Subword vocabulary.

[0130] S34. Extract the second sentence vector and the second sequence number vector corresponding to each text from the second text sample data according to the training vocabulary.

[0131] In one embodiment, step S34 is the same as step S111. To avoid repetition, it will not be described again here. For details, please refer to the analysis of step S111 above.

[0132] S35. Add the second sentence vector to the second sequence number vector to obtain the second summed vector corresponding to each text.

[0133] In one embodiment, step S35 is the same as step S112. To avoid repetition, it will not be described again here. For details, please refer to the analysis of step S112 above.

[0134] S36. Concatenate M of the second summed vectors to obtain the second text vector, and perform matrix embedding operation on the second text vector to obtain the second text embedding matrix.

[0135] In one embodiment, step S36 is the same as step S113. To avoid repetition, it will not be described again here. For details, please refer to the analysis of step S113 above.

[0136] S37. Convert the second text embedding matrix into a training matrix that conforms to the model input format, and use the training matrix to train the BERT model.

[0137] In one embodiment, step S37 is the same as step S121. To avoid repetition, it will not be described again here. For details, please refer to the analysis of step S121 above.

[0138] Specifically, the second text embedding matrix can be added to the preset position embedding matrix to obtain the training matrix. The training matrix can then be input into the BERT model for model training, and the BERT model can be fine-tuned using the training matrix to obtain the preset neural network model.

[0139] Reference Figure 2 The diagram shows a structural schematic of the BERT model provided in an embodiment of the present invention.

[0140] Optionally, the specific structure of the BERT model is as follows: Figure 2 As shown, it includes an encoding module, an olive-shaped feedforward neural network, and a softmax processing module connected in sequence. There are n encoding modules, which are connected in series.

[0141] In one embodiment, the encoding module includes: a linear layer, a residual connection layer, a normalization layer, and a feedforward neural network;

[0142] The feedforward neural network consists of multiple linear layers plus activation functions, where the activation functions are shown in the following equation:

[0143] ;

[0144] The feedforward neural network is represented as:

[0145] ;

[0146] The internal processing flow of the input data in the encoding module may include the following steps:

[0147] S41. The input data is transformed by the linear layer, the residual connection layer and the normalization layer to obtain the linear layer matrix, the residual connection layer matrix and the normalization layer matrix respectively.

[0148] In one embodiment, the specific calculation formulas for the matrix transformation of the input data by the linear layer, the residual connection layer, and the normalization layer are shown below:

[0149] ;

[0150] Where Q is the linear layer matrix, K is the residual connection layer, and V is the normalized layer matrix. For input data, The weights of the linear layer, The weights of the residual connection layer are... The weights of the normalized layer.

[0151] S42. Calculate the attention score based on the linear layer matrix, the residual connection layer, and the normalized layer matrix.

[0152] In one embodiment, the attention score is calculated using the following formula:

[0153] ;

[0154] Where T represents transpose; It is a constant.

[0155] Its function is to reduce The value of is adjusted so that the Softmax function can be pushed into a region with a moderate gradient, which is beneficial for subsequent backpropagation.

[0156] S43. Input the attention score into the residual connection layer for the first forward and reverse propagation, and input the result of the first forward and reverse propagation into the normalization layer for the first integration matrix to obtain the first integration result.

[0157] The first forward and backward propagation operation of the residual connection layer preserves previous information during forward propagation and solves the problems of vanishing and exploding gradients during backward propagation. The first layer normalization operation integrates the matrices, making the model more likely to converge.

[0158] The specific calculation formula for the first integrated matrix operation is as follows:

[0159] .

[0160] S44. The first integration result is input into the feedforward neural network, and the output result of the feedforward neural network is input into the residual connection layer for the second forward and reverse propagation. The second forward and reverse propagation is input into the normalization layer for the second integration matrix.

[0161] Specifically, the output of the feedforward neural network can be input into another residual connection layer and a normalization layer, and then propagated in both directions for the second time through the residual connection layer. The normalization layer performs a second integration matrix operation, and finally outputs to the next encoding module or an olive-shaped feedforward neural network.

[0162] The processing method for the residual connection layer and the normalization layer in this step is the same as in step S43.

[0163] Because the BERT model uses multiple stacked encoding modules, steps S41-S44 need to be repeated N times, with the output of each step serving as the input for the next step, to obtain the output. Output Dimensions and starting input They have the same dimensions.

[0164] Output The input will be an olive-shaped feedforward neural network. This olive-shaped feedforward neural network consists of three linear layers, the first of which will... The first linear layer maps the matrix from Space1 to the high-dimensional space Space2, where data more suitable for the cleaning task can be learned. The second linear layer maps the matrix from Space2 back to the high-dimensional space Space2. The space in which it is located.

[0165] Finally, the output of the third linear layer of the olive-shaped feedforward neural network is passed through the Softmax layer of the softmax processing module to obtain the final output. Then output The cross-entropy is calculated with the correct text labels, and the parameters of the BERT model are updated using the backpropagation algorithm to finally obtain the mature BERT model.

[0166] Reference Figure 3 The diagram shows an operation flowchart of an energy data cleaning method based on a Subword vocabulary provided by an embodiment of the present invention.

[0167] Specifically, subword processing can be performed first to generate a subword vocabulary. Then, embedding operations can be performed using the subword vocabulary. Specifically, the corresponding characters can be extracted from the training text data using the subword vocabulary to form a text embedding matrix, and this text embedding matrix can be added to a preset positional embedding matrix. The resulting matrix can then be used to train the cleaned model. Finally, when obtaining the text data to be cleaned, the corresponding characters can be extracted from the cleaned text data again using the subword vocabulary to form a text embedding matrix. This text embedding matrix can then be added to a preset positional embedding matrix as input data for the model. This input data is then fed into the trained BERT model for cleaning processing to obtain the cleaned data.

[0168] In this embodiment, the present invention provides an energy data cleaning method based on a subword vocabulary. Its advantages are as follows: the present invention can extract sentences and serial numbers from energy data using a character-level subword vocabulary, convert the extracted sentences and serial numbers into an input matrix for a model, and then input the input matrix into a preset model for cleaning to obtain cleaned data. Since the subword vocabulary is at the character-level, it is not only small in scale but also does not lose information during the conversion process, thus saving the amount of data required for subsequent cleaning calculations, shortening the processing time, and improving the efficiency of cleaning data processing.

[0169] This invention also provides an energy data cleaning device based on a Subword vocabulary, see [link to related document]. Figure 4 The diagram shows a schematic representation of an energy data cleaning device based on a Subword vocabulary according to an embodiment of the present invention.

[0170] As an example, the energy data cleaning device based on the Subword vocabulary may include:

[0171] The text conversion module 401 is used to convert each text of the energy data into a first text embedding matrix using a preset Subword vocabulary after acquiring the energy data. The Subword vocabulary is a text list at the character granularity level.

[0172] The matrix generation module 402 is used to convert the first text embedding matrix into a comprehensive matrix that conforms to the model input format;

[0173] The cleaning processing module 403 is used to input the comprehensive matrix into a preset neural network model for cleaning processing to obtain cleaned data.

[0174] Optionally, the energy data consists of N texts, each text containing at least one sentence, where N is greater than or equal to 1;

[0175] The text conversion module is also used for:

[0176] Identify the sentence corresponding to each text in the energy data and the sequence number contained in each sentence, and use the preset Subword vocabulary to convert the sentence corresponding to each text and the sequence number contained in each sentence into vectors, respectively obtaining the first sentence vector and the first sequence number vector corresponding to each text;

[0177] Add the first sentence vector to the first sequence number vector to obtain the first summed vector for each text.

[0178] The first text vector is obtained by concatenating N of the first summed vectors. The first text vector is then subjected to a matrix embedding operation to obtain the first text embedding matrix.

[0179] Optionally, the matrix embedding operation specifically involves embedding a (1*n format) text vector into an n*m matrix;

[0180] The embedding calculation is shown in the following formula:

[0181] .

[0182] Optionally, the construction method of the preset Subword vocabulary includes:

[0183] Input the first text sample data and perform character-level segmentation on the first text sample data to obtain the first segmented character;

[0184] Count the frequency of each first segmenting character and its adjacent characters to form a character pair, and obtain multiple frequency values ​​of the first character pair;

[0185] Select several target first character pair frequency values ​​that are greater than a preset frequency value from the plurality of first character pair frequency values, and use the character pairs corresponding to each target first character pair frequency value to construct and generate a Subword vocabulary.

[0186] Optionally, the preset neural network model is a BERT model obtained by training the model using sample data about energy information;

[0187] The model training process includes:

[0188] Enter the second text sample data, perform character-level segmentation on the second text sample data to obtain the second segmentation character. The second text sample data consists of M texts, where M is greater than or equal to 1.

[0189] Count the frequency of each second dividing character forming a character pair with its adjacent character to obtain multiple frequency values ​​of second character pairs;

[0190] Select several target second character pair frequency values ​​that are greater than a preset frequency value from the plurality of second character pair frequency values, and use the character pairs corresponding to each target second character pair frequency value to construct and generate a training vocabulary;

[0191] Based on the training vocabulary, extract the second sentence vector and the second sequence number vector corresponding to each text from the second text sample data;

[0192] Add the second sentence vector to the second sequence number vector to obtain the second summed vector for each text.

[0193] The second text vector is obtained by concatenating M of the second summed vectors. The second text vector is then subjected to a matrix embedding operation to obtain the second text embedding matrix.

[0194] The second text embedding matrix is ​​converted into a training matrix that conforms to the model input format, and the training matrix is ​​used to train the BERT model.

[0195] Optionally, the BERT model includes: an encoding module, an olive-shaped feedforward neural network, and a softmax processing module connected in sequence;

[0196] There are n encoding modules, and these n encoding modules are connected in series.

[0197] Optionally, the encoding module includes: a linear layer, a residual connection layer, a normalization layer, and a feedforward neural network;

[0198] The feedforward neural network consists of multiple linear layers plus activation functions, where the activation functions are shown in the following equation:

[0199] ;

[0200] The feedforward neural network is represented as:

[0201] ;

[0202] The internal processing flow of the input data in the encoding module includes:

[0203] The input data is transformed by the linear layer, the residual connection layer, and the normalization layer to obtain the linear layer matrix, the residual connection layer matrix, and the normalization layer matrix, respectively.

[0204] The attention score is calculated based on the linear layer matrix, the residual connection layer, and the normalized layer matrix;

[0205] The attention score is input into the residual connection layer for the first forward and reverse propagation, and the result of the first forward and reverse propagation is input into the normalization layer for the first integration matrix to obtain the first integration result.

[0206] The first integration result is input into the feedforward neural network, and the output of the feedforward neural network is input into the residual connection layer for a second forward and reverse propagation. The second forward and reverse propagation is then input into the normalization layer for a second integration matrix.

[0207] Optionally, the specific calculation formulas for the matrix transformation of the input data by the linear layer, the residual connection layer, and the normalization layer are shown in the following equations:

[0208] ;

[0209] Where Q is the linear layer matrix, K is the residual connection layer, and V is the normalized layer matrix. For input data, The weights of the linear layer, The weights of the residual connection layer are... The weights of the normalization layer;

[0210] The formula for calculating the attention score is as follows:

[0211] ;

[0212] Where T represents transpose; It is a constant.

[0213] Optionally, the matrix generation module is further configured to:

[0214] The first text embedding matrix is ​​added to the preset position embedding matrix to obtain a comprehensive matrix. The preset position embedding matrix is ​​obtained by acquiring the third text sample data, segmenting the third text sample data, inputting it into a preset neural network model for pre-training, and then completing the training. The preset position embedding matrix serves as a learnable parameter of the preset neural network model.

[0215] Those skilled in the art will understand that, for ease of description and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0216] Furthermore, this application also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the energy data cleaning method based on the Subword vocabulary as described in the above embodiments.

[0217] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a computer-executable program for causing a computer to perform the energy data cleaning method based on the Subword vocabulary as described in the above embodiments.

[0218] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for energy data cleaning based on Subword vocabulary, characterized in that, The method includes: After acquiring energy data, each text in the energy data is converted into a first text embedding matrix using a preset subword vocabulary, where the subword vocabulary is a text list at the character granularity level. The first text embedding matrix is ​​converted into a comprehensive matrix that conforms to the model input format; The comprehensive matrix is ​​input into a preset neural network model for cleaning to obtain cleaned data; The construction method of the preset Subword vocabulary includes: Input the first text sample data and perform character-level segmentation on the first text sample data to obtain the first segmented character; Count the frequency of each first segmenting character and its adjacent characters to form a character pair, and obtain multiple frequency values ​​of the first character pair; Select several target first character pair frequency values ​​that are greater than a preset frequency value from the plurality of first character pair frequency values, and use the character pairs corresponding to each target first character pair frequency value to construct and generate a Subword vocabulary; The energy data consists of N texts, each of which contains at least one sentence, and N is greater than or equal to 1. The step of converting each text in the energy data into a first text embedding matrix using a preset subword vocabulary includes: Identify the sentence corresponding to each text in the energy data and the sequence number contained in each sentence, and use the preset Subword vocabulary to convert the sentence corresponding to each text and the sequence number contained in each sentence into vectors, respectively obtaining the first sentence vector and the first sequence number vector corresponding to each text; Add the first sentence vector to the first sequence number vector to obtain the first summed vector for each text. The first text vector is obtained by concatenating N of the first summed vectors. The first text vector is then subjected to a matrix embedding operation to obtain the first text embedding matrix.

2. The Subword vocabulary based energy data cleaning method of claim 1, wherein, The matrix embedding operation specifically involves embedding a 1*n format text vector into an n*m matrix. Embedded computation is shown in the following formula: 。 3. The Subword vocabulary based energy data cleaning method of claim 1, wherein, The preset neural network model is a BERT model obtained by training the model using sample data about energy information. The model training process includes: Enter the second text sample data, perform character-level segmentation on the second text sample data to obtain the second segmented character. The second text sample data consists of M texts, where M is greater than or equal to 1. Count the frequency of each second dividing character forming a character pair with its adjacent character to obtain multiple frequency values ​​of second character pairs; Select several target second character pair frequency values ​​that are greater than a preset frequency value from the plurality of second character pair frequency values, and use the character pairs corresponding to each target second character pair frequency value to construct and generate a training vocabulary; Based on the training vocabulary, extract the second sentence vector and the second sequence number vector corresponding to each text from the second text sample data; Add the second sentence vector to the second sequence number vector to obtain the second summed vector for each text. The second text vector is obtained by concatenating M of the second summed vectors. The second text vector is then subjected to a matrix embedding operation to obtain the second text embedding matrix. The second text embedding matrix is ​​converted into a training matrix that conforms to the model input format, and the training matrix is ​​used to train the BERT model.

4. The Subword vocabulary based energy data cleaning method of claim 3, wherein, The BERT model includes: an encoding module, an olive-shaped feedforward neural network, and a softmax processing module connected in sequence; There are n encoding modules, and these n encoding modules are connected in series.

5. The Subword vocabulary based energy data cleaning method of claim 4, wherein, The encoding module includes: a linear layer, a residual connection layer, a normalization layer, and a feedforward neural network; The feedforward neural network consists of multiple linear layers plus activation functions, where the activation functions are shown in the following equation: ； The feedforward neural network is represented as: ； The internal processing flow of the input data in the encoding module includes: The input data is transformed by the linear layer, the residual connection layer, and the normalization layer to obtain the linear layer matrix, the residual connection layer matrix, and the normalization layer matrix, respectively. The attention score is calculated based on the linear layer matrix, the residual connection layer, and the normalized layer matrix; The attention score is input into the residual connection layer for the first forward and reverse propagation, and the result of the first forward and reverse propagation is input into the normalization layer for the first integration matrix to obtain the first integration result. The first integration result is input into the feedforward neural network, and the output of the feedforward neural network is input into the residual connection layer for a second forward and reverse propagation. The second forward and reverse propagation is then input into the normalization layer for a second integration matrix.

6. The energy data cleaning method based on a subword vocabulary according to claim 5, characterized in that, The specific calculation formulas for the matrix transformation of the input data by the linear layer, the residual connection layer, and the normalization layer are shown in the following equations: ； Where Q is the linear layer matrix, K is the residual connection layer, and V is the normalized layer matrix. For input data, The weights of the linear layer, The weights of the residual connection layer are... The weights of the normalization layer; The formula for calculating the attention score is as follows: ； Where T represents transpose; It is a constant.

7. The energy data cleaning method based on a subword vocabulary according to any one of claims 1-6, characterized in that, The step of converting the first text embedding matrix into a comprehensive matrix that conforms to the model input format includes: The first text embedding matrix is ​​added to the preset position embedding matrix to obtain a comprehensive matrix. The preset position embedding matrix is ​​obtained by acquiring the third text sample data, segmenting the third text sample data, inputting it into a preset neural network model for pre-training, and then completing the training. The preset position embedding matrix serves as a learnable parameter of the preset neural network model.

8. An energy data cleaning device based on a subword vocabulary, characterized in that, The device includes: The text conversion module is used to convert each text of the energy data into a first text embedding matrix after acquiring the energy data using a preset Subword vocabulary, wherein the Subword vocabulary is a text list at the character granularity level. The matrix generation module is used to convert the first text embedding matrix into a comprehensive matrix that conforms to the model input format; The cleaning module is used to input the comprehensive matrix into a preset neural network model for cleaning processing to obtain cleaned data; The construction method of the preset Subword vocabulary includes: Input the first text sample data and perform character-level segmentation on the first text sample data to obtain the first segmented character; Count the frequency of each first segmenting character and its adjacent characters to form a character pair, and obtain multiple frequency values ​​of the first character pair; Select several target first character pair frequency values ​​that are greater than a preset frequency value from the plurality of first character pair frequency values, and use the character pairs corresponding to each target first character pair frequency value to construct and generate a Subword vocabulary; The energy data consists of N texts, each of which contains at least one sentence, and N is greater than or equal to 1. The step of converting each text in the energy data into a first text embedding matrix using a preset subword vocabulary includes: Identify the sentence corresponding to each text in the energy data and the sequence number contained in each sentence, and use the preset Subword vocabulary to convert the sentence corresponding to each text and the sequence number contained in each sentence into vectors, respectively obtaining the first sentence vector and the first sequence number vector corresponding to each text; Add the first sentence vector to the first sequence number vector to obtain the first summed vector for each text. The first text vector is obtained by concatenating N of the first summed vectors. The first text vector is then subjected to a matrix embedding operation to obtain the first text embedding matrix.

Images