Energy big data cleaning method, device and equipment and storage medium

Abstract

The present application provides an energy big data cleaning method, device and equipment and storage medium, the method comprises: obtaining the text data to be cleaned, converting each character in the text data to be cleaned into a single character vector respectively, and performing word embedding processing to obtain a single character embedding vector corresponding to each character; position embedding processing is carried out on each character in the text data to be cleaned, and the position vector of each character is obtained; the position vector of each character in the text data to be cleaned is added to the corresponding single character embedding vector, the combined vector corresponding to each character is obtained, and the sequence of each character and the combined vector form a text data matrix to be cleaned; the text data to be cleaned matrix is input into the trained Transform model, and the cleaning text data output by the Transform model is obtained. The present application fuses the semantic and grammatical information of the text with the position information of the text by adopting the word embedding and position embedding operations, effectively improving the accuracy of energy big data cleaning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data cleaning, and particularly relates to an energy big data cleaning method, device, equipment and storage medium. BACKGROUND

[0002] With the continuous improvement of information technology, industrialization technology and production capacity in China, the scale of energy economic data is huge, the growth is rapid, the types are various, and the structural differences are great, which has become a major problem that the energy sector cannot ignore. How to combine massive energy economic big data with current high-tech to build an efficient and green energy economic big data cleaning model has become an important issue that each energy department and enterprise needs to solve urgently.

[0003] In the current research field of energy economic big data cleaning model, most workers focus on database technology, clustering, association analysis, conditional function dependence and other methods related to statistical machine learning. For example, the literature of Meng Xiangping et al. proposes an abnormal data cleaning model based on hadoop, and the literature of Qu Zhaoyang et al. proposes an energy big data cleaning model based on Spark framework. The above two cleaning models do not break out of the technical framework of distributed file system. The literature of Lin Nvgui et al. proposes a big data cleaning method based on improved K-means, and the literature of Zou Wang et al. proposes a data cleaning algorithm based on clustering, which also does not break out of the category of clustering algorithm. The literature of Huangfu Hanceng et al. proposes an automatic repair method for power data with order dependence, which extracts the features of the order properties of power data and automatically repairs the missing data. The literature of Yan Yingjie et al. proposes a power transmission equipment state big data cleaning method based on time series analysis, which uses iterative test method to fit the time series data. Further, in a paper of electronic information journal this year, Kou Junyan et al. proposes an abnormal data cleaning algorithm based on deep learning, which uses iterative threshold shrinkage algorithm (ISTA) to solve the data cleaning problem in Internet of Things, but the model data does not consider the text, only for digital text.

[0004] Currently, existing data cleaning methods rely heavily on algorithms based on traditional file systems or traditional, modified statistical machine learning methods. These methods fail to consider the rapid growth and changing trends of today's energy and economic big data. While past statistical machine learning methods might handle small amounts of energy and economic data, they struggle with the massive and diverse datasets of today's energy and economic data, resulting in significant computational inefficiencies. Furthermore, few algorithms are specifically designed for cleaning energy and economic text data. Current cleaning models may find processing Chinese text particularly challenging, and the lack of standardized masking libraries leads to low data cleaning accuracy. Additionally, current cleaning models suffer from a lack of generalization and applicability to various scenarios. Summary of the Invention

[0005] The present invention aims to provide a method, apparatus, equipment and storage medium for cleaning energy big data, so as to solve the above-mentioned technical problems and thereby improve the accuracy of cleaning energy big data.

[0006] To address the aforementioned technical problems, this invention provides a method for cleaning energy big data, comprising:

[0007] The text data to be cleaned is obtained, and each character in the text data to be cleaned is converted into a single character vector based on the index in the pre-built etymological dictionary. Then, word embedding processing is performed on each single character vector to obtain the single character embedding vector corresponding to each character.

[0008] Each character in the text data to be cleaned is subjected to position embedding processing to obtain the position vector of each character;

[0009] The position vector of each character in the text data to be cleaned is added to the corresponding single character embedding vector to obtain the combined vector corresponding to each character, and the text data matrix to be cleaned is formed based on the order of each character in the text data to be cleaned and the combined vector.

[0010] The text data matrix to be cleaned is input into a trained Transformer model to obtain the cleaned text data output by the Transformer model; wherein, the Transformer model is trained based on preset text sample data and the text data matrix and text data mask matrix corresponding to the text sample data.

[0011] Furthermore, the training methods for the Transformer model include:

[0012] Construct an etymological dictionary based on a pre-defined attribute corpus;

[0013] Obtain text sample data, convert each character in the text sample data into a single character vector based on the index in the etymology dictionary, and select several single character vectors for attribute masking processing according to the attribute corpus to obtain several single character mask vectors;

[0014] Each character vector is processed by word embedding to obtain the character embedding vector corresponding to each character, and each character mask vector is processed by word embedding to obtain the corresponding character embedding mask vector.

[0015] Each character in the text sample data is subjected to position embedding processing to obtain the position vector of each character;

[0016] The position vector of each character in the text sample data is added to the corresponding single character embedding vector to obtain the combination vector corresponding to each character, and a text sample data matrix is ​​formed based on the order of each character in the text sample data and the combination vector.

[0017] The position vector of each character in the text sample data is added to the corresponding single-character embedding mask vector to obtain the corresponding mask combination vector, and a text data mask matrix is ​​formed based on the order of each character in the text sample data and the corresponding mask combination vector.

[0018] The text data matrix is ​​input into the encoder of a pre-built Transformer model, and self-attention processing is performed within the encoder. The output of the encoder is used as the K and V matrices of the second layer of multi-head attention in the decoder of the Transformer model. The text data mask matrix is ​​used as the input of the decoder to train the Transformer model.

[0019] Furthermore, the step of converting each character in the text sample data into a single-character vector based on the index in the etymological dictionary specifically includes:

[0020] For each character in the etymological dictionary, construct a unit vector and establish an index between each character and the unit vector;

[0021] Based on the index between each character and the unit vector, the unit vector corresponding to each character in the text sample data is matched and used as the single character vector of each character in the text sample data.

[0022] Furthermore, the step of performing word embedding processing on each character vector to obtain the character embedding vector corresponding to each character specifically includes:

[0023] Based on a preset mapping matrix, the single-character vector corresponding to each character is mapped to obtain the single-character embedding vector corresponding to each character.

[0024] Further, the step of inputting the text data matrix into the encoder of a pre-built Transformer model, performing self-attention processing within the encoder, using the encoder output as the K and V matrices of the second layer of multi-head attention in the decoder of the Transformer model, and using the text data mask matrix as the input to the decoder to train the Transformer model specifically includes:

[0025] The text data matrix is ​​input into the encoder of the Transformer model. In the first layer of the encoder, a multi-head self-attention mechanism is used to linearly map the text data matrix to obtain matrices Q, K and V formed by the dot product of the text data matrix with three preset weights.

[0026] The attention matrix is ​​calculated based on the matrix Q, the matrix K, and the matrix V; wherein the dimension of the matrix K is used to transform the attention matrix into the form of a standard normal distribution;

[0027] The attention matrix is ​​subjected to a residual connection operation, and the attention matrix with the residual connection operation is input into the feedforward neural network of the Transformer model. The output data of the feedforward neural network is then input into the encoder.

[0028] Repeat the above three steps until the preset number of times is reached;

[0029] The final output data of the encoder is used as the K and V matrices of the second layer multi-head attention of the decoder of the Transformer model;

[0030] The text data mask matrix is ​​used as input to the decoder to train the Transformer model.

[0031] The present invention also provides an energy big data cleaning device, comprising:

[0032] The word embedding module is used to acquire the text data to be cleaned, convert each character in the text data to be cleaned into a single character vector based on the index in the pre-built etymological dictionary, and perform word embedding processing on each single character vector to obtain the single character embedding vector corresponding to each character.

[0033] The position embedding module is used to perform position embedding processing on each character in the text data to be cleaned, so as to obtain the position vector of each character.

[0034] The data matrix generation module is used to add the position vector of each character in the text data to be cleaned to the corresponding single character embedding vector to obtain the combination vector corresponding to each character, and to form a text data matrix to be cleaned based on the order of each character in the text data to be cleaned and the combination vector.

[0035] The data cleaning module is used to input the text data matrix to be cleaned into a trained Transformer model to obtain the cleaned text data output by the Transformer model; wherein, the Transformer model is trained based on preset text sample data and the text data matrix and text data mask matrix corresponding to the text sample data.

[0036] The present invention also provides a terminal device, including a processor and a memory storing a computer program, wherein the processor executes the computer program to implement any of the energy big data cleaning methods described above.

[0037] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the energy big data cleaning methods described above.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] This invention provides a method, apparatus, device, and storage medium for cleaning energy big data. The method includes: acquiring text data to be cleaned; converting each character in the text data to be cleaned into a single-character vector and performing word embedding processing to obtain a single-character embedding vector corresponding to each character; performing position embedding processing on each character in the text data to be cleaned to obtain a position vector for each character; adding the position vector of each character in the text data to the corresponding single-character embedding vector to obtain a combined vector corresponding to each character, and forming a text data matrix to be cleaned based on the order of the characters and the combined vectors; inputting the text data matrix to be cleaned into a trained Transformer model to obtain cleaned text data output by the Transformer model. This invention, by employing word embedding and position embedding operations, fuses the semantic and syntactic information of the text with the positional information of the text, effectively improving the accuracy of cleaning energy big data. Attached Figure Description

[0040] Figure 1 This is one of the flowcharts of the energy big data cleaning method provided by the present invention;

[0041] Figure 2 This is the second flowchart of the energy big data cleaning method provided by the present invention;

[0042] Figure 3 This is a schematic diagram of the encoder and decoder structure in the Transformer model provided by this invention;

[0043] Figure 4 This is a schematic diagram of the position vectors of characters at different positions in the text data provided by this invention, corresponding to the position data in the 100th and 101st dimensions.

[0044] Figure 5 This is a schematic diagram of the energy big data cleaning device provided by the present invention. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] Please see Figure 1 This invention provides a method for cleaning energy big data, which may include the following steps:

[0047] S1. Obtain the text data to be cleaned, convert each character in the text data to be cleaned into a single character vector based on the index in the pre-built etymological dictionary, and perform word embedding processing on each single character vector to obtain the single character embedding vector corresponding to each character.

[0048] S2. Perform position embedding processing on each character in the text data to be cleaned to obtain the position vector of each character;

[0049] S3. Add the position vector of each character in the text data to be cleaned to the corresponding single character embedding vector to obtain the combination vector corresponding to each character, and form a text data matrix to be cleaned based on the order of each character in the text data to be cleaned and the combination vector.

[0050] S4. Input the text data matrix to be cleaned into the trained Transformer model to obtain the cleaned text data output by the Transformer model; wherein, the Transformer model is trained based on preset text sample data and the text data matrix and text data mask matrix corresponding to the text sample data.

[0051] In this embodiment of the invention, the training method of the Transformer model further includes:

[0052] Construct an etymological dictionary based on a pre-defined attribute corpus;

[0053] Obtain text sample data, convert each character in the text sample data into a single character vector based on the index in the etymology dictionary, and select several single character vectors for attribute masking processing according to the attribute corpus to obtain several single character mask vectors;

[0054] Each character vector is processed by word embedding to obtain the character embedding vector corresponding to each character, and each character mask vector is processed by word embedding to obtain the corresponding character embedding mask vector.

[0055] Each character in the text sample data is subjected to position embedding processing to obtain the position vector of each character;

[0056] The position vector of each character in the text sample data is added to the corresponding single character embedding vector to obtain the combination vector corresponding to each character, and a text sample data matrix is ​​formed based on the order of each character in the text sample data and the combination vector.

[0057] The position vector of each character in the text sample data is added to the corresponding single-character embedding mask vector to obtain the corresponding mask combination vector, and a text data mask matrix is ​​formed based on the order of each character in the text sample data and the corresponding mask combination vector.

[0058] The text data matrix is ​​input into the encoder of a pre-built Transformer model, and self-attention processing is performed within the encoder. The output of the encoder is used as the K and V matrices of the second layer of multi-head attention in the decoder of the Transformer model. The text data mask matrix is ​​used as the input of the decoder to train the Transformer model.

[0059] In this embodiment of the invention, the step of converting each character in the text sample data into a single-character vector based on the index in the etymological dictionary specifically includes:

[0060] For each character in the etymological dictionary, construct a unit vector and establish an index between each character and the unit vector;

[0061] Based on the index between each character and the unit vector, the unit vector corresponding to each character in the text sample data is matched and used as the single character vector of each character in the text sample data.

[0062] In this embodiment of the invention, the step of performing word embedding processing on each character vector to obtain the character embedding vector corresponding to each character specifically includes:

[0063] Based on a preset mapping matrix, the single-character vector corresponding to each character is mapped to obtain the single-character embedding vector corresponding to each character.

[0064] In this embodiment of the invention, further, the step of inputting the text data matrix into the encoder of a pre-built Transformer model, performing self-attention processing within the encoder, using the encoder output as the K and V matrices of the second layer multi-head attention of the decoder of the Transformer model, and using the text data mask matrix as the input of the decoder to train the Transformer model specifically includes:

[0065] The text data matrix is ​​input into the encoder of the Transformer model. In the first layer of the encoder, a multi-head self-attention mechanism is used to linearly map the text data matrix to obtain matrices Q, K and V formed by the dot product of the text data matrix with three preset weights.

[0066] The attention matrix is ​​calculated based on the matrix Q, the matrix K, and the matrix V; wherein the dimension of the matrix K is used to transform the attention matrix into the form of a standard normal distribution;

[0067] The attention matrix is ​​subjected to a residual connection operation, and the attention matrix with the residual connection operation is input into the feedforward neural network of the Transformer model. The output data of the feedforward neural network is then input into the encoder.

[0068] Repeat the above three steps until the preset number of times is reached;

[0069] The final output data of the encoder is used as the K and V matrices of the second layer multi-head attention of the decoder of the Transformer model;

[0070] The text data mask matrix is ​​used as input to the decoder to train the Transformer model.

[0071] Based on the above scheme, and to facilitate a better understanding of the energy big data cleaning method provided in the embodiments of the present invention, the following detailed description is provided:

[0072] It should be noted that, in order to address the problems of low data cleaning accuracy and poor data cleaning model stability in the existing technologies, this invention provides an energy big data cleaning method based on Transformer and masking. Please refer to... Figure 2 The embodiments of the present invention can be implemented through the following steps:

[0073] Step S1: Construct an attribute corpus and build an etymology dictionary based on the attribute corpus.

[0074] Using database technology, frequently occurring words in the text are extracted and categorized based on their features to facilitate subsequent attribute masking operations. During classification, based on word attributes, the categorization can include:

[0075] Energy-related terms include: "crude oil", "raw coal", "natural gas", "wind power", "hydropower", and "nuclear power".

[0076] Terms related to country types;

[0077] Terms related to energy economic institutions;

[0078] Terms related to energy production equipment include: "wind turbine", "hydropower turbine", "hydropower station", "nuclear power station", and "photovoltaic power station".

[0079] Numeric text: “one”, “two”, “three”, “four”, “five”, “six”, etc.

[0080] Terms related to energy economic behavior include: "growth", "improvement", "decline", "shrinkage", and "reduction".

[0081] Words describing the state of the energy economy include: "strong", "sluggish", "high speed", "stable", "suddenly", etc.

[0082] The above only lists some different types of corpora. When classifying corpora, the types may include, but are not limited to, those listed above. Based on the corpora of the types mentioned above, a basic attribute corpus is built. Each category of the corpus represents its attribute, and each attribute has its own attribute value. Here, we assign different English labels to the six categories: "a", "b", "c", "d", "e", and "f". This simple attribute corpus is now complete, and it can be further modified based on the text content.

[0083] The words in the corpus are compiled to form an etymological dictionary.

[0084] When constructing an etymology dictionary, assuming the total number of distinct characters in the dictionary is m, then each character is matched with a unit vector, represented as:

[0085]

[0086] A1, A2, A3, ..., Am represent the first, second, third, ..., last character in the dictionary, respectively.

[0087] In this embodiment, the total number of characters included in the etymology dictionary is 100, so the dimension (i.e., length) of the unit vector of each character is also 100.

[0088] After constructing the attribute corpus and etymology dictionary, the next step is to input sample data and perform processing such as segmentation on the sample data.

[0089] Step S2: Enter text sample data.

[0090] In this embodiment, the sample text data is: "In 2021, the average utilization rates of hydropower, wind power, and photovoltaic power generation reached approximately 98%, 97%, and 98%, respectively." Of course, each character in the text is included in the etymological dictionary. As an example, the character "hydropower" in the text is masked, and the masked text is: "In 2021, the average utilization rates of [*], wind power, and photovoltaic power generation reached approximately 98%, 97%, and 98%, respectively."

[0091] Step S3: Perform word embedding processing on the entered text sample data.

[0092] Formula (1) represents the unit vector of each character included in the etymology dictionary. Then, based on the characters contained in the input text sample data and combined with the index in the constructed etymology dictionary, the unit vectors corresponding to these characters are found.

[0093] For example, the length of the entered text sample data "The average utilization rates of hydropower, wind power, and photovoltaic power generation in 2021 were approximately 98%, 97%, and 98%, respectively" is n. Using the index, the unit vector of each character in the entered text sample data can be found from formula (1):

[0094]

[0095] The set {B1,B2,B3,…,Bn} belongs to the set {A1,A2,A3,…,Am}, meaning that the characters in the entered text sample data are also the first few characters in the etymological dictionary.

[0096] After obtaining the single-character vector of each character, attribute masking is performed on one or more single-character vectors based on the attribute corpus constructed in step s1 (selecting error-prone words in the text for attribute masking according to a preset strategy) to obtain single-character mask vectors. This masking process is existing technology, and those skilled in the art know how to do it, so it will not be described in detail here.

[0097] Then, word embedding is performed on each character vector to obtain a character embedding vector; word embedding is performed on each character mask vector to obtain a character embedding mask vector.

[0098] When performing word embedding processing, single-character embedding vectors Specifically:

[0099] Mapping a unit vector to another space, the mapping matrix is ​​A, and the matrix size is n*n, i.e.:

[0100]

[0101] Where T represents the transpose of the matrix, and a, b, d, e represent the parameters of the mapping matrix.

[0102] The processing method for single-word embedding mask vectors is the same, so it will not be repeated here.

[0103] This embedding step goes beyond focusing on individual textual information within energy big data; it helps establish connections between words and phrases within the data. By mapping these connections to different domains, more information and relationships within the energy data can be uncovered. This can be achieved by directly calling the embedding function in the PyTorch framework.

[0104] After obtaining the single-character embedding vector and single-character embedding mask vector of each character in the input text sample data, proceed to the next step of processing.

[0105] Step S4: Perform position embedding processing on the entered text sample data.

[0106] After word embedding, the expression of the association information between words is obtained. Then, positional embedding is used to extract the positional relationships in sentences. Because the data being processed is text, the positional association information between words and sentences is crucial. If the positions of one word and another are swapped, the semantics may deviate significantly, and grammatical errors may also occur. Therefore, in addition to considering the information of the words themselves, the positional information between words should also be considered to assist in cleaning energy big data. During positional embedding, each character in the input text sample data needs to be individually embedded to obtain the position vector of each character. The specific processing method is as follows:

[0107] First, set the dimension of the position vector to be the same as the dimension of the single-word embedding vector, and of course, the dimension of the single-word embedding mask vector is also the same.

[0108] Then, the position data of each dimension in the position vector of each character is calculated. This position data is the value at the corresponding dimension (position) in the position vector. The formula for calculating this position data is:

[0109]

[0110]

[0111] Where pos represents the position of the word in the sentence, d model 2i represents the sentence length, 2i represents an even sentence length, and 2i+1 represents an odd sentence length. fi represents the frequency of each Chinese character in the corpus. The higher the frequency, the larger the amplitude of its position vector.

[0112] After adopting the above positional embedding method, each word corresponds to a trigonometric function with different amplitudes and periods. That is, even if two words in the text are the same, as long as their positions are different, the calculated positional embedding vectors will be completely different. This effectively preserves the positional information of the text. Similarly, even if two words are in the same position in a sentence, if their frequencies of occurrence are different, their positional embedding vectors will also be different. The positional vector of each word is calculated based on the words at different positions in the input text data. The curves plotted for the positional data (values) corresponding to the positional vectors of all words in the 100th and 101st dimensions are shown below. Figure 4 As shown in the figure. The horizontal axis represents the position of the character in the text data, and the vertical axis represents the position data of the corresponding dimension in the position vector (i.e., the value of the corresponding dimension in the vector).

[0113] After obtaining the position vector of each character in the input text sample data, proceed to the next step.

[0114] Step S5: Add the single-character embedding vector and position vector corresponding to each character in the input text sample data. Here, adding means directly adding the values ​​of each dimension in the vector to obtain the combined vector of each character. Then, according to the order of each character in the text sample data and the corresponding combined vector, combine them to form a text data matrix. This text data matrix is ​​the input for subsequent model training.

[0115] Similarly, the position vector of each character in the text sample data is added to the corresponding single-character embedding mask vector to obtain the mask combination vector of the character. Based on the order of each character in the text sample data and the corresponding mask combination vector, a text data mask matrix is ​​formed.

[0116] Step S6: Input the text data matrix obtained in step S5 into the encoder of the Transformer model (translation model). The Transformer model can be an existing model. Perform self-attention processing in the encoder. The output of the encoder is used as the K and V matrices of the second layer of multi-head attention of the decoder of the Transformer model. Use the text data mask matrix obtained in step S5 as the input of the decoder to complete the training of the Transformer model.

[0117] The Transformer model includes an encoder and a decoder, where the encoder and decoder are as follows:Figure 3 As shown.

[0118] The specific method for training the Transformer model is as follows:

[0119] Step S61: Input the text data matrix obtained in step S5 into the encoder of the Transformer model. In the first layer of the encoder, a multi-head self-attention mechanism is used to perform a linear mapping on the text data matrix obtained in step S5, that is, to assign three weights W. Q W K and W V Multiply the text data matrix obtained in step S5 by these three weights to form three matrices Q, K, and V. The specific formulas for calculating matrices Q, K, and V are as follows:

[0120] Q = Linear(X) embedding ) = X embedding W Q

[0121] K = Linear(X) embedding ) = X embedding W K

[0122] V = Linear(X) embedding ) = X embedding W V

[0123] Among them, X embedding W represents the input text data matrix. Q W K and W V Indicates weight;

[0124] Step S62: Using the Q, K, and V matrices obtained in step S61, calculate the attention matrix. The calculation formula is as follows:

[0125]

[0126] Where T represents transpose, d k The dimension of matrix K is used to transform the attention matrix into a standard normal distribution, making the results more stable after softmax normalization.

[0127] Step S63: Perform a residual join operation on the attention matrix calculated in step S62. The residual join is to ensure that the matrix does not lose its original information. The specific calculation formula is as follows:

[0128] X' attention =X embedding +X attention

[0129] X” attention =LayerNorm(X' attention )

[0130] Among them, X embedding X represents the input text data matrix. attention This represents the matrix X' after the input matrix has been processed by the attention mechanism. attention This represents the sum of the input text data matrix and the matrix processed by the attention mechanism; X” attention This represents the normalized matrix.

[0131] Step S64: The attention matrix, after residual connection operation, is input into the feedforward neural network. The feedforward neural network is activated using a multi-layer linear mapping plus activation function, calculated as follows:

[0132] X hidden =Activate(Linear(Linear...Linear(X” attention )))

[0133] Among them, X” attention This represents the normalized matrix;

[0134] In step S65, the output of the feedforward neural network is input back into the encoder, and steps S61-S64 are repeated M times, where M is a hyperparameter.

[0135] In step S66, the final output of the encoder in step S65 is used as the K and V matrices of the second layer of multi-head attention of the decoder of the Transformer model; the text data mask matrix obtained in step S5 is used as the input of the decoder to complete the training of the Transformer model.

[0136] Each input is shifted one position to the right, starting from the input mask "S" and ending at the decoder output mask "E". The masked word vectors can perfectly output the unmasked word vectors. The correct word vectors are cross-entropyed with the output, thus guiding the model to perform gradient optimization in the correct direction.

[0137] The first layer of the decoder's multi-head attention mechanism also involves masking. The purpose of this masking is that during training, for each predicted word, the model only knows the vectors of all previously encountered words, unaware of upcoming ones. This perfectly simulates the real testing process; in other words, it prevents the model from cheating during training. Setting the subsequent weights to infinity results in them becoming 0 after the final softmax, achieving the masking effect.

[0138] Step S7: The text data to be cleaned is processed sequentially through steps S3-S5 to obtain a text data matrix. The obtained text data matrix is ​​then input into the Transformer model to output the cleaned data. The data is then reverse-transcribed to obtain the final desired text.

[0139] It should be noted that, compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0140] 1. By establishing a standard attribute corpus and etymological dictionary, and performing attribute masking on each or part of the vectors based on the attribute corpus, the cleaning model's cleaning capabilities become more targeted and focused, improving its accuracy in data cleaning. 2. By employing word embedding and positional embedding operations, the semantic and syntactic information of the text is integrated with its positional information, facilitating the training of the cleaning model and further enhancing its accuracy. 3. The cleaning model adopts a Transformer-based structure, enabling it to perfectly adapt to massive data environments and maintain excellent stability and accuracy.

[0141] 2. In the location embedding process, a bold innovation was made to the calculation formula for location information. Not only was the period adjusted, but the amplitude was also adjusted. In the calculation formula, fi represents the frequency of each Chinese character in the corpus. The higher the frequency, the larger the amplitude of its location vector. Here, each word corresponds to a trigonometric function with a different amplitude and a different period. That is, even if two words in the text are the same, as long as their positions are different, the calculated location embedding vectors will be completely different. This effectively preserves the location information of the text. Similarly, even if two words are in the same position in a sentence, if their frequencies of occurrence are different, their location embedding vectors will also be different. This achieves the fusion of the semantic and syntactic information of the text with the location information of the text. The uniqueness of the location information is relatively high, which is more conducive to the cleaning training of the cleaning model and improves the accuracy of the cleaning model in data cleaning.

[0142] 3. The first layer of the multi-head attention mechanism in the decoder of the Transformer model also performs masking. The purpose of masking is that during training, for each predicted word, the model only knows all the word vectors that have appeared before, but not the word vectors that will appear later. This perfectly simulates the real testing process. In other words, it is to prevent the model from cheating during training. The weights in the later layers are set to infinity, and after the softmax, they will become 0, thus achieving the effect of masking and improving the accuracy of the cleaning model in cleaning data.

[0143] It should be noted that, for the sake of simplicity, the above methods or process embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0144] Please see Figure 5 This invention also provides an energy big data cleaning device, comprising:

[0145] The word embedding module 1 is used to acquire the text data to be cleaned, convert each character in the text data to be cleaned into a single character vector based on the index in the pre-built etymological dictionary, and perform word embedding processing on each single character vector to obtain the single character embedding vector corresponding to each character.

[0146] Position embedding module 2 is used to perform position embedding processing on each character in the text data to be cleaned, so as to obtain the position vector of each character;

[0147] The data matrix generation module 3 is used to add the position vector of each character in the text data to be cleaned to the corresponding single character embedding vector to obtain the combination vector corresponding to each character, and to form a text data matrix to be cleaned based on the order of each character in the text data to be cleaned and the combination vector.

[0148] The data cleaning module 4 is used to input the text data matrix to be cleaned into the trained Transformer model to obtain the cleaned text data output by the Transformer model; wherein, the Transformer model is trained based on preset text sample data and the text data matrix and text data mask matrix corresponding to the text sample data.

[0149] It is understood that the above-described device embodiments correspond to the method embodiments of the present invention. The energy big data cleaning device provided by the embodiments of the present invention can implement the energy big data cleaning method provided by any one of the method embodiments of the present invention.

[0150] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the energy big data cleaning methods described above.

[0151] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0152] Those skilled in the art will clearly understand that, for convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0153] The terminal device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0154] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device via various interfaces and lines.

[0155] The memory can be used to store the computer program. The processor implements various functions of the terminal device by running or executing the computer program stored in the memory and calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc.; the data storage area may store data created based on the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0156] The storage medium is a computer-readable storage medium, and the computer program is stored in the computer-readable storage medium. When executed by a processor, the computer program can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0157] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for cleaning energy big data, characterized in that, include: The text data to be cleaned is obtained, and each character in the text data to be cleaned is converted into a single character vector based on the index in the pre-built etymological dictionary. Then, word embedding processing is performed on each single character vector to obtain the single character embedding vector corresponding to each character. Each character in the text data to be cleaned is subjected to position embedding processing to obtain the position vector of each character; The position vector of each character in the text data to be cleaned is added to the corresponding single character embedding vector to obtain the combined vector corresponding to each character, and the text data matrix to be cleaned is formed based on the order of each character in the text data to be cleaned and the combined vector. The text data matrix to be cleaned is input into a trained Transformer model to obtain the cleaned text data output by the Transformer model; wherein, the Transformer model is trained based on preset text sample data and the text data matrix and text data mask matrix corresponding to the text sample data.

2. The energy big data cleaning method according to claim 1, characterized in that, The training methods for the Transformer model include: Construct an etymological dictionary based on a pre-defined attribute corpus; Obtain text sample data, convert each character in the text sample data into a single character vector based on the index in the etymology dictionary, and select several single character vectors for attribute masking processing according to the attribute corpus to obtain several single character mask vectors; Each character vector is processed by word embedding to obtain the character embedding vector corresponding to each character, and each character mask vector is processed by word embedding to obtain the corresponding character embedding mask vector. Each character in the text sample data is subjected to position embedding processing to obtain the position vector of each character; The position vector of each character in the text sample data is added to the corresponding single character embedding vector to obtain the combination vector corresponding to each character, and a text sample data matrix is ​​formed based on the order of each character in the text sample data and the combination vector. The position vector of each character in the text sample data is added to the corresponding single-character embedding mask vector to obtain the corresponding mask combination vector, and a text data mask matrix is ​​formed based on the order of each character in the text sample data and the corresponding mask combination vector. The text data matrix is ​​input into the encoder of a pre-built Transformer model, and self-attention processing is performed within the encoder. The output of the encoder is used as the K and V matrices of the second layer of multi-head attention in the decoder of the Transformer model. The text data mask matrix is ​​used as the input of the decoder to train the Transformer model.

3. The energy big data cleaning method according to claim 2, characterized in that, The process of converting each character in the text sample data into a single-character vector based on the index in the etymological dictionary specifically includes: For each character in the etymological dictionary, construct a unit vector and establish an index between each character and the unit vector; Based on the index between each character and the unit vector, the unit vector corresponding to each character in the text sample data is matched and used as the single character vector of each character in the text sample data.

4. The energy big data cleaning method according to claim 2, characterized in that, The step of performing word embedding processing on each character vector to obtain the character embedding vector corresponding to each character specifically includes: Based on a preset mapping matrix, the single-character vector corresponding to each character is mapped to obtain the single-character embedding vector corresponding to each character.

5. The energy big data cleaning method according to claim 2, characterized in that, The process of inputting the text data matrix into the encoder of a pre-built Transformer model, performing self-attention processing within the encoder, using the encoder output as the K and V matrices of the second layer multi-head attention in the decoder of the Transformer model, and using the text data mask matrix as the input to the decoder to train the Transformer model specifically includes: The text data matrix is ​​input into the encoder of the Transformer model. In the first layer of the encoder, a multi-head self-attention mechanism is used to linearly map the text data matrix to obtain matrices Q, K and V formed by the dot product of the text data matrix with three preset weights. The attention matrix is ​​calculated based on the matrix Q, the matrix K, and the matrix V; wherein the dimension of the matrix K is used to transform the attention matrix into the form of a standard normal distribution; The attention matrix is ​​subjected to a residual connection operation, and the attention matrix with the residual connection operation is input into the feedforward neural network of the Transformer model. The output data of the feedforward neural network is then input into the encoder. Repeat the above three steps until the preset number of times is reached; The final output data of the encoder is used as the K and V matrices of the second layer multi-head attention of the decoder of the Transformer model; The text data mask matrix is ​​used as input to the decoder to train the Transformer model.

6. An energy big data cleaning device, characterized in that, include: The word embedding module is used to acquire the text data to be cleaned, convert each character in the text data to be cleaned into a single character vector based on the index in the pre-built etymological dictionary, and perform word embedding processing on each single character vector to obtain the single character embedding vector corresponding to each character. The position embedding module is used to perform position embedding processing on each character in the text data to be cleaned, so as to obtain the position vector of each character. The data matrix generation module is used to add the position vector of each character in the text data to be cleaned to the corresponding single character embedding vector to obtain the combination vector corresponding to each character, and to form a text data matrix to be cleaned based on the order of each character in the text data to be cleaned and the combination vector. The data cleaning module is used to input the text data matrix to be cleaned into a trained Transformer model to obtain the cleaned text data output by the Transformer model; wherein, the Transformer model is trained based on preset text sample data and the text data matrix and text data mask matrix corresponding to the text sample data.

7. The energy big data cleaning device according to claim 6, characterized in that, The training methods for the Transformer model include: Construct an etymological dictionary based on a pre-defined attribute corpus; Obtain text sample data, convert each character in the text sample data into a single character vector based on the index in the etymology dictionary, and select several single character vectors for attribute masking processing according to the attribute corpus to obtain several single character mask vectors; Each character vector is processed by word embedding to obtain the character embedding vector corresponding to each character, and each character mask vector is processed by word embedding to obtain the corresponding character embedding mask vector. Each character in the text sample data is subjected to position embedding processing to obtain the position vector of each character; The position vector of each character in the text sample data is added to the corresponding single character embedding vector to obtain the combination vector corresponding to each character, and a text sample data matrix is ​​formed based on the order of each character in the text sample data and the combination vector. The position vector of each character in the text sample data is added to the corresponding single-character embedding mask vector to obtain the corresponding mask combination vector, and a text data mask matrix is ​​formed based on the order of each character in the text sample data and the corresponding mask combination vector. The text data matrix is ​​input into the encoder of a pre-built Transformer model, and self-attention processing is performed within the encoder. The output of the encoder is used as the K and V matrices of the second layer of multi-head attention in the decoder of the Transformer model. The text data mask matrix is ​​used as the input of the decoder to train the Transformer model.

8. The energy big data cleaning device according to claim 7, characterized in that, The word embedding module is also used for: For each character in the etymological dictionary, construct a unit vector and establish an index between each character and the unit vector; Based on the index between each character and the unit vector, the unit vector corresponding to each character in the text sample data is matched and used as the single character vector of each character in the text sample data.

9. A terminal device, comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the energy big data cleaning method according to any one of claims 1 to 5.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the energy big data cleaning method as described in any one of claims 1 to 5.

Images