An application method and device for a wind and solar storage text file library

Abstract

The present invention relates to the field of electric power technology, and discloses an application method and device of a wind, solar, and storage text file library. The method collects wind, solar, and storage text file data from a preset data source; uses a preset C‑TALBERT model to perform feature extraction on the wind, solar, and storage text file data, and screens out a number of wind, solar, and storage text files; uses a Simhash algorithm to determine the correlation between a number of wind, solar, and storage text files, and performs data deduplication; imports the deduplicated wind, solar, and storage text files into a first message queue; classifies a number of wind, solar, and storage text files in the message queue, and stores them in a wind, solar, and storage text file library. The present invention uses a wind, solar, and storage text file library to implement data sorting, analysis, and storage for wind, solar, and storage text files, thereby improving the efficiency of file classification and the accuracy of file analysis.

Description

Technical Field

[0001] The present invention relates to the field of electric power technology, and in particular to an application method and device for a wind, solar, and storage text file library. Background Art

[0002] As an emerging form of energy, the wind, solar and energy storage industry has broad development prospects. With the continuous innovation of science and technology and the growing market of renewable energy, wind, solar and energy storage text files are constantly innovating and applying, so the analysis and storage methods of wind, solar and energy storage text files need to be further improved.

[0003] At present, the analysis of wind, solar and energy storage text files by the wind, solar and energy storage text file library has limitations. In the process of collecting wind, solar and energy storage text file data, due to the large amount of information and high repetitiveness, the efficiency of the wind, solar and energy storage text file library is low in the process of processing and analysis; it is unable to identify and associate files of the same type of content, resulting in low accuracy of file classification, which does not meet existing needs. Summary of the invention

[0004] The present invention provides an application method and device of a wind and solar storage text file library, which can use the wind and solar storage text file library to implement data sorting, analysis and storage of wind and solar storage text files, thereby improving the efficiency of file classification and the accuracy of file analysis.

[0005] In order to solve the above technical problems, the present invention provides an application method of a wind and solar storage text file library, comprising:

[0006] Collecting text file data of wind, solar and storage in the preset data source;

[0007] Using a preset C-TALBERT model to extract features from the wind, solar and energy storage text file data, and screening out a number of wind, solar and energy storage text files;

[0008] The Simhash algorithm is used to determine the relevance between several wind, solar and storage text files to remove duplicate data;

[0009] Importing the deduplicated copies of the wind and solar storage text files into the first message queue;

[0010] A plurality of the wind and solar storage text files are classified in the message queue and stored in a wind and solar storage text file library.

[0011] Furthermore, the collection of wind, solar and storage text file data in the preset data source is specifically:

[0012] Search for keywords of wind, solar and storage text files in the preset data source to obtain search results;

[0013] The search results are collected by using a web crawler, an incremental crawling scheduler, data attribute identification, crawler configuration or crawler exception management to form wind and solar storage text file data.

[0014] Furthermore, the preset C-TALBERT model is used to extract features from the wind, solar and storage text file data, specifically:

[0015] Perform MLM and SOP pre-training tasks on the ALBERT model to form the C-TALBERT model;

[0016] Utilize the underlying Transformer of the C-TALBERT model to extract the surface information of the wind, solar and storage text file data to form a surface information feature representation;

[0017] The middle-layer Transformer of the C-TALBERT model is used to extract the syntactic information of the wind, solar and storage text file data to form a syntactic information feature representation;

[0018] The high-level Transformer of the C-TALBERT model is used to extract the semantic information of the wind and solar storage text file data to form a semantic information feature representation.

[0019] Furthermore, the preset C-TALBERT model is used to extract features from the wind, solar and storage text file data, and a number of wind, solar and storage text files are screened out, specifically:

[0020] Combining surface information feature representation, syntactic information feature representation and semantic information feature representation into vectors to form several groups of file vectors;

[0021] The TextCNN convolutional neural network of the C-TALBERT model is used to perform multi-label classification tasks on several groups of file vectors, and several wind and solar storage text files are screened out.

[0022] Furthermore, the Simhash algorithm is used to determine the relevance between several copies of the wind, solar and storage text files to perform data deduplication, specifically:

[0023] Perform word segmentation, weight calculation and dimension reduction on a plurality of wind, solar and energy storage text files respectively, and obtain feature vectors corresponding to the plurality of wind, solar and energy storage text files;

[0024] Based on the feature vector, the Simhash algorithm is used to calculate the Simhash value of each wind, solar and storage text file; wherein the Simhash value is divided into preset parts for storage;

[0025] The similarity between each wind, solar and storage text file is calculated according to the Simhash value, and data deduplication processing is performed on several copies of the wind, solar and storage text files based on the similarity.

[0026] Further, the deduplicated copies of the wind and solar storage text files are imported into the first message queue, specifically:

[0027] Use natural language processing technology and information extraction methods to perform text file analysis and structured data extraction on the deduplicated wind, solar and storage text files to obtain the structured data corresponding to each wind, solar and storage text file;

[0028] Converting the structured data into a format such that the structured data meets the data storage requirements of the message queue;

[0029] Creating a first message queue using a message queue system;

[0030] Import the structured data corresponding to each wind, solar and storage text file into the first message queue in batches.

[0031] Furthermore, the plurality of wind and solar storage text files are classified in the message queue, specifically:

[0032] Mapping the plurality of wind and solar storage text files into a plurality of wind and solar storage text file sentence vectors using a deep learning encoder;

[0033] Using a wind-solar-storage text file classifier, a Softmax calculation is performed on a plurality of wind-solar-storage text file sentence vectors to obtain a first classification prediction probability corresponding to each wind-solar-storage text file;

[0034] Calculate the uncertainty of the first classification prediction probability of each wind, solar and storage text file;

[0035] When the uncertainty is lower than a preset uncertainty threshold, the prediction category of the corresponding wind, solar and storage text file is obtained based on the first classification prediction probability;

[0036] When the uncertainty is higher than a preset uncertainty threshold, the title vector of the wind and solar storage text file is calculated, and the second classification prediction probability of the wind and solar storage text file is calculated in combination with the sentence vector of the wind and solar storage text file, and the corresponding prediction category of the wind and solar storage text file is obtained based on the second classification prediction probability.

[0037] Furthermore, the wind and solar storage text file library is specifically:

[0038] The text file library of Fengguang Storage includes MySQL database, file system library and index library;

[0039] The Mysql database is used to store data attribute information of wind and solar storage text files;

[0040] The file system library is used to store text files of various formats;

[0041] The index library is used to store structured information of text files.

[0042] Furthermore, the wind and solar storage text file library also includes:

[0043] Set permissions for users of the wind and solar storage text file library based on user type, specifically:

[0044] The management user settings include system management permissions for crawler configuration, exception handling, and file addition, deletion, modification, and query;

[0045] Set file analysis and viewing permissions for research users, including system search, file review and file download.

[0046] The present invention provides an application method of a wind-solar storage text file library, which collects wind-solar storage text file data in a preset data source; uses a preset C-TALBERT model to extract features of the wind-solar storage text file data, and screens out a number of wind-solar storage text files; uses a Simhash algorithm to determine the correlation between a number of wind-solar storage text files, and performs data deduplication; imports the deduplicated wind-solar storage text files into a first message queue; classifies a number of wind-solar storage text files in the message queue, and stores them in a wind-solar storage text file library. The present invention uses a wind-solar storage text file library to realize data sorting, analysis and storage of wind-solar storage text files, strictly screens and deduplicates the collected data, removes useless data, and improves the efficiency of data analysis; calculates the similarity of files, associates wind-solar storage text files, and improves the accuracy of file classification.

[0047] Accordingly, the present invention provides an application device for a wind and light storage text file library, including: a collection module, a screening module, a deduplication module, an import module and a classification module;

[0048] The acquisition module is used to collect text file data of wind, solar and storage in the preset data source;

[0049] The screening module is used to extract features of the wind, solar and storage text file data using a preset C-TALBERT model, and screen out a number of wind, solar and storage text files;

[0050] The deduplication module is used to use the Simhash algorithm to determine the relevance between several copies of the wind, solar and storage text files to perform data deduplication;

[0051] The import module is used to import the deduplicated copies of the wind and solar storage text files into the first message queue;

[0052] The classification module is used to classify a plurality of the wind and solar storage text files in the message queue and store them in a wind and solar storage text file library.

[0053] The present invention provides an application device for a wind-solar storage text file library. Based on the organic combination of modules, wind-solar storage text file data is collected in a preset data source; a preset C-TALBERT model is used to extract features of the wind-solar storage text file data, and a number of wind-solar storage text files are screened out; a Simhash algorithm is used to determine the correlation between a number of wind-solar storage text files, and data deduplication is performed; the deduplicated wind-solar storage text files are imported into a first message queue; a number of wind-solar storage text files are classified in the message queue, and stored in a wind-solar storage text file library. The present invention uses a wind-solar storage text file library to realize data sorting, analysis and storage of wind-solar storage text files, strictly screens and deduplicates the collected data, removes useless data, and improves the efficiency of data analysis; and the wind-solar storage text files are associated by calculating the similarity of the files, thereby improving the accuracy of file classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A flow chart of an embodiment of an application method of a wind and solar storage text file library provided by the present invention;

[0055] Figure 2 A flow chart of another embodiment of the application method of the wind and solar storage text file library provided by the present invention;

[0056] Figure 3 A structural schematic diagram of an embodiment of an application device for a wind and light storage text file library provided by the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0059] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0060] Example 1

[0061] See also Figure 1 , is a flow chart of an embodiment of an application method of a wind and solar storage text file library provided by the present invention, the method includes steps 101 to 104, each step is specifically as follows:

[0062] Step 101: Collecting text file data of wind, solar and storage in a preset data source.

[0063] Further, in the first embodiment of the present invention, the wind and solar storage text file data is collected in the preset data source, specifically:

[0064] Search for keywords of wind, solar and storage text files in the preset data source to obtain search results;

[0065] The search results are collected by using a web crawler, an incremental crawling scheduler, data attribute identification, crawler configuration or crawler exception management to form wind and solar storage text file data.

[0066] In the first embodiment of the present invention, documents or data sets are downloaded from domestic and foreign websites, file databases or other sources to obtain the original data of electronic storage text files. The collection methods include web crawlers, incremental crawling schedulers, data attribute identification, crawler configuration, crawler exception management, etc. The sites are classified according to the website structure, security policy and other characteristics. A series of crawlers are designed based on the Scrapy crawler framework, and each crawler is responsible for data collection of a type of site.

[0067] In the first embodiment of the present invention, after collecting text file data, for each file, a text file processing technology library (such as NLTK, spaCy, etc.) is used to preprocess the data, including removing unnecessary punctuation marks, stop words, etc., to reduce noise and data redundancy.

[0068] Step 102: Use the preset C-TALBERT model to perform feature extraction on the wind, solar and energy storage text file data, and screen out a number of wind, solar and energy storage text files.

[0069] Furthermore, in the first embodiment of the present invention, the preset C-TALBERT model is used to extract features from the wind, solar and storage text file data, specifically:

[0070] Perform MLM and SOP pre-training tasks on the ALBERT model to form the C-TALBERT model;

[0071] Utilize the underlying Transformer of the C-TALBERT model to extract the surface information of the wind, solar and storage text file data to form a surface information feature representation;

[0072] The middle-layer Transformer of the C-TALBERT model is used to extract the syntactic information of the wind, solar and storage text file data to form a syntactic information feature representation;

[0073] The high-level Transformer of the C-TALBERT model is used to extract the semantic information of the wind and solar storage text file data to form a semantic information feature representation.

[0074] Furthermore, in the first embodiment of the present invention, the preset C-TALBERT model is used to extract features of the wind, solar and storage text file data, and a number of wind, solar and storage text files are screened out, specifically:

[0075] Combining surface information feature representation, syntactic information feature representation and semantic information feature representation into vectors to form several groups of file vectors;

[0076] The TextCNN convolutional neural network of the C-TALBERT model is used to perform multi-label classification tasks on several groups of file vectors, and several wind and solar storage text files are screened out.

[0077] In the first embodiment of the present invention, due to the large number and complex types of wind, solar and storage text files, some file types are not very effective. Many files only have the words "wind, solar and storage" in the text, which are not very relevant to the construction of wind, solar and storage. In addition, important documents of superior departments may also be repeated due to forwarding and interpretation by other subordinate units. Therefore, it is necessary to comprehensively consider standards such as authority, uniqueness, and relevance, strictly screen text files one by one, organize text file names, release time, and release units, and finally organize wind, solar and storage text files stored in the wind, solar and storage text file library.

[0078] In the first embodiment of the present invention, before screening the wind, solar and storage text files, the ALBERT model is first used to perform MLM and SOP pre-training tasks to obtain a Curated-TAPAALBERT (C-TALBERT) model suitable for the multi-label task field for screening. MLM refers to the random use of mask tokens ([MASK]) for words in a sentence, and then predicting the original words at the [MASK] position using relevant contextual information; SOP is sentence order prediction, which determines whether there is a sequential relationship between two sentences or whether they are in reverse order. The wind, solar and storage text file data is input into the trained C-TALBERT model so that the bottom-level Transformer of the model performs surface information feature representation of text files, the middle-level Transformer generates syntactic information feature representation of text files, and the high-level Transformer generates semantic information feature representation of text files. Then, the three generated feature representations are combined to form a file vector, and the file vectors of different groups are respectively input into the TextCNN convolutional neural network, so that TextCNN can further extract the semantic information features of the file vectors of different groups, and select the optimal dynamic text file feature vector for multi-label classification tasks to complete the screening of wind, solar and storage text files. In order to prevent the model from overfitting, in the two-layer fully connected neural network for label prediction, the first fully connected layer is connected, a dropout is performed, the vector dimension is kept unchanged, and then it is input into the second fully connected layer to output the predicted probability of each category label.

[0079] Step 103: Use the Simhash algorithm to determine the relevance between the plurality of wind, solar and storage text files and perform data deduplication.

[0080] Furthermore, in the first embodiment of the present invention, the Simhash algorithm is used to determine the relevance between the plurality of wind, solar and storage text files to perform data deduplication, specifically:

[0081] Perform word segmentation, weight calculation and dimension reduction on a plurality of wind, solar and energy storage text files respectively, and obtain feature vectors corresponding to the plurality of wind, solar and energy storage text files;

[0082] Based on the feature vector, the Simhash algorithm is used to calculate the Simhash value of each wind, solar and storage text file; wherein the Simhash value is divided into preset parts for storage;

[0083] The similarity between each wind, solar and storage text file is calculated according to the Simhash value, and data deduplication processing is performed on several copies of the wind, solar and storage text files based on the similarity.

[0084] In the first embodiment of the present invention, the Simhash algorithm is used to perform a series of calculations such as word segmentation, hashing, weighting, merging, and dimension reduction on the collected wind, solar, and storage text files, and finally a 64-bit information fingerprint of the text files can be generated. First, Jieba word segmentation software is used to segment the text files, stop words are removed based on the vocabulary, and the TF-IDF algorithm is used to calculate weights and reduce dimensions, and the text files are represented as feature vectors. Based on the Chinese feature vector, a 64-bit signature is calculated for each wind, solar, and storage text file. The process does not involve pairwise comparison of the original text files, but converts the complex full-text similarity comparison into a fixed-length signature value comparison. The closer the signature value, the more similar the corresponding text files are. The calculation process of Simhas value is as follows: use 64-dimensional vector V to represent fingerprint, initialized to 0; use md5 hash algorithm to hash each word into 64-bit hash value represented by 0 and 1; if the i-th bit of the word hash value is 1, then increase the weight value of the word at the i-th bit of V; if the i-th bit of the word hash value is 0, then reduce the weight value of the word at the i-th bit of V; after processing each word, if the i-th bit of V is positive, change the bit to 1, and if it is negative, change it to 0. The calculated Simhash value is the weighted combination of the hash value of each feature value of the text file. Using big data storage technology, the original text files and fingerprint values ​​are stored in the distributed key-value system HBase, and a massive distributed Simhash fingerprint library is constructed, which can improve the retrieval efficiency of Simhas values ​​of massive text files.

[0085] In the first embodiment of the present invention, the method for determining the similarity of two text files is to perform an XOR operation on their Simhash values, and the formula is as follows:

[0086] |hamningDist(simhash(str1),simhash(str2))|≤k

[0087] Among them, hammingDist is a function for calculating the Hamming distance between two integers, that is, the number of different bits in the binary encoding of two integers, K is the maximum tolerated number of different bits, which can be 3, and str1 and str2 are arrays with different original addresses in the Simhash value.

[0088] Based on the above calculation, the similarity between text files can be obtained, and by setting a similarity threshold, it can be determined whether data deduplication is required based on the similarity.

[0089] In the first embodiment of the present invention, the present invention divides the 64-bit Simhash value into 4 parts and establishes a 16bit index for storage. Compared with storing the 64-bit Simhash value with a 64bit index, the number of similarity calculations for a single text file of the present invention is reduced to: Where n is the total number of files, and the number of conventional pairwise comparison calculations is Therefore, the overall calculation times of the present invention are reduced to about the original The number of similarity calculations is greatly reduced, and the calculation efficiency is improved.

[0090] Step 104: import the deduplicated copies of the wind and solar storage text files into the first message queue.

[0091] Further, in the first embodiment of the present invention, the plurality of deduplicated wind and solar storage text files are imported into the first message queue, specifically:

[0092] Use natural language processing technology and information extraction methods to perform text file analysis and structured data extraction on the deduplicated wind, solar and storage text files to obtain the structured data corresponding to each wind, solar and storage text file;

[0093] Converting the structured data into a format such that the structured data meets the data storage requirements of the message queue;

[0094] Creating a first message queue using a message queue system;

[0095] Import the structured data corresponding to each wind, solar and storage text file into the first message queue in batches.

[0096] In the first embodiment of the present invention, natural language processing (NLP) technology and information extraction methods are used to perform text-type file analysis and structured data extraction on each file, including keyword recognition, entity extraction, topic modeling, relationship extraction, etc.; the extracted structured data is converted into JSON, XML and other formats suitable for transmission and storage in the message queue, ensuring that the structure and fields of the data are clear and meet the requirements of the message queue; a message queue system such as RabbitMQ and Apache Kafka is selected, and installed and configured according to its documentation, and a queue or topic is created to receive and store the extracted structured data; according to the API or client library of the message queue system, the message queue is connected to and the data is sent to the queue or topic one by one, thereby completing the batch import of structured data into the message queue.

[0097] Step 105: classify a plurality of the wind, solar and storage text files in the message queue, and store them in a wind, solar and storage text file library.

[0098] Further, in the first embodiment of the present invention, the classification of the plurality of wind and solar storage text files in the message queue is specifically as follows:

[0099] Mapping the plurality of wind and solar storage text files into a plurality of wind and solar storage text file sentence vectors using a deep learning encoder;

[0100] Using a wind-solar-storage text file classifier, a Softmax calculation is performed on a plurality of wind-solar-storage text file sentence vectors to obtain a first classification prediction probability corresponding to each wind-solar-storage text file;

[0101] Calculate the uncertainty of the first classification prediction probability of each wind, solar and storage text file;

[0102] When the uncertainty is lower than a preset uncertainty threshold, the prediction category of the corresponding wind, solar and storage text file is obtained based on the first classification prediction probability;

[0103] When the uncertainty is higher than a preset uncertainty threshold, the title vector of the wind and solar storage text file is calculated, and the second classification prediction probability of the wind and solar storage text file is calculated in combination with the sentence vector of the wind and solar storage text file, and the corresponding prediction category of the wind and solar storage text file is obtained based on the second classification prediction probability.

[0104] In the first embodiment of the present invention, the text files stored in the Fengguangchu text file library can be divided into 10 categories, such as core text files, leadership speeches, text file interpretations, text file related news, text file research papers, and text file research projects. There are large differences in the data attributes of different types of files, and usually the structured information of files from the same site column or search result list is similar. Therefore, it is necessary to divide the message topics in the message queue based on the source of the file for classification so that the files under the same message topic have the same data structure.

[0105] As an example of the first embodiment of the present invention, a wind and light storage text file x is set, and after deep learning network coding, the file is mapped into a wind and light storage text file sentence vector Where d is the dimension of the latent vector output by the deep learning network; h x The data is sent to the wind, solar and storage text file classifier for classification. The wind, solar and storage text file classifier consists of a fully connected layer, which is used to map the d-dimensional vector to N-dimensional, where N is the number of text file business categories. The Softmax calculation is performed on the mapping result to obtain the first classification prediction probability:

[0106] p 2 =Softmax(Wh x +b)

[0107] Where p is a probability vector, which represents the first classification prediction probability of the text file x in each category; W and b are the weight matrix and bias term of the fully connected layer, respectively.

[0108] Use cross entropy loss as the objective function for model optimization:

[0109]

[0110] In the formula, m is the number of files and N is the number of categories; is the true result of the i-th sample in the j-th category, 1 if it belongs to this category, otherwise 0; is the predicted probability that the i-th wind-solar-storage text class sample belongs to the j-th wind-solar-storage text class file.

[0111] In the first embodiment of the present invention, for the classification of wind, solar and storage text files, the probability distribution output by the classifier shows the certainty of the classification prediction of the text file sample to a certain extent. The lower the uncertainty of the prediction probability, the higher the accuracy of the prediction result. Therefore, after calculating the prediction probability of the category to which it belongs, the uncertainty of the prediction probability of the first classification is calculated first. The present invention calculates the uncertainty of the prediction probability of wind, solar and storage text files based on entropy. The larger the entropy, the greater the uncertainty of the random variable. For normalization processing, the present invention defines the sample prediction uncertainty as the ratio of the entropy of the prediction probability to the entropy of the uniform distribution, specifically:

[0112]

[0113] Where p(i) is the probability of predicting that the wind, solar and storage text file belongs to the i-th category, and N is the number of categories.

[0114] Based on the three-way decision classification method, for the first classification prediction probability with low uncertainty, the corresponding wind, solar and storage text file prediction category is directly output; for the first classification prediction probability with uncertainty exceeding the threshold, a delayed decision is made, and the feature representation of the text file is enhanced with the help of title information before classification again. The title representation vector of the wind, solar and storage text file is obtained through the twin network h t and h x The dimensions are the same, both are d-dimensional vectors. t and h x Add and fuse to obtain the scenery storage text file representation h after the title information is enhanced f ,h f =h x +h t . f Send it to the classifier for secondary classification to obtain a new second classification prediction probability p′, p′=Softmax(Wh f + b). Based on the second classification prediction probability, the category corresponding to the maximum probability is selected as the prediction category of the wind, solar and storage text files:

[0115]

[0116] In the formula, is the final predicted category for the i-th sample, p' i is the predicted probability distribution of the ith sample.

[0117] Further, in the first embodiment of the present invention, the wind and light storage text file library is specifically:

[0118] The text file library of Fengguang Storage includes MySQL database, file system library and index library;

[0119] The Mysql database is used to store data attribute information of wind and solar storage text files;

[0120] The file system library is used to store text files of various formats;

[0121] The index library is used to store structured information of text files.

[0122] In the first embodiment of the present invention, the functions of the wind and solar storage text file library include text file classification, text file association relationship analysis, file retrieval, file query, file download and file statistical analysis. The storage database in the wind and solar storage text file library includes a Mysql database, a file system library and an index library. The Mysql database is used to store the data attribute information of the file, the file system library is used to store the original html, txt, pdf, doc and other text files in various formats, and the index library such as Solr is used to store text files and some structured information to achieve full-text indexing.

[0123] Furthermore, in the first embodiment of the present invention, the wind and light storage text file library also includes:

[0124] Set permissions for users of the wind and solar storage text file library based on user type, specifically:

[0125] The management user settings include system management permissions for crawler configuration, exception handling, and file addition, deletion, modification, and query;

[0126] Set file analysis and viewing permissions for research users, including system search, file review and file download.

[0127] In the first embodiment of the present invention, users of the Fengguangchu text file library generally include two types of users: administrators and researchers. System management permissions such as crawler configuration, exception handling, file addition, deletion, modification and query are set for administrators, and permissions are set for researchers to retrieve, view and download files from the system, perform statistical analysis of text files and visualize the results.

[0128] In the first embodiment of the present invention, see Figure 2, is a flow chart of another embodiment of the application method of the wind and solar storage text file library provided by the present invention, the data acquisition subsystem acquires wind and solar storage text file data from various data sources, uses the data cleaning subsystem to perform data preprocessing on the acquired wind and solar storage text file data, uses the file screening subsystem to screen wind and solar storage text files from the wind and solar storage text file data, and inputs the wind and solar storage text files into the file analysis subsystem, so that the file analysis subsystem performs data deduplication, file classification and other analyses on the wind and solar storage text files according to the structured data extracted by the OCR subsystem, and stores the wind and solar storage text files and analysis results in the wind and solar storage text file library, and sets usage permissions for two users, the administrator and researcher of the wind and solar storage text file library.

[0129] In summary, the first embodiment of the present invention provides an application method of a wind and solar storage text file library, which collects wind and solar storage text file data from a preset data source; uses a preset C-TALBERT model to extract features of the wind and solar storage text file data, and screens out a number of wind and solar storage text files; uses the Simhash algorithm to determine the correlation between a number of wind and solar storage text files, and performs data deduplication; imports the deduplicated wind and solar storage text files into a first message queue; classifies a number of wind and solar storage text files in the message queue, and stores them in a wind and solar storage text file library. The present invention uses a wind and solar storage text file library to organize, analyze, and store wind and solar storage text files, strictly screens and deduplicates the collected data, removes useless data, and improves the efficiency of data analysis; and calculates the similarity of files, associates wind and solar storage text files, and improves the accuracy of file classification.

[0130] Example 2

[0131] See also Figure 3 , is a structural diagram of an embodiment of an application device for a wind and light storage text file library provided by the present invention, the device includes a collection module 201, a screening module 202, a deduplication module 203, an import module 204 and a classification module 205;

[0132] The collection module 201 is used to collect the text file data of the scenery storage in the preset data source;

[0133] The screening module 202 is used to extract features of the wind, solar and storage text files using a preset C-TALBERT model, and screen out a number of wind, solar and storage text files;

[0134] The deduplication module 203 is used to use the Simhash algorithm to determine the relevance between the plurality of wind, solar and storage text files to perform data deduplication;

[0135] The import module 204 is used to import the deduplicated copies of the wind and solar storage text files into the first message queue;

[0136] The classification module 205 is used to classify a plurality of the wind, solar and storage text files in the message queue and store them in a wind, solar and storage text file library.

[0137] Further, in the second embodiment of the present invention, the wind and solar storage text file data is collected in the preset data source, specifically:

[0138] Search for keywords of wind, solar and storage text files in the preset data source to obtain search results;

[0139] The search results are collected by using a web crawler, an incremental crawling scheduler, data attribute identification, crawler configuration or crawler exception management to form wind and solar storage text file data.

[0140] Furthermore, in the second embodiment of the present invention, the preset C-TALBERT model is used to extract features from the wind, solar and storage text file data, specifically:

[0141] Perform MLM and SOP pre-training tasks on the ALBERT model to form the C-TALBERT model;

[0142] Utilize the underlying Transformer of the C-TALBERT model to extract the surface information of the wind, solar and storage text file data to form a surface information feature representation;

[0143] The middle-layer Transformer of the C-TALBERT model is used to extract the syntactic information of the wind, solar and storage text file data to form a syntactic information feature representation;

[0144] The high-level Transformer of the C-TALBERT model is used to extract the semantic information of the wind and solar storage text file data to form a semantic information feature representation.

[0145] Furthermore, in the second embodiment of the present invention, the preset C-TALBERT model is used to extract features of the wind, solar and storage text file data, and a number of wind, solar and storage text files are screened out, specifically:

[0146] Combining surface information feature representation, syntactic information feature representation and semantic information feature representation into vectors to form several groups of file vectors;

[0147] The TextCNN convolutional neural network of the C-TALBERT model is used to perform multi-label classification tasks on several groups of file vectors, and several wind and solar storage text files are screened out.

[0148] Furthermore, in the second embodiment of the present invention, the Simhash algorithm is used to determine the relevance between several copies of the wind, light and storage text files to perform data deduplication, specifically:

[0149] Perform word segmentation, weight calculation and dimension reduction on a plurality of wind, solar and energy storage text files respectively, and obtain feature vectors corresponding to the plurality of wind, solar and energy storage text files;

[0150] Based on the feature vector, the Simhash algorithm is used to calculate the Simhash value of each wind, solar and storage text file; wherein the Simhash value is divided into preset parts for storage;

[0151] The similarity between each wind, solar and storage text file is calculated according to the Simhash value, and data deduplication processing is performed on several copies of the wind, solar and storage text files based on the similarity.

[0152] Further, in the second embodiment of the present invention, the plurality of deduplicated wind and solar storage text files are imported into the first message queue, specifically:

[0153] Use natural language processing technology and information extraction methods to perform text file analysis and structured data extraction on the deduplicated wind, solar and storage text files to obtain the structured data corresponding to each wind, solar and storage text file;

[0154] Converting the structured data into a format such that the structured data meets the data storage requirements of the message queue;

[0155] Creating a first message queue using a message queue system;

[0156] Import the structured data corresponding to each wind, solar and storage text file into the first message queue in batches.

[0157] Furthermore, in the second embodiment of the present invention, several copies of the wind and light storage text files are classified in the message queue, specifically:

[0158] Mapping the plurality of wind and solar storage text files into a plurality of wind and solar storage text file sentence vectors using a deep learning encoder;

[0159] Using a wind-solar-storage text file classifier, a Softmax calculation is performed on a plurality of wind-solar-storage text file sentence vectors to obtain a first classification prediction probability corresponding to each wind-solar-storage text file;

[0160] Calculate the uncertainty of the first classification prediction probability of each wind, solar and storage text file;

[0161] When the uncertainty is lower than a preset uncertainty threshold, the prediction category of the corresponding wind, solar and storage text file is obtained based on the first classification prediction probability;

[0162] When the uncertainty is higher than a preset uncertainty threshold, the title vector of the wind and solar storage text file is calculated, and the second classification prediction probability of the wind and solar storage text file is calculated in combination with the sentence vector of the wind and solar storage text file, and the corresponding prediction category of the wind and solar storage text file is obtained based on the second classification prediction probability.

[0163] Further, in the second embodiment of the present invention, the wind and light storage text file library is specifically:

[0164] The text file library of Fengguang Storage includes MySQL database, file system library and index library;

[0165] The Mysql database is used to store data attribute information of wind and solar storage text files;

[0166] The file system library is used to store text files of various formats;

[0167] The index library is used to store structured information of text files.

[0168] Furthermore, in the second embodiment of the present invention, the wind and light storage text file library also includes:

[0169] Set permissions for users of the wind and solar storage text file library based on user type, specifically:

[0170] The management user settings include system management permissions for crawler configuration, exception handling, and file addition, deletion, modification, and query;

[0171] Set file analysis and viewing permissions for research users, including system search, file review and file download.

[0172] In summary, the second embodiment of the present invention provides an application device for a wind and solar storage text file library, which collects wind and solar storage text file data in a preset data source based on the organic combination of modules; uses a preset C-TALBERT model to extract features of the wind and solar storage text file data, and screens out a number of wind and solar storage text files; uses the Simhash algorithm to determine the correlation between a number of wind and solar storage text files, and performs data deduplication; imports the deduplicated wind and solar storage text files into a first message queue; classifies a number of wind and solar storage text files in the message queue, and stores them in a wind and solar storage text file library. The present invention uses a wind and solar storage text file library to organize, analyze, and store wind and solar storage text files, strictly screens and deduplicates the collected data, removes useless data, and improves the efficiency of data analysis; and calculates the similarity of files, associates wind and solar storage text files, and improves the accuracy of file classification.

[0173] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. An application method of a wind and solar storage text file library, characterized in that: include: Collecting text file data of wind, solar and storage in the preset data source; Using a preset C-TALBERT model to extract features from the wind, solar and energy storage text file data, and screening out a number of wind, solar and energy storage text files; The Simhash algorithm is used to determine the relevance between several wind, solar and storage text files to remove duplicate data; Importing the deduplicated copies of the wind and solar storage text files into the first message queue; Classifying a plurality of the wind and solar storage text files in a message queue and storing them in a wind and solar storage text file library; The preset C-TALBERT model is used to extract features from the wind, solar and storage text file data, and a number of wind, solar and storage text files are selected, specifically: Perform MLM and SOP pre-training tasks on the ALBERT model to form the C-TALBERT model; Utilize the underlying Transformer of the C-TALBERT model to extract the surface information of the wind, solar and storage text file data to form a surface information feature representation; The middle-layer Transformer of the C-TALBERT model is used to extract the syntactic information of the wind, solar and storage text file data to form a syntactic information feature representation; The high-level Transformer of the C-TALBERT model is used to extract the semantic information of the wind, solar and storage text file data to form a semantic information feature representation; Combining surface information feature representation, syntactic information feature representation and semantic information feature representation into vectors to form several groups of file vectors; The TextCNN convolutional neural network of the C-TALBERT model is used to perform multi-label classification tasks on several groups of file vectors, and several wind, solar and storage text files are screened out; The classification of several wind and solar storage text files in the message queue is specifically as follows: Mapping the plurality of wind and solar storage text files into a plurality of wind and solar storage text file sentence vectors using a deep learning encoder; Using a wind-solar-storage text file classifier, a Softmax calculation is performed on a plurality of wind-solar-storage text file sentence vectors to obtain a first classification prediction probability corresponding to each wind-solar-storage text file; Calculate the uncertainty of the first classification prediction probability of each wind, solar and storage text file; When the uncertainty is lower than a preset uncertainty threshold, the prediction category of the corresponding wind, solar and storage text file is obtained based on the first classification prediction probability; When the uncertainty is higher than a preset uncertainty threshold, the title vector of the wind and solar storage text file is calculated, and the second classification prediction probability of the wind and solar storage text file is calculated in combination with the sentence vector of the wind and solar storage text file, and the corresponding prediction category of the wind and solar storage text file is obtained based on the second classification prediction probability.

2. The application method of the wind and light storage text file library according to claim 1 is characterized in that: The method of collecting the text file data of the wind, solar and storage system from the preset data source is as follows: Search for keywords of wind, solar and storage text files in the preset data source to obtain search results; The search results are collected by using a web crawler, an incremental crawling scheduler, data attribute identification, crawler configuration or crawler exception management to form wind and solar storage text file data.

3. The application method of the wind and light storage text file library according to claim 2 is characterized in that: The Simhash algorithm is used to determine the relevance between several wind, solar and storage text files and perform data deduplication, specifically: Perform word segmentation, weight calculation and dimension reduction on a plurality of wind, solar and energy storage text files respectively, and obtain feature vectors corresponding to the plurality of wind, solar and energy storage text files; Based on the feature vector, the Simhash algorithm is used to calculate the Simhash value of each wind, solar and storage text file; wherein the Simhash value is divided into preset parts for storage; The similarity between each wind, solar and storage text file is calculated according to the Simhash value, and data deduplication processing is performed on several copies of the wind, solar and storage text files based on the similarity.

4. The application method of the wind and light storage text file library according to claim 3 is characterized in that: The step of importing the deduplicated copies of the wind and solar storage text files into the first message queue is specifically as follows: Use natural language processing technology and information extraction methods to perform text file analysis and structured data extraction on the deduplicated wind, solar and storage text files to obtain the structured data corresponding to each wind, solar and storage text file; Converting the structured data into a format such that the structured data meets the data storage requirements of the message queue; Creating a first message queue using a message queue system; Import the structured data corresponding to each wind, solar and storage text file into the first message queue in batches.

5. The application method of the wind and light storage text file library according to claim 4 is characterized in that: The wind and solar storage text file library is specifically: The text file library of Fengguang Storage includes MySQL database, file system library and index library; The Mysql database is used to store data attribute information of wind and solar storage text files; The file system library is used to store text files of various formats; The index library is used to store structured information of text files.

6. The application method of the wind and light storage text file library according to claim 5 is characterized in that: The wind and solar storage text file library also includes: Set permissions for users of the wind and solar storage text file library based on user type, specifically: The management user settings include system management permissions for crawler configuration, exception handling, and file addition, deletion, modification, and query; Set file analysis and viewing permissions for research users, including system search, file review and file download.

7. An application device for a text-based file library, characterized in that: include: Acquisition module, screening module, deduplication module, import module and classification module; The acquisition module is used to collect text file data of wind, solar and storage in the preset data source; The screening module is used to extract features of the wind, solar and storage text file data using a preset C-TALBERT model, and screen out a number of wind, solar and storage text files; The deduplication module is used to use the Simhash algorithm to determine the relevance between several copies of the wind, solar and storage text files to perform data deduplication; The import module is used to import the deduplicated copies of the wind and solar storage text files into the first message queue; The classification module is used to classify a plurality of the wind and solar storage text files in the message queue and store them in a wind and solar storage text file library; The screening module is specifically: Perform MLM and SOP pre-training tasks on the ALBERT model to form the C-TALBERT model; Utilize the underlying Transformer of the C-TALBERT model to extract the surface information of the wind, solar and storage text file data to form a surface information feature representation; The middle-layer Transformer of the C-TALBERT model is used to extract the syntactic information of the wind, solar and storage text file data to form a syntactic information feature representation; The high-level Transformer of the C-TALBERT model is used to extract the semantic information of the wind, solar and storage text file data to form a semantic information feature representation; Combining surface information feature representation, syntactic information feature representation and semantic information feature representation into vectors to form several groups of file vectors; The TextCNN convolutional neural network of the C-TALBERT model is used to perform multi-label classification tasks on several groups of file vectors, and several wind, solar and storage text files are screened out; The classification module is specifically: Mapping the plurality of wind and solar storage text files into a plurality of wind and solar storage text file sentence vectors using a deep learning encoder; Using a wind-solar-storage text file classifier, a Softmax calculation is performed on a plurality of wind-solar-storage text file sentence vectors to obtain a first classification prediction probability corresponding to each wind-solar-storage text file; Calculate the uncertainty of the first classification prediction probability of each wind, solar and storage text file; When the uncertainty is lower than a preset uncertainty threshold, the prediction category of the corresponding wind, solar and storage text file is obtained based on the first classification prediction probability; When the uncertainty is higher than a preset uncertainty threshold, the title vector of the wind and solar storage text file is calculated, and the second classification prediction probability of the wind and solar storage text file is calculated in combination with the sentence vector of the wind and solar storage text file, and the corresponding prediction category of the wind and solar storage text file is obtained based on the second classification prediction probability.

Images