Construction method and system of industrial data multi-classification model and application of construction method and system

Abstract

The invention provides a construction method and system of an industrial data multi-classification model and application of the industrial data multi-classification model, and relates to the technical field of language analysis and classification process.The method comprises the steps that special structure metadata of industrial data is obtained; classifying and marking the special structure metadata to obtain an initial label of the special structure metadata; pre-training the language representation model based on the special structure metadata to obtain a pre-trained language model of the special structure metadata; and constructing a continuous prompt template on the pre-training language model through the special structure metadata containing the initial label, and obtaining an industrial data multi-classification model according to the continuous prompt template. According to the method, text classification of industrial field data is carried out through the language representation model based on the continuous prompt template, the gap between upstream and downstream tasks is reduced, the constructed multi-classification model is high in accuracy, and then text multi-classification can be accurately achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of language analysis and classification processing, and in particular to an industrialized data multi-classification method, system, device and medium. Background Art

[0002] With the rapid development of the Internet, the speed of data generation is also increasing. According to statistics, the global Internet generates 1.826PB (petabyte) of data every day. At the same time, with the advancement of informatization in various industries, a large amount of industry data is also generated every day, such as communication satellites, meteorology, water and electricity data. These data also contain rich value. Only by effectively managing and developing data can the value of data be fully explored. How to effectively manage and govern data has become the basis for the development of the digital economy. Carrying out industrial data classification and grading can effectively improve the data management of manufacturing enterprises. Industrial data classification and grading is the premise for realizing differentiated security protection management of enterprise data, and is a necessary means for enterprise data to be orderly shared, circulated, developed and utilized, and to release the maximum value. Industrial big data is the digital description of industrial information and the storage image in cyberspace. In addition to the common characteristics of big data, such as large scale, fast speed, diverse types, and low quality, industrial big data also has the characteristics of multi-domain, systematic, and high throughput. Multi-domain means that the data generated by industrial big data comes from different disciplines in the industrial field. For example, the power system can involve multiple fields such as mechanics, electromagnetics, electricity, and thermodynamics at the same time. Systematic means that industrial big data is usually interrelated, such as between product components, between production steps, and the design, manufacturing, and maintenance of the product life cycle. High throughput means that industrial automation equipment usually writes a large amount of data in a short time. Smart devices embedded with sensors have become an important part of the industrial Internet. How to simplify the huge industrial big data has become the focus of data management. Text classification is usually used as a basic task to understand industrial big data. Short texts are increasingly regarded by the industry as an excellent way to operate and manage industrial big data because of their short content, easy reading and dissemination.

[0003] In the processing of industrial data, the existing technology only relies on fine-tuning the pre-trained language model to achieve text classification. The multi-classification model obtained in this way has poor reliability and cannot well implement the text classification task of industrial data. Summary of the invention

[0004] In order to overcome the above-mentioned deficiencies of the prior art, the present invention proposes a method for constructing an industrial data multi-classification model, comprising:

[0005] Get specialized structured metadata for industrialized data;

[0006] Classify and label the dedicated structure metadata to obtain an initial label of the dedicated structure metadata;

[0007] Pre-training a language representation model based on the dedicated structural metadata to obtain a pre-trained language model of the dedicated structural metadata;

[0008] A continuous prompt template is built on the pre-trained language model through dedicated structure metadata containing initial labels, and an industrial data multi-classification model is obtained based on the continuous prompt template.

[0009] Preferably, the method further comprises:

[0010] Select the data short text set as the training set;

[0011] The continuous prompt template is optimized through the training set to obtain an optimized continuous prompt template, and an optimized industrial data multi-classification model is obtained according to the optimized continuous prompt template.

[0012] Preferably, a data short text set is selected as a training set, including:

[0013] A random sampling method is used to select some data short texts from the data short text set as the prompt template training set, and a specified proportion of data short texts are extracted from the remaining data short texts in the data short text set as the label expansion training set, and the remaining data short texts are used as the template test set.

[0014] Preferably, the continuous prompt template is optimized through the training set to obtain an optimized continuous prompt template, and an optimized industrial data multi-classification model is obtained according to the optimized continuous prompt template, including:

[0015] Optimizing the parameters of the continuous prompt template through the prompt template training set to obtain the initially optimized continuous prompt template, and obtaining the initially optimized industrial data multi-classification model based on the initially optimized continuous prompt template;

[0016] According to the initially optimized continuous prompt template, the initial label is expanded through the label expansion training set to obtain the extended label of the special structure metadata; according to the extended label, the initially optimized industrial data multi-classification model is trained through the template test set, the initially optimized continuous prompt template is adjusted, and the re-optimized industrial data multi-classification model is obtained according to the adjusted continuous prompt template.

[0017] Preferably, obtaining the dedicated structure metadata of industrialized data includes:

[0018] Obtain data characteristics of industrialized data;

[0019] The common features of data characteristics are extracted, and specialized structured metadata for industrialized data is constructed based on the common features.

[0020] Preferably, classifying and labeling the dedicated structure metadata to obtain an initial label of the dedicated structure metadata includes:

[0021] Classify and mark the dedicated structure metadata to obtain data features of the dedicated structure metadata;

[0022] Mapping the dedicated structure metadata to corresponding initial labels in combination with data features of the dedicated structure metadata;

[0023] The types of initial labels include one or more of the following: regular identifiable labels, text identifiable labels, enumeration labels, numerical quantity labels, encoding labels, and remark description labels.

[0024] Preferably, pre-training the language representation model based on the dedicated structural metadata to obtain a pre-trained language model of the dedicated structural metadata includes:

[0025] The dedicated structure metadata is segmented at the granularity of tokens to obtain multiple short texts of the dedicated structure metadata;

[0026] Connect multiple short texts, input the connected multiple short texts into the language representation model, and obtain the context representation of each character in the short text;

[0027] The multi-head attention mechanism of the transformer layer in the language representation model is used to understand the contextual representation of each character in the short text, output the embedding vector of each character, and obtain the contextual embedding initialization vector of each character;

[0028] The language representation model is pre-trained according to the context embedding initialization vector of each character to obtain a pre-trained language model of dedicated structure metadata.

[0029] Preferably, constructing a continuous prompt template on a pre-trained language model by means of dedicated structure metadata including initial labels comprises:

[0030] Continuously prompt each transformation layer of the pre-trained language model;

[0031] Cue-based fine-tuning P-Tunning adds the continuous cue in each transformation layer as a prefix tag to the input sequence to obtain the continuous cue parameters for a specific task;

[0032] The continuous hint template is constructed on the pre-trained language model using an automatic gradient descent algorithm, using dedicated structured metadata containing initial labels and task-specific continuous hint parameters.

[0033] Based on the same inventive concept, the present invention also provides a system for constructing an industrial data multi-classification model, including:

[0034] Dedicated structural metadata acquisition module, initial label acquisition module, pre-trained language model acquisition module and multi-classification model acquisition module;

[0035] A dedicated structure metadata acquisition module is used to acquire dedicated structure metadata of industrialized data;

[0036] An initial label acquisition module is used to classify and label the dedicated structure metadata to obtain the initial label of the dedicated structure metadata;

[0037] A pre-trained language model acquisition module is used to pre-train the language representation model based on the dedicated structural metadata to obtain a pre-trained language model of the dedicated structural metadata;

[0038] The multi-classification model acquisition module is used to build a continuous prompt template on the pre-trained language model through dedicated structure metadata containing initial labels, and obtain an industrialized data multi-classification model according to the continuous prompt template.

[0039] Preferably, the system further comprises: a multi-classification model optimization module;

[0040] Multi-classification model optimization module, including:

[0041] The training set selection submodule is used to select a data short text set as a training set;

[0042] The industrial data multi-classification model optimization submodule is used to optimize the continuous prompt template through the training set to obtain the optimized continuous prompt template, and obtain the optimized industrial data multi-classification model according to the optimized continuous prompt template.

[0043] Preferably, the training set selection submodule is specifically used for:

[0044] A random sampling method is used to select some data short texts from the data short text set as the prompt template training set, and a specified proportion of data short texts are extracted from the remaining data short texts in the data short text set as the label expansion training set, and the remaining data short texts are used as the template test set.

[0045] Preferably, the industrial data multi-classification model optimization submodule includes:

[0046] An industrial data multi-classification model initial optimization unit is used to optimize the parameters of the continuous prompt template through the prompt template training set to obtain the initially optimized continuous prompt template, and obtain the initially optimized industrial data multi-classification model according to the initially optimized continuous prompt template;

[0047] The industrial data multi-classification model re-optimization unit is used to expand the initial label through the label expansion training set according to the initial optimized continuous prompt template, so as to obtain the extended label of the special structure metadata; train the initially optimized industrial data multi-classification model through the template test set according to the extended label, adjust the initially optimized continuous prompt template, and obtain the re-optimized industrial data multi-classification model according to the adjusted continuous prompt template.

[0048] Preferably, the dedicated structure metadata acquisition module is specifically used for:

[0049] Obtain data characteristics of industrialized data;

[0050] Common features of the data features are extracted, and dedicated structured metadata of the industrialized data is constructed according to the common features.

[0051] Preferably, the initial tag acquisition module includes:

[0052] A data feature acquisition submodule is used to classify and mark the dedicated structure metadata and obtain data features of the dedicated structure metadata;

[0053] An initial label mapping submodule, used to map the dedicated structure metadata to the corresponding initial label in combination with the data features of the dedicated structure metadata;

[0054] The types of initial labels include one or more of the following: regular identifiable labels, text identifiable labels, enumeration labels, numerical quantity labels, encoding labels, and remark description labels.

[0055] Preferably, the pre-trained language model acquisition module includes:

[0056] Multiple short text acquisition submodules are used to segment the dedicated structure metadata at the granularity of tokens to obtain multiple short texts of the dedicated structure metadata;

[0057] A character context representation acquisition submodule is used to connect multiple short texts, input the connected multiple short texts into the language representation model, and obtain the context representation of each character in the short text;

[0058] The initialization vector acquisition submodule is used to understand the contextual representation of each character in the short text through the multi-head attention mechanism of the transformer layer in the language representation model, output the embedding vector of each character, and obtain the contextual embedding initialization vector of each character;

[0059] The pre-trained language model acquisition submodule is used to pre-train the language representation model according to the context embedding initialization vector of each character to obtain a pre-trained language model of dedicated structural metadata.

[0060] Preferably, the multi-classification model acquisition module includes:

[0061] The continuous prompting submodule is used to continuously prompt each transformation layer of the pre-trained language model;

[0062] The continuous cue parameter acquisition submodule is used to fine-tune P-Tunning based on the cue, adding the continuous cue in each transformation layer as a prefix tag to the input sequence to obtain the continuous cue parameters for a specific task;

[0063] The continuous hint template construction submodule is used to build a continuous hint template on the pre-trained language model using an automatic gradient descent algorithm through dedicated structure metadata containing initial labels and task-specific continuous hint parameters.

[0064] Based on the same inventive concept, the present invention also provides a method for multi-classification of industrial data, which is characterized by comprising:

[0065] Obtain industrial data to be processed;

[0066] The industrial data multi-classification model as described above is used to perform multi-classification processing on the industrial data to be processed.

[0067] Based on the same inventive concept, the present invention also provides an industrial data multi-classification system, comprising:

[0068] A module for acquiring data to be processed, used for acquiring industrial data to be processed;

[0069] The multi-classification processing module is used to perform multi-classification processing on the industrial data to be processed using the multi-classification model for industrial data as described above.

[0070] Based on the same inventive concept, the present invention also provides an electronic device, comprising: at least one processor and a memory; the memory and the processor are connected via a bus;

[0071] The memory is used to store one or more programs;

[0072] When the one or more programs are executed by the at least one processor, a method for constructing an industrial data multi-classification model as described above is implemented, or a method for multi-classification of industrial data as described above is implemented.

[0073] Based on the same inventive concept, the present invention also provides a readable storage medium having an execution program stored thereon. When the execution program is executed, it implements a method for constructing an industrial data multi-classification model as described above, or implements a method for multi-classification of industrial data as described above.

[0074] Compared with the closest prior art, the present invention has the following beneficial effects:

[0075] The present invention provides a method for constructing a multi-classification model of industrialized data, comprising: obtaining special structural metadata of industrialized data; classifying and labeling the special structural metadata to obtain an initial label of the special structural metadata; pre-training a language representation model based on the special structural metadata to obtain a pre-trained language model of the special structural metadata; constructing a continuous prompt template on the pre-trained language model through the special structural metadata containing the initial label, and obtaining a multi-classification model of industrialized data according to the continuous prompt template. The present invention performs text classification of industrial field data through a language representation model based on a continuous prompt template, narrows the gap between upstream and downstream tasks, and constructs a multi-classification model with high accuracy, thereby being able to accurately realize text multi-classification.

[0076] The present invention also provides a method for multi-classification of industrial data, comprising: obtaining industrial data to be processed; and performing multi-classification processing on the industrial data to be processed using the multi-classification model of industrial data constructed as described above. The present invention performs multi-classification processing on the industrial data to be processed using the multi-classification model of industrial data. Since the constructed multi-classification model has high accuracy, the multi-classification model can be used to accurately realize text multi-classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 A schematic diagram of a process for constructing a multi-classification model for industrial data provided by the present invention;

[0078] Figure 2 A specific example flow chart of the method for constructing an industrial data multi-classification model provided by the present invention;

[0079] Figure 3 A flowchart of a method for constructing a multi-classification model of power data provided by the present invention;

[0080] Figure 4 A schematic diagram of the structure of a system for constructing a multi-classification model for industrial data provided by the present invention;

[0081] Figure 5 A schematic diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0082] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0083] Embodiment 1:

[0084] The present invention provides a method for constructing an industrial data multi-classification model, specifically, Figure 1 The flowchart of the method for constructing a multi-classification model of industrial data provided by an embodiment of the present invention includes the following steps as shown in the figure:

[0085] S1: Acquire specialized structural metadata for industrialized data;

[0086] S2: classify and label the dedicated structure metadata to obtain the initial label of the dedicated structure metadata;

[0087] S3: pre-training the language representation model based on the dedicated structural metadata to obtain a pre-trained language model of the dedicated structural metadata;

[0088] S4: A continuous prompt template is constructed on the pre-trained language model through dedicated structure metadata containing initial labels, and an industrial data multi-classification model is obtained based on the continuous prompt template.

[0089] The present invention performs text classification of industrial data through a language representation model based on a continuous prompt template, thereby narrowing the gap between upstream and downstream tasks, and constructing a multi-classification model with high accuracy, thereby being able to accurately achieve text multi-classification.

[0090] The present invention is used to construct a multi-classification model for industrialized data, and special structural metadata for industrialized data is obtained in S1. Since the data types in different industrial fields vary greatly, it is necessary to select the type of industrialized data for which a multi-classification model needs to be constructed according to specific tasks in the early stage of data governance. For example, it may be power system data, mechanical system data, or industrial and mining system data. It is necessary to select the type of industrialized data for constructing a multi-classification model according to the specific task, and then construct the special structural metadata of this type according to the type of industrialized data. It can be understood that the special structural metadata constructed according to the same type of industrialized data are the same.

[0091] In some implementations, after selecting the type of industrial data, data features of the industrial data under this type are obtained, and then common features of these data features are extracted, and dedicated structure metadata is constructed according to the common features of the data features.

[0092] Special structured metadata is used to support functions such as indicating storage location, historical data, resource search, file records, etc., or it is structured data (structured data) used to provide information about a certain resource. Special structured metadata is constructed according to the meaning and purpose of industrial data, as well as the expression form, classification method, and storage data type of the data. Special structured metadata is an electronic directory. In order to clarify the metadata management goals and achieve the purpose of cataloging, it is necessary to describe and collect the content or characteristics of the data, thereby achieving the purpose of assisting data retrieval. It is required to achieve unified management of statistical data, that is, if the industrial data is power system data, the directories of many power data source units can be managed to facilitate the extraction and reference of data at any time.

[0093] After obtaining the dedicated structure metadata, the dedicated structure metadata is classified and labeled through S2 to obtain the initial label of the dedicated structure metadata. In some embodiments, the dedicated structure metadata is classified and labeled to obtain the data features of the dedicated structure metadata; the dedicated structure metadata is mapped to the corresponding initial label in combination with the data features of the dedicated structure metadata. The types of the initial label include one or more of the following: regular identifiable class labels, text identifiable class labels, enumeration class labels, numerical quantity class labels, encoding class labels, and remark description class labels.

[0094] Specifically, the special structure metadata is classified and processed respectively, the characteristics of the special structure metadata are found, and the characteristics of the special structure metadata are retained during the classification process. Then, these special structure metadata are mapped to appropriate labels, maintaining most of the characteristics of the original industrialized data and improving its robustness.

[0095] S3 pre-trains the language representation model based on the dedicated structural metadata to obtain a pre-trained language model of the dedicated structural metadata, including:

[0096] Segment the dedicated structure metadata at the granularity of tokens to obtain multiple short texts of the dedicated structure metadata;

[0097] Connect multiple short texts, input the connected multiple short texts into the language representation model, and obtain the context representation of each character in the short text;

[0098] The multi-head attention mechanism of the transformer layer in the language representation model is used to understand the contextual representation of each character in the short text, output the embedding vector of each character, and obtain the contextual embedding initialization vector of each character;

[0099] The language representation model is pre-trained according to the context embedding initialization vector of each character to obtain a pre-trained language model of dedicated structure metadata.

[0100] Since the special structure metadata is in the form of short texts, the content of short texts is brief, easy to read and spread, and widely used by the public. It can be found everywhere in the industrial data of various industries. However, the brevity and missing content will cause data sparsity. Unlike paragraphs or articles, the syntax of short texts cannot always be observed, and the ambiguity and errors in short texts often make the content unclear and the semantics difficult to understand, resulting in insufficient features for model classification to make category judgments, which in turn makes the classification task difficult.

[0101] In order to overcome the difficulties in short text classification, it is necessary to obtain more semantic and syntactic information and require more advanced text expressions. The present invention can use the deep learning BERT (Bidirectional Encoder Representation from Transformers) model as a language representation model to analyze the special structure metadata. The BERT model is based on the Transformer language model, pre-trains the Encoder network of the Transformer model, and uses the Transformer structural self-supervised learning method to perform pre-training on large-scale unlabeled corpus, thereby capturing the rich semantic information in the special structure metadata.

[0102] The following are some commonly used BERT models. Different BERT models have different characteristics and are suitable for different usage scenarios. As shown below:

[0103] BERT-base: This is the most basic BERT model, with 12 Transformer encoder layers and 110M parameters. The input embedding dimension of BERT-base is 768, and the dimension of the hidden layer is also 768.

[0104] BERT-large (BERT-big data): Larger than BERT-base, it contains 24 Transformer encoder layers with a total of 340M parameters. The embedding vector dimension and hidden layer dimension are both 1024.

[0105] BERT-wwm (BERT-whole word): is an improved version of BERT, which uses the whole word masking method for pre-training and can better handle the Chinese word segmentation problem.

[0106] BERT-multilingual: BERT is a multilingual BERT model that can process texts in multiple languages ​​at the same time. The pre-training tasks of this model include large-scale texts from multiple languages.

[0107] BERT-uncased: A model trained by converting uppercase letters in English text to lowercase letters, which is suitable for case-insensitive tasks.

[0108] BERT-cased: A model trained by preserving the case of uppercase and lowercase letters in English text, which is suitable for case-sensitive tasks.

[0109] In the present invention, since the data object processed is Chinese short text data, BERT-wwm can be used to pre-train the dedicated structure metadata.

[0110] The BERT model is used through pre-training and fine-tuning. The pre-training is carried out according to two tasks, namely the Masked Language Model (MLM) task and the Next Sentence Prediction (NSP) task.

[0111] The Masked Language Task (MLM) is to randomly cover the words in the input sentence with the [MASK] label, and then use the BERT model to predict the characters at the MASK position. This training task is used to obtain the vector (embedding) corresponding to each character. In particular, the [CLS] character can be added, which can be understood as the [CLS] character has nothing to do with other characters in the sentence, and the entire sentence can be considered fairly. The vector (embedding) of the [CLS] character is regarded as the vector (embedding) of the entire sentence.

[0112] The sentence prediction task (NSP) is to determine whether the second sentence follows the first sentence in the text given two sentences in an article. If the question and answer are input into the BERT model as context during training, this task can also be understood as determining whether the question and answer match.

[0113] After constructing the BERT-wwm model through the masked language task and the sentence prediction task, the constructed BERT-wwm model is further pre-trained using the patented structural metadata without labels, and segmented at the granularity of tokens to obtain multiple short texts of dedicated structural metadata; then the short text is connected with multiple other short texts, and the connected multiple short texts are input into the Transformer layer of the BERT-wwm model to obtain the contextual representation of each character in the short text; the contextual representation of each character is understood through the multi-head attention mechanism of the Transformer layer, and the embedding vector of each character is output to obtain the contextual embedding initialization vector of each character; finally, the BERT-wwm model is further pre-trained according to the contextual embedding initialization vector of each character to obtain a pre-trained language model of dedicated structural metadata without labels.

[0114] After processing the unlabeled dedicated structure metadata to obtain the pre-trained language model, S4 constructs a continuous prompt template on the pre-trained language model through the dedicated structure metadata containing the initial label, and then obtains the industrialized data multi-classification model based on the constructed continuous prompt template.

[0115] S4 includes:

[0116] Continuously prompt each transformer layer of the pre-trained language model;

[0117] P-Tunning adds the continuous prompts in each Transformer layer as prefix tokens to the input sequence to obtain the continuous prompt parameters for a specific task.

[0118] By including dedicated structural metadata of initial labels and continuous prompt parameters for specific tasks, an automatic gradient descent algorithm is used to build a continuous prompt template on the pre-trained language model, and then an industrial data multi-classification model is obtained based on the constructed continuous prompt template.

[0119] Among them, in the present invention, prompt fine-tuning (P-Tunning) can be used to construct a prompt template. P-Tunning is not to randomly initialize several new tokens for direct training, but to calculate these vectors (embedding) through a small LSTM (Long Short-Term Memory, long short-term memory network) model, and this LSTM model is set to be learnable.

[0120] In the present invention, prompt fine-tuning P-TunningV2 can be used to fine-tune the prompt words. P-TunningV2 uses deep prompt tunning as an excellent technology applied to entity recognition tasks, which can be migrated to the field of text classification. P-TunningV2 only adjusts a small number of continuous prompts, which greatly reduces the storage and memory usage of each task compared to traditional fine-tuning methods. The core of P-TunningV2 is to introduce continuous trainable prompts at each layer of the model, not just the input layer. This approach significantly improves the adaptability and flexibility of the model, enabling it to better handle complex natural language understanding (NLU) tasks.

[0121] Compared with P-TunningV1, P-TunningV2 improves the model's ability to process complex information by applying prompts at each layer of the model, especially when dealing with difficult tasks such as sequence labeling, showing better performance. Compared with traditional full parameter fine-tuning (Full Fine-Tunning, FFT), P-TunningV2 only needs to adjust a very small proportion of parameters, greatly reducing the resource consumption of training and storage.

[0122] Specifically, in the present invention, the prompts of different transformation layers are added as prefix tags to the input sequence, and are independent of each other between layers, rather than being calculated by the previous transformation layer. Adding continuous prompts to the front of the sequence and adding trainable prompts to each layer can allow the model to have more task-specific continuous prompt parameters. Then, a continuous prompt template is constructed on the pre-trained language model using an automatic gradient descent algorithm using dedicated structured metadata containing the initial label and the continuous prompt parameters for the specific task.

[0123] So far, the present invention provides a method for constructing a multi-classification model of industrialized data, in which a pre-trained language model based on a prompt template is fine-tuned to realize text classification in industrialized data, thereby improving the training effect of small samples, and the prompt template used is a continuous vectorized template, which has more adjustment and optimization space than the traditional discrete natural language template. The multi-classification model of industrialized data constructed in the above manner has high accuracy and can realize text multi-classification more accurately.

[0124] In some embodiments, after obtaining the industrial data multi-classification model according to the continuous prompt template, external knowledge can also be introduced, such as optimizing the industrial data multi-classification model through a data short text set, to further improve the performance of the multi-classification model, so that it can better realize text multi-classification.

[0125] Select the data short text set as the training set;

[0126] The continuous prompt template is optimized through the training set to obtain an optimized continuous prompt template, and an optimized industrial data multi-classification model is obtained according to the optimized continuous prompt template.

[0127] Firstly, external knowledge is introduced to obtain a training set. Specifically, a part of the data short texts are selected from the data short text set by random sampling as the prompt template training set, and a rated proportion of the data short texts are extracted from the remaining data short texts in the data short text set as the label expansion training set, and the remaining short texts are used as the template test set.

[0128] In the present invention, a random sampling method can be used to extract 10% of the data short texts in the data short text set as a prompt template training set, 10% of the data short texts can be extracted as a label expansion training set, and the remaining data short texts can be used as a template test set.

[0129] After obtaining various training sets, the continuous prompt template parameters are optimized through the prompt template training set to obtain the initially optimized continuous prompt template, and the initially optimized industrial data multi-classification model is obtained according to the initially optimized continuous prompt template;

[0130] Furthermore, the initial labels are expanded through the label expansion training set according to the initially optimized continuous prompt template to obtain the expanded labels of the dedicated structured metadata; the initially optimized industrial data multi-classification model is trained through the template test set according to the expanded labels, and the initially optimized continuous prompt template is adjusted; the industrial data multi-classification model after optimization is obtained again according to the adjusted continuous prompt template. The label words are expanded and improved by introducing external knowledge to better achieve the text classification effect.

[0131] Specifically, in the present invention, if a multi-classification model of power data is constructed, 10% of the short texts of power data are selected as the prompt template training set, and the parameters of the prompt template are optimized according to the stochastic gradient descent algorithm, so as to obtain the multi-classification model of industrialized data after the initial optimization. The constructed pre-training model for learning a large number of power-related knowledge and the constructed prompt template suitable for this data set are used to constrain the vocabulary generation of the BERT model, that is, the BERT model containing power professional knowledge is used to apply the existing prompt template for prediction. The data set of this model can be less than 10,000. Experiments have shown that the effect of using the stochastic gradient descent algorithm is better than that of the small batch gradient descent algorithm. Therefore, the stochastic gradient descent algorithm is used in the present invention to optimize the parameters of the prompt template.

[0132] Then select 10% of the short texts of power data as the label expansion training set, perform the text prediction task according to the initially optimized prompt template, and predict the new label through the existing initial label. Expand the regular recognizable class labels, text recognizable class labels, enumeration class labels, numerical quantity class labels, encoding class labels, and note description class labels into 10-20 more detailed label classes to increase the dimension of the trainable data set. Specifically, using Prompt Learning for text classification tasks is an emerging way to use pre-trained language models. In prompt learning, a label word mapping (verbalizer) is required to convert the prediction of the vocabulary in the word list at the [MASK] position into a classification label. For example, under the mapping {POLITICS:"politics",SPORTS:"sports"}, the prediction score of the pre-trained model for the label word politics / sports at the [MASK] position will be regarded as the prediction score for the label POLITICS / SPORTS.

[0133] After expanding the initial labels, all the remaining 80% of the power data short texts are selected as the template test set. The industrial data multi-classification model after the initial optimization is trained according to the expanded labels, and the continuous prompt template is adjusted to obtain the industrial data multi-classification model after further optimization, so that the industrial data multi-classification model achieves the optimal performance.

[0134] Compared with binary text classification, multi-classification of text poses a higher challenge to the performance of the model and requires the model to learn semantic information more finely. At the same time, the dataset of the present invention has less metadata, and the classification is highly dependent on expert knowledge and is difficult to expand. Therefore, a large-scale pre-trained model based on the prompt paradigm is used to complete the classification task.

[0135] The entire optimization process of the prompt template includes: first, the prompt template is preliminarily optimized through a small amount of data with initial labels, and then the BERT model containing power expertise is used to apply the existing prompt template for prediction, and then the predicted labels are statistically analyzed to complete the label expansion. Finally, the results of the multi-classification model are closer to the upstream pre-training of the model, which improves the model performance of the prompt template.

[0136] It can be understood that the proportions of selecting short text sets from the data short text set as the prompt template training set, the label expansion training set and the template test set can be set according to needs. This embodiment only provides a specific implementation method and is not intended to constitute a limitation on this solution.

[0137] Through the method for constructing a multi-classification model for industrialized data provided by the present invention, a model is trained and generated based on BERT on unlabeled special structure metadata, the parameters of the continuous prompt template are optimized using a gradient descent algorithm, and the initial label is expanded using the trained prompt template to obtain a more refined new label, and the prompt template is trained again using the new label. The problem of fewer training sets is solved, the convergence speed of the model can be greatly accelerated, the problem that traditional models are difficult to migrate between different industrialized data is solved, and the generalization of the model is increased. In addition, it can effectively improve the data management of manufacturing enterprises, realize differentiated security protection management of enterprise data, and promote the orderly sharing, circulation, development and utilization of enterprise data to release the maximum value.

[0138] like Figure 2 As shown, it is a specific example flow chart of the method for constructing the industrial data multi-classification model provided by the present invention, and this example processes power data.

[0139] In step 1, the dedicated structural metadata of the power data is obtained, and various initial labels of the dedicated structural metadata are obtained; in step 2, pre-training is continued on the unlabeled dedicated structural metadata based on BERT to obtain the context embedding vector representation of each word, and the pre-trained language model of the dedicated structural metadata is obtained; in step 3, a continuous prompt template is constructed based on p-tunningV2 on the pre-trained language model through the dedicated structural metadata with initial labels, and then a multi-classification model of the power data is obtained according to the constructed continuous prompt template; in steps 4 to 6, the constructed continuous prompt template is optimized twice by introducing external knowledge, that is, the short text of the power data as a data set, and then the optimized multi-classification model of the power data is obtained according to the optimized continuous prompt template, so that the performance of the multi-classification model of the power data is optimized.

[0140] like Figure 3As shown, it is a flowchart of the method for constructing a multi-classification model for power data.

[0141] The construction of the multi-classification model of power data is mainly divided into three parts. The first part is to use the masked language task (MLM) to pre-train the language representation model on the power-specific structure metadata without initial labels; to solve the semantic gap problem between the pre-trained language model in the original data set and the power-specific data set of this task. The second part is to construct a continuous prompt template to obtain a multi-classification model of power data; its role is to have a more adequate adjustment method when converting the classification task into a masked language task (MLM) that adapts to the pre-trained language model. The third part is to introduce external knowledge, that is, short texts of power data as a data set to expand label options and obtain an optimized multi-classification model of power data; the purpose is to increase the classification accuracy through label expansion when the predicted answer of the continuous prompt template is different from the label of the text classification task, and then optimize the multi-classification model of power data.

[0142] Embodiment 2:

[0143] Based on the same inventive concept, the present invention also provides a system 400 for constructing an industrial data multi-classification model. The system structure is as follows: Figure 4 As shown, including:

[0144] A dedicated structure metadata acquisition module 401, an initial label acquisition module 402, a pre-trained language model acquisition module 403 and a multi-classification model acquisition module 404;

[0145] A dedicated structure metadata acquisition module 401 is used to acquire dedicated structure metadata of industrialized data;

[0146] An initial label acquisition module 402 is used to classify and label the dedicated structure metadata to obtain an initial label of the dedicated structure metadata;

[0147] A pre-trained language model acquisition module 403 is used to pre-train the language representation model based on the dedicated structural metadata to obtain a pre-trained language model of the dedicated structural metadata;

[0148] The multi-classification model acquisition module 404 is used to construct a continuous prompt template on the pre-trained language model through the dedicated structure metadata containing the initial label, and obtain the industrialized data multi-classification model according to the continuous prompt template.

[0149] Preferably, the system further comprises: a multi-classification model optimization module;

[0150] Multi-classification model optimization module, including:

[0151] The training set selection submodule is used to select a data short text set as a training set;

[0152] The industrial data multi-classification model optimization submodule is used to optimize the continuous prompt template through the training set to obtain the optimized continuous prompt template, and obtain the optimized industrial data multi-classification model according to the optimized continuous prompt template.

[0153] Preferably, the training set selection submodule is specifically used for:

[0154] A random sampling method is used to select some data short texts from the data short text set as the prompt template training set, and a specified proportion of data short texts are extracted from the remaining data short texts in the data short text set as the label expansion training set, and the remaining data short texts are used as the template test set.

[0155] Preferably, the industrial data multi-classification model optimization submodule includes:

[0156] An industrial data multi-classification model initial optimization unit is used to optimize the parameters of the continuous prompt template through the prompt template training set to obtain the initially optimized continuous prompt template, and obtain the initially optimized industrial data multi-classification model according to the initially optimized continuous prompt template;

[0157] The industrial data multi-classification model re-optimization unit is used to expand the initial label through the label expansion training set according to the initial optimized continuous prompt template, so as to obtain the extended label of the special structure metadata; train the initially optimized industrial data multi-classification model through the template test set according to the extended label, adjust the initially optimized continuous prompt template, and obtain the re-optimized industrial data multi-classification model according to the adjusted continuous prompt template.

[0158] Preferably, the dedicated structure metadata acquisition module is specifically used for:

[0159] Obtain data characteristics of industrialized data;

[0160] Common features of the data features are extracted, and dedicated structured metadata of the industrialized data is constructed according to the common features.

[0161] Preferably, the initial tag acquisition module includes:

[0162] A data feature acquisition submodule is used to classify and mark the dedicated structure metadata and obtain data features of the dedicated structure metadata;

[0163] An initial label mapping submodule, used to map the dedicated structure metadata to the corresponding initial label in combination with the data features of the dedicated structure metadata;

[0164] The types of initial labels include one or more of the following: regular identifiable labels, text identifiable labels, enumeration labels, numerical quantity labels, encoding labels, and remark description labels.

[0165] Preferably, the pre-trained language model acquisition module includes:

[0166] Multiple short text acquisition submodules are used to segment the dedicated structure metadata at the granularity of tokens to obtain multiple short texts of the dedicated structure metadata;

[0167] A character context representation acquisition submodule is used to connect multiple short texts, input the connected multiple short texts into the language representation model, and obtain the context representation of each character in the short text;

[0168] The initialization vector acquisition submodule is used to understand the contextual representation of each character in the short text through the multi-head attention mechanism of the transformer layer in the language representation model, output the embedding vector of each character, and obtain the contextual embedding initialization vector of each character;

[0169] The pre-trained language model acquisition submodule is used to pre-train the language representation model according to the context embedding initialization vector of each character to obtain a pre-trained language model of dedicated structural metadata.

[0170] Preferably, the multi-classification model acquisition module includes:

[0171] The continuous prompting submodule is used to continuously prompt each transformation layer of the pre-trained language model;

[0172] The continuous cue parameter acquisition submodule is used to fine-tune P-Tunning based on the cue, adding the continuous cue in each transformation layer as a prefix tag to the input sequence to obtain the continuous cue parameters for a specific task;

[0173] The continuous hint template construction submodule is used to build a continuous hint template on the pre-trained language model using an automatic gradient descent algorithm through dedicated structure metadata containing initial labels and task-specific continuous hint parameters.

[0174] Embodiment 3:

[0175] Based on the same inventive concept, the present invention also provides an industrial data multi-classification method, comprising:

[0176] Obtain industrial data to be processed;

[0177] The industrial data multi-classification model as described above is used to perform multi-classification processing on the industrial data to be processed.

[0178] Embodiment 4:

[0179] Based on the same inventive concept, the present invention also provides an industrial data multi-classification system, comprising:

[0180] A module for acquiring data to be processed, used for acquiring industrial data to be processed;

[0181] The multi-classification processing module is used to perform multi-classification processing on the industrial data to be processed using the system of the multi-classification model of industrial data constructed as described above.

[0182] Embodiment 5:

[0183] Based on the same inventive concept, Figure 5 As shown, the present invention also provides an electronic device, which may be a computer device, a single-chip device, an intelligent mobile device, etc. The electronic device in this embodiment may include a processor, a memory, a transceiver component, etc. The memory, the processor, and the transceiver component are connected via a bus; the memory may be used to store an execution program, and an exemplary execution program may include instructions; the processor is used to execute the instructions stored in the memory. The memory may also be used to store data, which may be called and / or modified when the instructions are executed.

[0184] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in a readable storage medium to implement the corresponding method flow or corresponding functions, so as to implement the steps of the method for constructing a multi-classification model of industrial data in the above-mentioned embodiment, or the steps of the method for implementing multi-classification of industrial data.

[0185] Embodiment 6:

[0186] Based on the same inventive concept, the present invention also provides a readable storage medium, specifically an electronic device readable storage medium (Memory), the electronic device readable storage medium is a memory device in the electronic device, which is used to store programs and data. It can be understood that the readable storage medium here can include both the built-in storage medium in the electronic device and the extended storage medium supported by the electronic device. The storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more execution programs (including program codes). It should be noted that the storage medium here can be a high-speed RAM memory or a non-volatile memory (non-volatile memory), such as at least one disk storage. The processor can load and execute one or more instructions stored in the storage medium to implement the steps of the method for constructing a multi-classification model of industrial data in the above embodiment, or to implement the method for multi-classification of industrial data.

[0187] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0188] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0189] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0190] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0191] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit its protection scope. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that after reading the present invention, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the application, but these changes, modifications or equivalent substitutions are all within the protection scope of the claims to be approved.

Claims

1. A method for constructing a multi-classification model for industrial data, characterized in that: include: Get specialized structured metadata for industrialized data; Classifying and labeling the dedicated structure metadata to obtain an initial label of the dedicated structure metadata; Pre-training a language representation model based on the dedicated structural metadata to obtain a pre-trained language model of the dedicated structural metadata; A continuous prompt template is constructed on the pre-trained language model through dedicated structural metadata including the initial label, and an industrialized data multi-classification model is obtained according to the continuous prompt template.

2. The method according to claim 1, characterized in that After obtaining the industrial data multi-classification model according to the continuous prompt template, the method further includes: Select the data short text set as the training set; The continuous prompt template is optimized through the training set to obtain an optimized continuous prompt template, and an optimized industrial data multi-classification model is obtained according to the optimized continuous prompt template.

3. The method according to claim 2, characterized in that The short text set of data is selected as a training set, including: A random sampling method is used to select part of the data short texts in the data short text set as the prompt template training set, and a rated proportion of the data short texts are extracted from the remaining data short texts in the data short text set as the label expansion training set, and the remaining data short texts are used as the template test set.

4. The method according to claim 3, characterized in that The step of optimizing the continuous prompt template by using the training set to obtain an optimized continuous prompt template, and obtaining an optimized industrial data multi-classification model according to the optimized continuous prompt template includes: Optimizing the parameters of the continuous prompt template through the prompt template training set to obtain the initially optimized continuous prompt template, and obtaining the initially optimized industrial data multi-classification model according to the initially optimized continuous prompt template; According to the initially optimized continuous prompt template, the initial label is expanded through the label expansion training set to obtain the expanded label of the special structure metadata; according to the expanded label, the initially optimized industrial data multi-classification model is trained through the template test set, the initially optimized continuous prompt template is adjusted, and the re-optimized industrial data multi-classification model is obtained according to the adjusted continuous prompt template.

5. The method according to claim 1, characterized in that The dedicated structure metadata for obtaining industrialized data includes: Obtain data characteristics of industrialized data; Common features of the data features are extracted, and dedicated structured metadata of the industrialized data is constructed according to the common features.

6. The method according to claim 1, characterized in that The classifying and labeling the dedicated structure metadata to obtain an initial label of the dedicated structure metadata includes: Classifying and marking the dedicated structure metadata to obtain data features of the dedicated structure metadata; Mapping the dedicated structure metadata to a corresponding initial tag in combination with data features of the dedicated structure metadata; The types of the initial labels include one or more of the following: regular identifiable labels, text identifiable labels, enumeration labels, numerical quantity labels, coding labels, and remark description labels.

7. The method according to any one of claims 1 to 4, characterized in that: The pre-training of the language representation model based on the dedicated structural metadata to obtain the pre-trained language model of the dedicated structural metadata includes: Segmenting the dedicated structure metadata at a granularity of tokens to obtain multiple short texts of the dedicated structure metadata; Connecting the multiple short texts, and inputting the connected multiple short texts into a language representation model to obtain a context representation of each character in the short text; The context representation of each character in the short text is understood through the multi-head attention mechanism of the transformer layer in the language representation model, and the embedding vector of each character is output to obtain the context embedding initialization vector of each character; The language representation model is pre-trained according to the context embedding initialization vector of each character to obtain a pre-trained language model of the dedicated structure metadata.

8. The method according to claim 7, characterized in that The step of constructing a continuous prompt template on the pre-trained language model by including the dedicated structure metadata of the initial label comprises: Continuously prompting each transformation layer of the pre-trained language model; P-Tunning based on prompt fine-tuning adds the continuous prompts in each transformation layer as prefix tags to the input sequence to obtain continuous prompt parameters for a specific task; A continuous prompt template is constructed on the pre-trained language model using an automatic gradient descent algorithm by including dedicated structural metadata of the initial label and continuous prompt parameters of the specific task.

9. A system for constructing a multi-classification model of industrial data, characterized in that: include: Dedicated structural metadata acquisition module, initial label acquisition module, pre-trained language model acquisition module and multi-classification model acquisition module; The dedicated structure metadata acquisition module is used to acquire the dedicated structure metadata of industrialized data; The initial label acquisition module is used to classify and label the dedicated structure metadata to obtain the initial label of the dedicated structure metadata; The pre-trained language model acquisition module is used to pre-train the language representation model based on the dedicated structural metadata to obtain the pre-trained language model of the dedicated structural metadata; The multi-classification model acquisition module is used to construct a continuous prompt template on the pre-trained language model through the dedicated structure metadata containing the initial label, and obtain the industrialized data multi-classification model according to the continuous prompt template.

10. The system according to claim 9, characterized in that The system also includes: a multi-classification model optimization module; The multi-classification model optimization module comprises: The training set selection submodule is used to select a data short text set as a training set; The industrial data multi-classification model optimization submodule is used to optimize the continuous prompt template through the training set to obtain an optimized continuous prompt template, and obtain an optimized industrial data multi-classification model according to the optimized continuous prompt template.

11. The system according to claim 10, characterized in that The training set selection submodule is specifically used for: A random sampling method is used to select part of the data short texts in the data short text set as the prompt template training set, and a rated proportion of the data short texts are extracted from the remaining data short texts in the data short text set as the label expansion training set, and the remaining data short texts are used as the template test set.

12. The system according to claim 11, characterized in that The industrial data multi-classification model optimization submodule includes: An industrial data multi-classification model initial optimization unit, used for optimizing the parameters of the continuous prompt template through the prompt template training set to obtain the continuous prompt template after initial optimization, and obtaining the industrial data multi-classification model after initial optimization according to the continuous prompt template after initial optimization; The industrial data multi-classification model re-optimization unit is used to expand the initial label through the label expansion training set according to the continuous prompt template after the initial optimization, so as to obtain the extended label of the special structure metadata; train the industrial data multi-classification model after the initial optimization through the template test set according to the extended label, adjust the continuous prompt template after the initial optimization, and obtain the re-optimized industrial data multi-classification model according to the adjusted continuous prompt template.

13. The system according to claim 9, characterized in that The dedicated structure metadata acquisition module is specifically used for: Obtain data characteristics of industrialized data; Common features of the data features are extracted, and dedicated structured metadata of the industrialized data is constructed according to the common features.

14. The system according to claim 9, characterized in that The initial label acquisition module includes: A data feature acquisition submodule, used to classify and mark the dedicated structure metadata to obtain data features of the dedicated structure metadata; An initial label mapping submodule, used to map the dedicated structure metadata to a corresponding initial label in combination with data features of the dedicated structure metadata; The types of the initial labels include one or more of the following: regular identifiable labels, text identifiable labels, enumeration labels, numerical quantity labels, coding labels, and remark description labels.

15. The system according to any one of claims 9 to 12, characterized in that: The pre-trained language model acquisition module includes: A plurality of short text obtaining submodules, used for segmenting the dedicated structure metadata at the granularity of tokens to obtain a plurality of short texts of the dedicated structure metadata; A character context representation obtaining submodule, used for connecting the multiple short texts, inputting the connected multiple short texts into a language representation model, and obtaining a context representation of each character in the short text; An initialization vector acquisition submodule is used to understand the context representation of each character in the short text through the multi-head attention mechanism of the transformer layer in the language representation model, output the embedding vector of each character, and obtain the context embedding initialization vector of each character; The pre-trained language model acquisition submodule is used to pre-train the language representation model according to the context embedding initialization vector of each character to obtain the pre-trained language model of the dedicated structure metadata.

16. The system according to claim 15, characterized in that The multi-classification model acquisition module includes: A continuous prompting submodule, used for continuously prompting each transformation layer of the pre-trained language model; A continuous hint parameter acquisition submodule, for adding the continuous hint in each transformation layer as a prefix tag to the input sequence based on the hint fine-tuning P-Tunning, to obtain the continuous hint parameter of the specific task; The continuous prompt template construction submodule is used to construct a continuous prompt template on the pre-trained language model by using an automatic gradient descent algorithm by including the dedicated structural metadata of the initial label and the continuous prompt parameters of the specific task.

17. A method for multi-classification of industrial data, characterized in that: include: Obtain industrial data to be processed; The industrial data to be processed is subjected to multi-classification processing using the multi-classification model for industrial data described in any one of claims 1 to 8.

18. A system for multi-classification of industrial data, characterized in that: include: A module for acquiring data to be processed, used for acquiring industrial data to be processed; A multi-classification processing module is used to perform multi-classification processing on the industrial data to be processed using the multi-classification model for industrial data described in any one of claims 1 to 8.

19. A computer device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the method for constructing an industrial data multi-classification model as described in any one of claims 1 to 8 is implemented, or the method for industrial data multi-classification as described in claim 17 is implemented.

20. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed, it implements the method for constructing an industrial data multi-classification model as described in any one of claims 1 to 8, or implements the method for industrial data multi-classification as described in claim 17.

Images