Language model pre-training method and device for electromechanical industry

Abstract

The invention discloses a language model pre-training method oriented to the electromechanical industry. The method comprises the following steps: collecting various data of the electromechanical industry; the method comprises the following steps: firstly, preprocessing various data of the electromechanical industry, then carrying out full-word shielding training by using a power industry text dictionary to obtain a source corpus, and then carrying out word segmentation operation and part-of-speech tagging on the source corpus to obtain a pre-training corpus oriented to the electromechanical industry; inputting the electromechanical industry-oriented pre-training corpus into a pre-training model for training to obtain an electromechanical industry-oriented pre-training language model; wherein when part-of-speech tagging is carried out on the source corpus, the source corpus is divided into a plurality of categories, and the plurality of categories comprise a file management category, an enterprise management category, a law category and a relevant knowledge category of an electromechanical construction technology; the pre-training language model is output in the form of a plurality of categories. By using the AI assistant provided by the invention, the efficiency and quality of electromechanical construction can be greatly improved, manual errors are reduced, and the construction cost is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of company-level intelligent services in the electromechanical industry, and relates to a language model pre-training method and device for the electromechanical industry. Background Art

[0002] In the company's system management and technical management, there are many complex systems, management methods, technical solutions, instructions for use, etc., which are cumbersome to query. Artificial intelligence can easily solve such problems and improve collaborative efficiency.

[0003] Although electromechanical AI language model products have been used to some extent in the market, the language models developed by different companies are different and still have some shortcomings, including but not limited to:

[0004] 1. Some small electromechanical companies have difficulty in obtaining AI language model data: For the electromechanical industry, obtaining and processing relevant data is an important challenge.

[0005] Since data comes from a wide range of sources and in various formats, as well as having a large amount of noise and redundant data, a lot of manpower and material resources are required to build enterprise-level AI language models.

[0006] 2. Insufficient interpretability and credibility: The interpretability and credibility of the output results of electromechanical AI language model products remain a key issue for users.

[0007] Some models may simply give a prediction result without providing sufficient explanation and basis, which makes it difficult for users to understand and trust the model's output.

[0008] 3. Out of touch with actual application scenarios: Some electromechanical AI language model products may not fully consider the needs and limitations of actual application scenarios during design and development, resulting in poor results in actual applications.

[0009] This may be due to insufficient communication between developers and users, or a lack of in-depth understanding of the actual application scenarios.

[0010] In view of the above shortcomings, the present invention is therefore produced. Summary of the invention

[0011] An object of the present invention is to solve at least the above problems and / or disadvantages and to provide at least the advantages which will be described hereinafter.

[0012] Another object of the present invention is to provide a language model pre-training method and device for the electromechanical industry. The present invention can automatically optimize the electromechanical construction process by training a large amount of electromechanical construction data.

[0013] To this end, the technical solution provided by the present invention is:

[0014] The language model pre-training method for the electromechanical industry includes the following steps:

[0015] Collect various data of the electromechanical industry;

[0016] First, various data of the electromechanical industry are preprocessed and then full-word masking training is performed using the electric power industry text dictionary to obtain source corpus, and then word segmentation and part-of-speech tagging are performed on the source corpus to obtain pre-training corpus for the electromechanical industry;

[0017] Inputting the pre-training corpus for the electromechanical industry into a pre-training model for training to obtain a pre-training language model for the electromechanical industry;

[0018] When part-of-speech tagging is performed on the source corpus, the source corpus is divided into several categories, including document management, enterprise management, law, and related knowledge of electromechanical construction technology;

[0019] The pre-trained language model is output in the form of several categories.

[0020] Preferably, in the language model pre-training method for the electromechanical industry, the various data of the electromechanical industry include company policies, employee handbooks, labor contracts, meeting minutes, relevant knowledge of electromechanical construction technology and management forms of various data.

[0021] Preferably, in the language model pre-training method for the electromechanical industry, the pre-processing includes data cleaning and data enhancement of various data of the electromechanical industry, and then full-word masking training is performed using a text dictionary of the electric power industry.

[0022] Preferably, in the language model pre-training method for the electromechanical industry, the pre-training model adopts a neural network algorithm or a decision tree algorithm or a BERT-BiLSTM-CRF model.

[0023] Preferably, in the language model pre-training method for the electromechanical industry, when the pre-training model is a BERT-BiLSTM-CRF model,

[0024] Performing entity extraction on the source corpus to obtain a time entity of the source corpus;

[0025] The source corpus is also annotated with causal relationships.

[0026] The source corpus with causal relationships annotated is input into the BERT-BiLSTM-CRF model to extract causal relationships of various data for the electromechanical industry;

[0027] Among them, when the source corpus outputs a word vector from the BERT layer, the word vector containing the time relationship is obtained by concatenating the word vector with the encoding vector of the time entity of the source corpus, and then the word vector containing the time relationship is further input into the BiLSTM layer for processing.

[0028] Preferably, in the language model pre-training method for the electromechanical industry, the causal relationship annotation adopts BIO annotation rules.

[0029] Language model pre-training device for the electromechanical industry, including:

[0030] Data collection module, which is used to collect various data in the electromechanical industry;

[0031] A data processing module is connected to the data collection module. First, the data processing module is used to pre-process various data of the electromechanical industry and then use the electric power industry text dictionary to perform full-word masking training to obtain source corpus. The data processing module is then used to perform word segmentation and part-of-speech tagging on the source corpus to obtain pre-training corpus for the electromechanical industry.

[0032] A pre-training model module, which is used to train the input pre-training corpus for the electromechanical industry to obtain a pre-training language model for the electromechanical industry;

[0033] The data processing module is used to classify the source corpus into several categories when performing part-of-speech tagging on the source corpus, and the several categories include document management, enterprise management, law, and related knowledge of electromechanical construction technology;

[0034] The pre-trained language model is output in the form of several categories.

[0035] Preferably, in the language model pre-training device for the electromechanical industry, the various data of the electromechanical industry include company policies, employee handbooks, labor contracts, meeting minutes, relevant knowledge of electromechanical construction technology and management forms of various data.

[0036] Preferably, in the language model pre-training device for the electromechanical industry, the pre-processing of the data processing module includes data cleaning and data enhancement of various data in the electromechanical industry, and then full-word masking training is performed using the electric power industry text dictionary.

[0037] Preferably, in the language model pre-training device for the electromechanical industry, the pre-training model module adopts a neural network algorithm or a decision tree algorithm or a BERT-BiLSTM-CRF model.

[0038] Preferably, in the language model pre-training device for the electromechanical industry, when the pre-training model module is a BERT-BiLSTM-CRF model,

[0039] The data processing module is also used to extract entities from the source corpus to obtain time entities of the source corpus;

[0040] The data processing module is also used to annotate the source corpus with causal relationships.

[0041] The source corpus with annotated causal relationships is input into the BERT-BiLSTM-CRF model to extract causal relationships of various data for the electromechanical industry.

[0042] Among them, when the source corpus outputs a word vector from the BERT layer, the word vector containing the time relationship is obtained by concatenating the word vector with the encoding vector of the time entity of the source corpus, and then the word vector containing the time relationship is further input into the BiLSTM layer for processing.

[0043] Preferably, in the language model pre-training device for the electromechanical industry, the causal relationship annotation adopts BIO annotation rules.

[0044] The present invention has at least the following beneficial effects:

[0045] The present invention strengthens data acquisition and processing technology and improves the generalization ability of the model by preprocessing, segmenting and annotating data; at the same time, the source corpus is divided into several categories, including file management, enterprise management, legal and related knowledge of electromechanical construction technology, etc., to ensure that the model can better adapt to the needs of actual application scenarios; and the data processing module is also used to annotate the source corpus with causal relationships, and input the source corpus annotated with causal relationships into the BERT-BiLSTM-CRF model to extract causal relationships of various data for the electromechanical industry, embodying the causal relationship between files in the model, increasing the interpretability and credibility of the model, and improving the user's trust in the model. At the same time, the present invention superimposes the time relationship in the training of the model, and the pre-trained language model for the electromechanical industry obtained by the method of the present invention combines the time factor, embodies the causal relationship, and improves the accuracy and reliability of the model. By using the AI ​​assistant of the present invention, the efficiency and quality of electromechanical construction can be greatly improved, human errors can be reduced, and construction costs can be reduced.

[0046] Other advantages, objectives and features of the present invention will be embodied in part through the following description, and in part will be understood by those skilled in the art through study and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1The figure is a flow chart of a language model pre-training method for the electromechanical industry in one of the solutions of the present invention. DETAILED DESCRIPTION

[0048] The present invention is further described in detail below in conjunction with the accompanying drawings so that those skilled in the art can implement the invention with reference to the description.

[0049] It should be understood that terms such as “having”, “including” and “comprising” used herein do not specify the existence or addition of one or more other elements or combinations thereof.

[0050] like Figure 1 As shown, the present invention provides a language model pre-training method for the electromechanical industry, comprising:

[0051] Collect various data in the electromechanical industry, including company-related systems, management methods, and various data in the electromechanical construction process;

[0052] First, various data of the electromechanical industry are preprocessed and then full-word masking training is performed using the electric power industry text dictionary to obtain source corpus, and then word segmentation and part-of-speech tagging are performed on the source corpus to obtain pre-training corpus for the electromechanical industry;

[0053] For example, in the full word masking training, the parameter settings are shown in Table 1:

[0054] Table 1. Parameter settings for full-word masking training

[0055]

[0056] This can guide iterative optimization of the text to ensure accurate understanding and coverage of core knowledge points and common application scenarios in the electromechanical industry.

[0057] The pre-trained corpus for the electromechanical industry is input into the pre-trained model for training to obtain a pre-trained language model for the electromechanical industry; in the present invention, the pre-trained model is referred to as the "mechanical small electrical" AI model.

[0058] When part-of-speech tagging is performed on the source corpus, the source corpus is divided into several categories, including document management, enterprise management, law, and related knowledge of electromechanical construction technology;

[0059] The pre-trained language model is output in the form of several categories.

[0060] In one of the solutions of the present invention, preferably, the various data of the electromechanical industry include company policies, employee handbooks, labor contracts, meeting minutes, relevant knowledge of electromechanical construction technology and management forms of various data.

[0061] Further, including:

[0062] 1. Collection and collation of rules and regulations and other documents:

[0063] ① Clarify the content of the document: First, determine the scope of the documents to be collected and organized, such as company policies, employee handbooks, labor contracts, meeting minutes, and other documents.

[0064] ② Form a team: Form a team responsible for collecting relevant text content.

[0065] This team should have knowledge of document management, law, business administration and mechanical and electrical construction technology so that they can accurately identify, classify and organize documents.

[0066] ③ Document review: Comprehensively review existing documents to ensure that all relevant documents have been collected.

[0067] This includes checking documents, electronic files and paper files in various departments.

[0068] At the same time, ensure the accuracy and completeness of the documents.

[0069] ④ File classification and organization: Classify files according to their nature and content.

[0070] For example, you can classify by department, file type, chronological order, etc.

[0071] During the organization process, ensure version control of files to avoid expired or invalid files.

[0072] ⑤ Establish a file management form: In order to better manage and maintain files, you can establish an electronic file management form.

[0073] This form should have functions such as search, version record, and permission marking to facilitate employees to access and use files.

[0074] ⑥ Regular update and maintenance: Rules and regulations and other documents may need to be modified and updated as the company develops and laws and regulations change.

[0075] Therefore, documents should be reviewed and updated regularly to ensure they are current and accurate.

[0076] At the same time, ensure that employees are aware of and comply with the new documents.

[0077] ⑦ Training and publicity: Train employees to let them understand the importance of company rules and regulations and other documents, and familiarize them with how to use AI.

[0078] In addition, through internal corporate publicity, employees' awareness of compliance with rules and regulations is improved.

[0079] ⑧Archive and backup: In order to prevent file loss or damage, files must be archived and backed up.

[0080] You can choose to store your files in the cloud or other secure storage media and regularly check the validity of archives and backups.

[0081] ⑨Continuous improvement: Regularly evaluate the content of relevant documents and collect employee feedback and suggestions to continuously improve and optimize the system's functionality and user experience.

[0082] ⑩ Comply with laws and regulations: In the process of collecting and organizing company rules and regulations and other documents, ensure compliance with relevant laws and regulations, such as labor laws, intellectual property laws, etc., to avoid legal disputes.

[0083] 2. Collection and collation of electromechanical construction related technologies:

[0084] ① Clarify the collection objectives: Before starting to collect mechanical and electrical construction technology, it is necessary to clarify the collection objectives and scope, including specific equipment, processes, engineering cases and other related technologies.

[0085] ② Consult professional literature: Consult relevant professional books, journals, papers and other literature to understand the latest mechanical and electrical construction technology and applications.

[0086] ③ Participate in industry conferences and exhibitions: Participate in industry conferences and exhibitions related to mechanical and electrical construction, communicate with peers, and learn about the latest technological developments and development trends.

[0087] ④ Establish technical files: classify and organize the collected electromechanical construction technologies, and establish technical files for easy reference and use.

[0088] ⑤ Field investigation and research: Conduct field investigation and research on the mechanical and electrical construction site to understand the technical applications and problems in actual construction and collect first-hand information.

[0089] ⑥Organize and summarize: Organize and summarize the collected electromechanical construction technologies, extract key technical points and application cases, and form systematic technical data.

[0090] ⑦ Update and maintenance: Regularly update and maintain technical information, and promptly add new technologies and processes to ensure the timeliness and accuracy of the information.

[0091] ⑧Protect intellectual property rights: When collecting and organizing electromechanical construction technologies, we must respect intellectual property rights and indicate the source and origin when quoting other people's technologies and achievements.

[0092] In one of the solutions of the present invention, preferably, the preprocessing includes data cleaning and data enhancement of various data in the electromechanical industry, and then full-word masking training is performed using a text dictionary of the electric power industry.

[0093] Furthermore, the collected relevant documents are referenced to train the "Jixiaodian" AI model.

[0094] ① Organize data: First, you need to organize the data set related to the AI ​​model to be trained.

[0095] These datasets can be structured data (such as tables, databases, etc.) or unstructured data (such as text, images, audio, etc.). These datasets should contain enough samples so that the model can learn and generalize to new data.

[0096] ②Data preprocessing: After collecting the data set, the data needs to be preprocessed.

[0097] This includes operations such as data cleaning, data labeling, and data enhancement.

[0098] Data cleaning can remove irrelevant data, duplicate data, etc. to improve data quality.

[0099] Data annotation is to add labels to the data so that the model can learn the characteristics and patterns of the data.

[0100] Data enhancement can increase data diversity and improve the generalization ability of the model by performing operations such as transformation, rotation, and scaling on the data.

[0101] In one of the schemes of the present invention, preferably, the pre-training model adopts a neural network algorithm or a decision tree algorithm or a BERT-BiLSTM-CRF model.

[0102] Choosing an appropriate model can improve training efficiency and model accuracy.

[0103] ④Training model: Input the preprocessed data set into the selected model for training.

[0104] During the training process, it is necessary to select a suitable loss function and optimizer to minimize the prediction error of the model.

[0105] Model evaluation and adjustment are also required during the training process to detect and solve problems in a timely manner.

[0106] For example, the loss function uses the cross entropy loss function, and the expression of the multi-classification problem is:

[0107]

[0108] Where M represents the number of categories; y ij Indicates whether the category is the same as the sample i category, if they are the same, it is 1, if they are different, it is 0; p ij It represents the predicted probability that the observed sample i belongs to category j.

[0109] In one of the solutions of the present invention, preferably, when the pre-trained model is a BERT-BiLSTM-CRF model,

[0110] Performing entity extraction on the source corpus to obtain a time entity of the source corpus;

[0111] The source corpus is also annotated with causal relationships.

[0112] The source corpus with causal relationships annotated is input into the BERT-BiLSTM-CRF model to extract causal relationships of various data for the electromechanical industry;

[0113] Among them, when the source corpus outputs a word vector from the BERT layer, the word vector containing the time relationship is obtained by concatenating the word vector with the encoding vector of the time entity of the source corpus, and then the word vector containing the time relationship is further input into the BiLSTM layer for processing.

[0114] In one of the solutions of the present invention, preferably, the causal relationship annotation adopts BIO annotation rules.

[0115] The language model pre-training device for the electromechanical industry comprises:

[0116] Data collection module, which is used to collect various data in the electromechanical industry;

[0117] A data processing module is connected to the data collection module. First, the data processing module is used to pre-process various data of the electromechanical industry and then use the electric power industry text dictionary to perform full-word masking training to obtain source corpus. The data processing module is then used to perform word segmentation and part-of-speech tagging on the source corpus to obtain pre-training corpus for the electromechanical industry.

[0118] A pre-training model module, which is used to train the input pre-training corpus for the electromechanical industry to obtain a pre-training language model for the electromechanical industry;

[0119] The data processing module is used to classify the source corpus into several categories when performing part-of-speech tagging on the source corpus, and the several categories include document management, enterprise management, law, and related knowledge of electromechanical construction technology;

[0120] The pre-trained language model is output in a form containing several categories.

[0121] In the above scheme, as a preference, it also includes:

[0122] An instruction output module is connected to the pre-training model module and is used to display the output results of the pre-training model module.

[0123] In the above scheme, as a preferred embodiment, the various data of the electromechanical industry include company policies, employee handbooks, labor contracts, meeting minutes, relevant knowledge of electromechanical construction technology and management forms of various data.

[0124] In the above scheme, preferably, the preprocessing of the data processing module includes data cleaning and data enhancement of various data in the electromechanical industry, and then full-word masking training is performed using the electric power industry text dictionary.

[0125] In the above scheme, preferably, the pre-training model module adopts a neural network algorithm or a decision tree algorithm or a BERT-BiLSTM-CRF model.

[0126] In the above scheme, preferably, when the pre-trained model module is a BERT-BiLSTM-CRF model,

[0127] The data processing module is also used to extract entities from the source corpus to obtain time entities of the source corpus;

[0128] The data processing module is also used to annotate the source corpus with causal relationships.

[0129] The source corpus text with causal relationships annotated is divided into training set, validation set and test set. The training set is first used to input the BERT-BiLSTM-CRF model to extract causal relationships of various data for the electromechanical industry. The validation set is then used to verify the training results. After verification, the model is tested using the test set.

[0130] When the source corpus outputs a word vector from the BERT layer, the word vector containing the time relationship is obtained by splicing the word vector with the encoding vector of the time entity of the source corpus, and then the word vector containing the time relationship is further input into the BiLSTM layer for processing. For example, the training environment configuration is shown in Table 2, and the parameter settings are shown in Table 3.

[0131] Table 2 Training environment configuration

[0132]

[0133] Table 3 Parameter settings

[0134]

[0135] The BERT-BiLSTM-CRF model with the above parameter settings has a strong ability to obtain dynamic word vectors, so that it can obtain better word vectors for training and prediction of the "Jixiaodian" model.

[0136] In the above solution, preferably, the causal relationship annotation adopts BIO annotation rules.

[0137] In order to enable those skilled in the art to better understand the technical solution of the present invention, the following embodiments are provided for illustration:

[0138] A language model pre-training method for the electromechanical industry, comprising the following steps:

[0139] Collect various data of the mechanical and electrical industry; the various data of the mechanical and electrical industry include company policies, employee handbooks, labor contracts, meeting minutes, relevant knowledge of mechanical and electrical construction technology and management forms of various data.

[0140] First, various data of the electromechanical industry are preprocessed and then full-word masking training is performed using the electric power industry text dictionary to obtain the source corpus. The preprocessing of the data processing module includes data cleaning and data enhancement of various data of the electromechanical industry, and then full-word masking training is performed using the electric power industry text dictionary. In the full-word masking training, the parameter settings are shown in Table 1:

[0141] Table 1. Parameter settings for full-word masking training

[0142]

[0143]

[0144] This can guide iterative optimization of the text to ensure accurate understanding and coverage of core knowledge points and common application scenarios in the electromechanical industry.

[0145] Then, the source corpus is segmented and POS is tagged, wherein when the source corpus is tagged, the source corpus is divided into several categories, including document management, enterprise management, law, and related knowledge of electromechanical construction technology;

[0146] Performing entity extraction on the source corpus to obtain a time entity of the source corpus;

[0147] The source corpus is also annotated with causal relationships as pre-training corpus for the electromechanical industry; the causal relationship annotation adopts the BIO annotation rule.

[0148] Input the pre-trained corpus for the electromechanical industry (the source corpus with causal relationships annotated) into the BERT-BiLSTM-CRF model to extract and train causal relationships of various data for the electromechanical industry, so as to obtain a pre-trained language model for the electromechanical industry;

[0149] Among them, when the source corpus outputs word vectors from the BERT layer, the word vectors are concatenated with the encoding vectors of the time entities in the source corpus to obtain word vectors containing time relationships, and then the word vectors containing time relationships are continuously input into the BiLSTM layer for processing;

[0150] The BERT-BiLSTM-CRF model outputs in several categories.

[0151] The training environment configuration of the BERT-BiLSTM-CRF model is shown in Table 2, and the parameter settings are shown in Table 3.

[0152] Table 2 Training Environment Configuration

[0153] Training Environment Configuration parameters operating system Windows 11 programming language Python 3.6.9 CPU Intel(R)Core(TM)i7-10750H CPU@2.60GHz GPU NVIDIAGeForce RTX3090

[0154] Table 3 Parameter Settings

[0155] parameter Setting Value Transformer Layers 15 Hidden layer dimensions 768 Optimizer Adam Learning rate 0.0001 LSTM dim 200 Batch size 128 Max sequence length 256 Dropout 0.5 Clip 5.0

[0156] The BERT-BiLSTM-CRF model with the above parameter settings has a strong ability to obtain dynamic word vectors. The model of this embodiment can learn the features of word level, syntactic structure and semantic information of context, improve the effect of Chinese entity recognition, and the "Jixiaodian" model obtained by training has good accuracy and reliability.

[0157] Exemplarily, the following operations are performed:

[0158] 1. Collection and collation of documents such as rules and regulations:

[0159] ① Clarify the content of the documents: First of all, it is necessary to determine the scope of the documents to be collected and collated, such as company policies, employee manuals, labor contracts, meeting minutes and other documents.

[0160] ② Form a team: Form a team responsible for collecting relevant text content.

[0161] This team should have relevant knowledge of document management, law, enterprise management and mechanical and electrical construction technology, so as to be able to accurately identify, classify and collate documents.

[0162] ③ Document review: Comprehensively review the existing documents to ensure that all relevant documents have been collected completely.

[0163] This includes checking the documents, electronic files and paper files of each department.

[0164] At the same time, it is necessary to ensure the accuracy and integrity of the documents.

[0165] ④ Document classification and collation: Classify the documents according to their nature and content.

[0166] For example, you can classify by department, file type, chronological order, etc.

[0167] During the organization process, ensure version control of files to avoid expired or invalid files.

[0168] ⑤ Establish a file management form: In order to better manage and maintain files, you can establish an electronic file management form.

[0169] This form should have functions such as search, version record, and permission marking to facilitate employees to access and use files.

[0170] ⑥ Regular update and maintenance: Rules and regulations and other documents may need to be modified and updated as the company develops and laws and regulations change.

[0171] Therefore, documents should be reviewed and updated regularly to ensure they are current and accurate.

[0172] At the same time, ensure that employees are aware of and comply with the new documents.

[0173] ⑦ Training and publicity: Train employees to let them understand the importance of company rules and regulations and other documents, and familiarize them with how to use AI.

[0174] In addition, through internal corporate publicity, employees' awareness of compliance with rules and regulations is improved.

[0175] ⑧Archive and backup: In order to prevent file loss or damage, files must be archived and backed up.

[0176] You can choose to store your files in the cloud or other secure storage media and regularly check the validity of archives and backups.

[0177] ⑨Continuous improvement: Regularly evaluate the content of relevant documents and collect employee feedback and suggestions to continuously improve and optimize the system's functionality and user experience.

[0178] ⑩ Comply with laws and regulations: In the process of collecting and organizing company rules and regulations and other documents, ensure compliance with relevant laws and regulations, such as labor laws, intellectual property laws, etc., to avoid legal disputes.

[0179] 2. Collection and collation of electromechanical construction related technologies:

[0180] ① Clarify the collection objectives: Before starting to collect mechanical and electrical construction technology, it is necessary to clarify the collection objectives and scope, including specific equipment, processes, engineering cases and other related technologies.

[0181] ② Consult professional literature: Consult relevant professional books, journals, papers and other literature to understand the latest mechanical and electrical construction technology and applications.

[0182] ③ Participate in industry conferences and exhibitions: Participate in industry conferences and exhibitions related to mechanical and electrical construction, communicate with peers, and learn about the latest technological developments and development trends.

[0183] ④ Establish technical files: classify and organize the collected electromechanical construction technologies, and establish technical files for easy reference and use.

[0184] ⑤ Field investigation and research: Conduct field investigation and research on the mechanical and electrical construction site to understand the technical applications and problems in actual construction and collect first-hand information.

[0185] ⑥Organize and summarize: Organize and summarize the collected electromechanical construction technologies, extract key technical points and application cases, and form systematic technical data.

[0186] ⑦ Update and maintenance: Regularly update and maintain technical information, and promptly add new technologies and processes to ensure the timeliness and accuracy of the information.

[0187] ⑧Protect intellectual property rights: When collecting and organizing electromechanical construction technologies, we must respect intellectual property rights and indicate the source and origin when quoting other people's technologies and achievements.

[0188] 3. Refer to the collected relevant documents to train the "Jixiaodian" AI model.

[0189] ① Organize data: First, you need to organize the data set related to the AI ​​model to be trained.

[0190] These datasets can be structured data (such as tables, databases, etc.) or unstructured data (such as text, images, audio, etc.). These datasets should contain enough samples so that the model can learn and generalize to new data.

[0191] ②Data preprocessing: After collecting the data set, the data needs to be preprocessed.

[0192] This includes operations such as data cleaning, data labeling, and data enhancement.

[0193] Data cleaning can remove irrelevant data, duplicate data, etc. to improve data quality.

[0194] Data annotation is to add labels to the data so that the model can learn the characteristics and patterns of the data.

[0195] Data enhancement can increase data diversity and improve the generalization ability of the model by performing operations such as transformation, rotation, and scaling on the data.

[0196] ③Select the model: Select the appropriate AI model based on the problem to be solved and the characteristics of the data set.

[0197] Common AI models include neural networks, decision trees, support vector machines, etc.

[0198] Choosing an appropriate model can improve training efficiency and model accuracy.

[0199] ④Training model: Input the preprocessed data set into the selected model for training.

[0200] During the training process, it is necessary to select a suitable loss function and optimizer to minimize the prediction error of the model.

[0201] Model evaluation and adjustment are also required during the training process to detect and solve problems in a timely manner.

[0202] The loss function uses the cross entropy loss function, and the expression of the multi-classification problem is:

[0203]

[0204] Where M represents the number of categories; y ij Indicates whether the category is the same as the sample i category, if they are the same, it is 1, if they are different, it is 0; p ij It represents the predicted probability that the observed sample i belongs to category j.

[0205] ⑤Model evaluation: After model training is completed, the model needs to be evaluated.

[0206] Evaluation indicators include accuracy, precision, recall, etc.

[0207] Evaluation can be used to quantify the performance of the model so that it can be optimized and improved.

[0208] ⑥Model optimization: Based on the evaluation results, the model can be optimized.

[0209] Optimization methods include adjusting model parameters, improving model structure, adding regularization terms, etc.

[0210] The performance and generalization ability of the model can be improved through optimization.

[0211] ⑦ Application model: Apply the trained model to practical problems for prediction and decision-making.

[0212] During the application process, the model also needs to be continuously monitored and maintained to ensure its accuracy and stability.

[0213] The present invention also provides an electronic device, comprising: at least one processor, and a memory connected to the at least one processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described. When the processor runs the computer program stored in the memory, it can perform the various operations and functions described in the embodiments of the present invention to ensure the speed and scale of data processing.

[0214] The present invention also provides a storage medium on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented. The specific implementation can be found in the method embodiment, which will not be described in detail here.

[0215] The present invention also achieves the following effects:

[0216] 1. Improve efficiency and productivity: Improve efficiency and productivity of other major tasks by reducing the learning and search of daily systems and management methods.

[0217] AI can automate many routine and repetitive question-answering tasks, freeing up employees’ time and energy so they can focus more on innovative and strategic work.

[0218] 2. Improve the efficiency of technical knowledge sharing: AI can build an intelligent question-and-answer system based on the various technical materials uploaded by company employees, instantly answer employees' questions about technical issues, and provide relevant explanations and examples.

[0219] 3. Competitive advantage and market leadership: Developing and using its own AI technology and establishing a language model that specializes in the electromechanical industry can enable the company to gain an advantage in the competition. By mastering advanced AI technology, the company can provide better products and services, attract more customers and partners, and consolidate its market leadership.

[0220] The number of modules and processing scale described here are used to simplify the description of the present invention. The application, modification and variation of the language model pre-training method for the electromechanical industry of the present invention are obvious to those skilled in the art.

[0221] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation modes, and they can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A language model pre-training method for the electromechanical industry, characterized in that: The steps include: Collect various data of the electromechanical industry; First, various data of the electromechanical industry are preprocessed and then full-word masking training is performed using the electric power industry text dictionary to obtain source corpus, and then word segmentation and part-of-speech tagging are performed on the source corpus to obtain pre-training corpus for the electromechanical industry; Inputting the pre-training corpus for the electromechanical industry into a pre-training model for training to obtain a pre-training language model for the electromechanical industry; When part-of-speech tagging is performed on the source corpus, the source corpus is divided into several categories, including document management, enterprise management, law, and related knowledge of electromechanical construction technology; The pre-trained language model is output in the form of several categories.

2. The language model pre-training method for the electromechanical industry according to claim 1, characterized in that: The various data of the electromechanical industry include company policies, employee handbooks, labor contracts, meeting minutes, relevant knowledge of electromechanical construction technology and management forms of various data.

3. The language model pre-training method for the electromechanical industry according to claim 1, characterized in that: The preprocessing includes data cleaning and data enhancement for various data in the electromechanical industry, and then full-word masking training is performed using a text dictionary in the electric power industry.

4. The language model pre-training method for the electromechanical industry according to claim 1, characterized in that: The pre-training model adopts a neural network algorithm or a decision tree algorithm or a BERT-BiLSTM-CRF model.

5. The language model pre-training method for the electromechanical industry as claimed in claim 4, characterized in that: When the pre-trained model is a BERT-BiLSTM-CRF model, Performing entity extraction on the source corpus to obtain a time entity of the source corpus; The source corpus is also annotated with causal relationships. The source corpus with causal relationships annotated is input into the BERT-BiLSTM-CRF model to extract causal relationships of various data for the electromechanical industry; Among them, when the source corpus outputs a word vector from the BERT layer, the word vector containing the time relationship is obtained by concatenating the word vector with the encoding vector of the time entity of the source corpus, and then the word vector containing the time relationship is further input into the BiLSTM layer for processing.

6. The language model pre-training method for the electromechanical industry as claimed in claim 5, characterized in that: The causal relationship annotation adopts BIO annotation rules.

7. A language model pre-training device for the electromechanical industry, characterized in that: include: Data collection module, which is used to collect various data in the electromechanical industry; A data processing module is connected to the data collection module. First, the data processing module is used to pre-process various data of the electromechanical industry and then use the electric power industry text dictionary to perform full-word masking training to obtain source corpus. The data processing module is then used to perform word segmentation and part-of-speech tagging on the source corpus to obtain pre-training corpus for the electromechanical industry. A pre-training model module, which is used to train the input pre-training corpus for the electromechanical industry to obtain a pre-training language model for the electromechanical industry; The data processing module is used to classify the source corpus into several categories when performing part-of-speech tagging on the source corpus, and the several categories include document management, enterprise management, law, and related knowledge of electromechanical construction technology; The pre-trained language model is output in the form of several categories it contains.

8. The language model pre-training device for the electromechanical industry according to claim 7, characterized in that: The various data of the electromechanical industry include company policies, employee handbooks, labor contracts, meeting minutes, relevant knowledge of electromechanical construction technology and management forms of various data.

9. The language model pre-training device for the electromechanical industry according to claim 7, characterized in that: The preprocessing of the data processing module includes data cleaning and data enhancement of various data in the electromechanical industry, and then full-word masking training is performed using the electric power industry text dictionary.

10. The language model pre-training device for the electromechanical industry according to claim 7, characterized in that: When the pre-trained model module is a BERT-BiLSTM-CRF model, The data processing module is also used to extract entities from the source corpus to obtain time entities of the source corpus; The data processing module is also used to annotate the source corpus with causal relationships. The source corpus with causal relationships annotated is input into the BERT-BiLSTM-CRF model to extract causal relationships of various data for the electromechanical industry; Among them, when the source corpus outputs a word vector from the BERT layer, the word vector containing the time relationship is obtained by concatenating the word vector with the encoding vector of the time entity of the source corpus, and then the word vector containing the time relationship is further input into the BiLSTM layer for processing.

Images