Construction method and device of coal mine vertical field large model

Abstract

The invention provides a construction method and device for a coal mine vertical field large model, and relates to the technical field of deep learning, and the method comprises the steps: obtaining field corpus data and general corpus data, and constructing a field word list and a general word list based on the field corpus data and the general corpus data; performing weighted fusion on the segmented words in the general word list and the field word list based on respective weights to obtain a fused word list; based on the fusion word list, utilizing domain corpus data and general corpus data to train the embedded model; loading a pre-trained original large language model, and replacing the embedded layer of the original large language model with the trained embedded model to obtain an updated large language model; and performing incremental pre-training on the updated large language model by using the domain corpus data and the general corpus data to obtain a coal mine vertical domain large model. The coal mine vertical field large model constructed by the method realizes accurate expression of professional knowledge of the coal mine industry.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a method and device for constructing a large vertical model of a coal mine. Background Art

[0002] Coal mines are open, complex, and giant systems. With the development of technologies such as mine digitization and smart mines, mine personnel use digital technology to monitor the mine environment and collect data, and quickly respond and make decisions based on the real-time environment on site, effectively improving the safety level of coal mine production. After the emergence of large language models, the qualitative change caused by quantitative changes has enabled large models to have "emergence capabilities" to handle complex problems that small models cannot solve.

[0003] Although the general large language model has a very strong universal ability, due to the uneven distribution of training data, overfitting of model parameters, label noise and other issues, the general large language model is insufficient in learning knowledge in the coal mining field and is prone to "hallucination". Therefore, the general large language model lacks professional data for the coal mining industry, resulting in the model's inability to accurately understand and generate relevant content when dealing with coal mining-specific terminology and workflows. Therefore, how to build a large model in the vertical field of coal mining is an issue that needs to be studied urgently. Summary of the invention

[0004] The present invention provides a method and device for constructing a large vertical field model of a coal mine, aiming to solve the problems existing in the above-mentioned background technology.

[0005] In order to solve the above-mentioned technical problems, the present invention is achieved as follows: In a first aspect, the present invention provides a method for constructing a large vertical model of a coal mine, the method comprising: Acquire domain corpus data and general corpus data, and construct a domain vocabulary and a general vocabulary based on the domain corpus data and the general corpus data respectively; Performing weighted fusion on each segmentation in the general vocabulary and the domain vocabulary based on their respective weights to obtain a fused vocabulary, wherein the weights represent the relative frequencies of the segmentation in the domain corpus data and the general corpus data; Based on the fusion vocabulary, an embedding model is trained using the domain corpus data and the general corpus data, wherein the embedding model is used to map word segmentation to corresponding word vectors based on semantic information; Loading the pre-trained original large language model, and replacing the trained embedding model with the embedding layer of the original large language model to obtain an updated large language model; The updated large language model is incrementally pre-trained using the domain corpus data and the general corpus data to obtain a large model for the coal mine vertical domain.

[0006] Optionally, the step of performing weighted fusion on each word in the general vocabulary and the domain vocabulary based on their respective weights to obtain a fused vocabulary includes: In the general vocabulary and the field vocabulary, the frequency and weight of each word segment are calculated; Based on the weights, each word segment in the general vocabulary and the domain vocabulary is linearly combined to obtain the fused vocabulary.

[0007] Optionally, the calculating of the frequency and weight of each word segment in the general vocabulary and the domain vocabulary includes: For each word segment in the general vocabulary and the domain vocabulary, determining the occurrence frequency of the word segment in the domain corpus data as a first word frequency, or determining the occurrence frequency of the word segment in the general corpus data as a second word frequency; For each word segment in the general vocabulary, calculate the relative frequency of the word segment in the general corpus data according to the first word frequency, and determine the relative frequency of the word segment in the general corpus data as a first linear item; For each word segment in the domain vocabulary, calculate the relative frequency of the word segment in the domain corpus data according to the second word frequency, and determine the relative frequency of the word segment in the general corpus data as a second linear item; For the same segmented words in the general vocabulary and the domain vocabulary, the logarithm of the ratio of the first word frequency to the second word frequency of the segmented words is determined as a logarithmic term; Preset weight adjustment parameters are respectively assigned to the first linear term, the second linear term and the logarithmic term, and the weight of each word segmentation is calculated.

[0008] Optionally, after obtaining the fused vocabulary, the method further includes: Checking whether there are any unmatched segmented words in the domain vocabulary, wherein the unmatched segmented words are segmented words that exist in the domain vocabulary but not in the general vocabulary; When the unmatched segmented word is detected, the segmented word is added to the fusion word list.

[0009] Optionally, the training of the embedding model based on the fused vocabulary using the domain corpus data and the general corpus data includes: Mixing the domain corpus data and the general corpus data according to a preset ratio to obtain training corpus data, wherein the general corpus data occupies a higher ratio than the domain corpus data; Based on the fusion vocabulary, the training corpus data is segmented to obtain a plurality of training segmented words; Based on the multiple training participles, the embedding model is trained according to preset training parameters, where the training parameters include batch size, learning rate, and loss function.

[0010] Optionally, the method further comprises: Obtaining a high-dimensional word vector output by the embedding model, and determining the core semantic information and secondary semantic information represented by the high-dimensional word vector; Performing dimensionality reduction processing on the high-dimensional word vector to obtain a corresponding low-dimensional word vector, wherein the core semantic information represented by the high-dimensional word vector is stored in the head dimension of the low-dimensional word vector, and the secondary semantic information represented by the high-dimensional word vector is stored in the tail dimension of the low-dimensional word vector; The low-dimensional word vector is stored in a data structure predefined by the embedding model.

[0011] Optionally, replacing the trained embedding model with the embedding layer of the original large language model to obtain an updated large language model includes: Replacing the embedding layer of the original large language model with the trained embedding model to obtain the embedded layer of the updated large language model; The embedding layer of the updated large language model is initialized using the word vectors output by the trained embedding model.

[0012] Optionally, the incremental pre-training of the updated large language model using the domain corpus data and the general corpus data to obtain a large model for the vertical domain of coal mines includes: Mixing the domain corpus data and the general corpus data according to a preset ratio to obtain training corpus data, wherein the general corpus data occupies a higher ratio than the domain corpus data; Adding a low-rank matrix and a bias matrix to the weight matrix of the linear layer of the updated large language model; Freeze the other layers of the updated large language model except the embedding layer, and train the embedding layer of the updated large language model using the training corpus data; When a preset iteration condition is met, unfreeze the other layers, and use the training corpus data to perform fine-tuning training on all layers of the updated large language model to obtain a low-rank adaptation model corresponding to the large language model; The low-rank adaptive model is merged with the original large language model to obtain the large model of the coal mine vertical field.

[0013] Optionally, the acquiring of domain corpus data and general corpus data, and constructing a domain vocabulary and a general vocabulary based on the domain corpus data and the general corpus data, respectively, includes: Organizing the pre-acquired coal mine vertical field data set into the field corpus data, and organizing the pre-acquired open source data set into the general corpus data, wherein the coal mine vertical field data set includes equipment monitoring data, mine geological exploration report data, forum data and industry website data; processing the general corpus data into the general vocabulary; The domain corpus data is processed into the domain vocabulary that is unified with the general vocabulary format.

[0014] In a second aspect, the present invention provides a device for constructing a large vertical model of a coal mine, the device comprising: An acquisition module, used for acquiring domain corpus data and general corpus data, and constructing a domain vocabulary and a general vocabulary based on the domain corpus data and the general corpus data respectively; A fusion module, used for weighted fusion of each segmentation in the general vocabulary and the domain vocabulary based on their respective weights to obtain a fused vocabulary, wherein the weights represent the relative frequencies of the segmentation in the domain corpus data and the general corpus data; A first training module is used to train an embedding model based on the fusion vocabulary using the domain corpus data and the general corpus data, wherein the embedding model is used to map word segmentation to corresponding word vectors based on semantic information; An updating module is used to load the pre-trained original large language model and replace the trained embedding model with the embedding layer of the original large language model to obtain an updated large language model; The second training module is used to perform incremental pre-training on the updated large language model using the domain corpus data and the general corpus data to obtain a large model for the coal mine vertical domain.

[0015] Optionally, the fusion module includes: A weight calculation submodule, used for calculating the frequency and weight of each word in the general vocabulary and the field vocabulary; The linear combination submodule is used to linearly combine the respective participles in the general vocabulary and the domain vocabulary based on the weights to obtain the fused vocabulary.

[0016] Optionally, the weight calculation submodule includes: A word frequency determination unit, configured to determine, for each word segment in the general word list and the domain word list, the frequency of occurrence of the word segment in the domain corpus data as a first word frequency, or to determine the frequency of occurrence of the word segment in the general corpus data as a second word frequency; A first relative frequency determination unit is configured to calculate, for each word segment in the general vocabulary, a relative frequency of the word segment in the general corpus data according to the first word frequency, and determine the relative frequency of the word segment in the general corpus data as a first linear item; A second relative frequency determination unit is used to calculate, for each word segment in the domain vocabulary, a relative frequency of the word segment in the domain corpus data according to the second word frequency, and determine the relative frequency of the word segment in the general corpus data as a second linear item; A logarithmic term determining unit, configured to determine, for the same segmented word in the general vocabulary and the domain vocabulary, the logarithm of the ratio of the first word frequency to the second word frequency of the segmented word as a logarithmic term; The linear combination unit is used to respectively assign preset weight adjustment parameters to the first linear term, the second linear term and the logarithmic term, and calculate the weight of each word segment.

[0017] Optionally, the device further comprises: A checking module, used to check whether there are unmatched segmented words in the domain vocabulary, wherein the unmatched segmented words are segmented words that exist in the domain vocabulary but not in the general vocabulary; The adding module is used to add the segmented word to the fusion word list when the unmatched segmented word is detected.

[0018] Optionally, the first training module includes: A first mixing submodule is used to mix the domain corpus data and the general corpus data according to a preset ratio to obtain training corpus data, wherein the general corpus data occupies a higher proportion than the domain corpus data; A word segmentation processing submodule, used for performing word segmentation processing on the training corpus data based on the fusion word list to obtain a plurality of training word segments; The first training submodule is used to train the embedding model based on the multiple training word segmentations according to preset training parameters, where the training parameters include batch size, learning rate and loss function.

[0019] Optionally, the device further comprises: A word vector acquisition module, used to acquire the high-dimensional word vector output by the embedding model, and determine the core semantic information and secondary semantic information represented by the high-dimensional word vector; A dimensionality reduction module, used to perform dimensionality reduction processing on the high-dimensional word vector to obtain a corresponding low-dimensional word vector, wherein the core semantic information represented by the high-dimensional word vector is stored in the head dimension of the low-dimensional word vector, and the secondary semantic information represented by the high-dimensional word vector is stored in the tail dimension of the low-dimensional word vector; The word vector saving module is used to save the low-dimensional word vector in a data structure predefined by the embedding model.

[0020] Optionally, the update module includes: A replacement submodule, used to replace the embedding layer of the original large language model with the trained embedding model to obtain the embedding layer of the updated large language model; The initialization submodule is used to initialize the embedding layer of the updated large language model using the word vector output by the trained embedding model.

[0021] Optionally, the second training module includes: A second mixing submodule is used to mix the domain corpus data and the general corpus data according to a preset ratio to obtain training corpus data, wherein the general corpus data occupies a higher ratio than the domain corpus data; A matrix adding submodule, used for adding a low-rank matrix and a bias matrix to the weight matrix of the linear layer of the updated large language model; A freezing submodule, used to freeze the other layers of the updated large language model except the embedding layer, and train the embedding layer of the updated large language model using the training corpus data; An unfreezing submodule, used for unfreezing the other layers when a preset iteration condition is met, and fine-tuning all layers of the updated large language model using the training corpus data to obtain a low-rank adaptation model corresponding to the large language model; The merging submodule is used to merge the low-rank adaptive model with the original large language model to obtain the large model of the coal mine vertical field.

[0022] Optionally, the acquisition module includes: A sorting submodule, used to sort the pre-acquired coal mine vertical field data set into the field corpus data, and to sort the pre-acquired open source data set into the general corpus data, wherein the coal mine vertical field data set includes equipment monitoring data, mine geological exploration report data, forum data and industry website data; A first processing submodule, configured to process the general corpus data into the general vocabulary; The second processing submodule is used to process the domain corpus data into the domain vocabulary in a format consistent with the general vocabulary.

[0023] The technical solution provided by the present invention brings at least the following beneficial effects: The present invention achieves deep learning and precise expression of professional knowledge of the coal mining industry by constructing a large model of the vertical field of coal mines and combining the domain corpus data with the general corpus data. By weighted fusion of the domain vocabulary and the general vocabulary, the model's ability to understand specific terms and workflows in the coal mining field can be effectively improved, avoiding the "hallucination" phenomenon that often occurs in general large language models when dealing with professional problems in coal mines. In addition, the embedded model can fully capture domain semantic information during the training process, thereby improving the performance accuracy of the model in tasks such as coal mine safety management, technical operations, and mine monitoring. Finally, the large model of the vertical field of coal mines obtained through incremental pre-training can provide more professional and accurate coal mine solutions while ensuring the versatility of the model, greatly improving coal mine management, operational efficiency, and safe production levels, and providing strong support for the intelligent and digital transformation of the coal mining industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 It is a schematic diagram of the steps of a method for constructing a large vertical model of a coal mine provided by an embodiment of the present invention; Figure 2 It is a structural block diagram of a device for constructing a large vertical field model of a coal mine provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] With the rapid development of digital and intelligent technologies, the coal mining industry is facing unprecedented opportunities and challenges. Traditional coal mine safety management and operation models often rely on manual experience and expertise. However, due to the complexity and diversity of coal mine safety issues, existing technologies are unable to cope with a large number of coal mine safety issues. In addition, although the existing large language models have shown strong versatility in many fields, there are significant deficiencies in knowledge learning and application in the coal mine professional field, resulting in insufficient ability of the model to handle coal mine-specific terminology, work processes, and safety management, which is prone to "hallucination" phenomenon and cannot provide accurate and professional solutions.

[0028] In view of the above-mentioned technical defects, the core concept of the present invention is to construct a domain vocabulary and a general vocabulary respectively by acquiring domain corpus data and general corpus data, and perform weighted fusion based on their respective weights to form a fused vocabulary. It can not only effectively improve the model's ability to understand the professional knowledge of the coal mining industry, but also enhance the performance of the model in specific fields while ensuring universality. By incrementally pre-training the updated large language model, the present invention aims to create a large model for the vertical field of coal mining with efficient reasoning capabilities and professional knowledge, providing strong technical support for safe production and intelligent management of the coal mining industry.

[0029] Figure 1 This is a schematic diagram of the steps of a method for constructing a large vertical model of a coal mine provided by an embodiment of the present invention. Figure 1 , the method comprising: Step S101 : acquiring domain corpus data and general corpus data, and constructing a domain vocabulary and a general vocabulary based on the domain corpus data and the general corpus data respectively.

[0030] In the field of coal mining, there are a large number of professional terms, unique workflows, and strict safety regulations, which are usually not included in the knowledge scope of general large language models. Compared with traditional methods, the present invention pre-collects a large amount of rich coal mining field-related texts and data, including industrial documents, technical reports, operating manuals, safety records, etc. Covering all aspects from mine operations to safety management. The collected data is cleaned and pre-processed, and steps such as noise removal, standardized text format, and processing missing values ​​are performed to ensure that high-quality input can be obtained in the subsequent model training stage. In the process of building a vertical model for coal mines, these professional knowledge are integrated into the incremental pre-training stage to improve the accuracy and effectiveness of the model in understanding and dealing with specific problems in the coal mining field. Significantly improve the performance of the model in the coal mining field, providing a more professional, accurate and safe solution. This not only improves the efficiency of coal mine management and operation, but also provides strong support for the industry's safe production and accident prevention.

[0031] The domain corpus data is text data collected specifically for the coal mining industry, including professional terms, industry standards, operating procedures, technical reports, safety management documents, etc. in the coal mining field, which can reflect the specific knowledge and language usage habits in the coal mining field. It can be understood that the core principles and technical framework of the present invention have wide applicability. With the equivalent replacement of domain corpus data, the present invention can also be applied to other vertical fields, such as medical, financial, legal, manufacturing and other industries. It is only necessary to collect relevant professional text data for a specific field, build a corresponding domain vocabulary, and perform data cleaning and preprocessing to effectively train a pre-trained model that meets the needs of the field.

[0032] The general corpus data refers to a wide range of text data that is not specific to the coal mining field, covers the language used in daily life, and comes from various public resources such as news articles, social media, Wikipedia, books, etc., reflecting the use of daily language and people's common expressions.

[0033] The domain segmentation is obtained by segmentation processing based on the domain corpus data, and includes segmentations commonly used in the coal mining field, such as "mine", "coal mining", "safety regulations", "ventilation system", "miner", etc. Similarly, the general segmentation is obtained by segmentation processing based on the general corpus data, and includes segmentations commonly used in daily life, applicable to a variety of language scenarios and topics, such as "people", "places", "work", "study", "safety", etc. In an optional implementation, step S101 specifically includes steps S1011-step S1013: Step S1011, organize the pre-acquired coal mine vertical field data set into the field corpus data, and organize the pre-acquired open source data set into the general corpus data, the coal mine vertical field data set includes equipment monitoring data, mine geological exploration report data, forum data and industry website data.

[0034] The coal mine vertical field dataset is derived from various data in the coal mine industry, including equipment monitoring data (monitoring data of equipment used for mining, transportation, ventilation, etc. in coal mines), mine geological exploration report data (including detailed descriptions of mine geological conditions, such as rock formations, mineral distribution, and groundwater levels), forum data (discussion data from forums related to the coal mine industry), and industry website data (articles, news, technical reports, etc. from coal mine industry websites). The open source dataset is not specific to the coal mine field, but is derived from a wide range of text data from various public resources.

[0035] The quality, diversity and relevance of coal mine vertical field datasets and open source datasets play a key role in the subsequent training quality and application effect. The embodiments of the present invention are directed to the processing flow of coal mine vertical field datasets or open source datasets, specifically including determining the data source, data collection, data cleaning, data organization and collation, privacy and security, and data quality assessment.

[0036] Specifically, the steps of determining the data source include determining the data source and obtaining data licenses. First, identify and determine the coal mine vertical field data sets that can be used in the coal mine vertical field, covering all aspects from mine operation to safety management. Ensure that appropriate data use licenses and authorizations are obtained from appropriate data providers to ensure the legality and compliance of data collection. The steps of data collection include collecting data from different sources using different methods. For industry data such as equipment monitoring data and mine geological exploration reports, data is collected manually by experts or people in the field. For public open source data, such as forum discussions, industry websites, etc., web crawler tools are used to crawl relevant content from online resources. The data cleaning steps include removing noise and redundancy, handling missing values, and standardization and normalization to ensure the quality, consistency and comparability of the data. Specifically, identify and remove noise and redundant information in the data, such as blank records, duplicate data, incomplete entries, etc., to ensure data quality. For processing missing values ​​in the data, the integrity and availability of the data can be ensured by filling default values, interpolating, or deleting records with missing data. Standardize the data, such as unifying the date format, unit, and specific terminology, to ensure the consistency and comparability of the data. The steps of data organization and collation include that the collected data may come from multiple sources and formats, and unified collation work is required, including file format conversion, text encoding unification, data structure standardization, etc., for subsequent processing and analysis. In order to facilitate management and retrieval, you can choose to establish an index and directory structure for the data. This can be achieved through a database management system or a simple folder structure to ensure orderly storage and rapid access to data. The steps of privacy, security, and quality assessment include anonymization or desensitization of data that may contain personal identity or sensitive information to protect the privacy and security of the data. Establish appropriate access control policies and permission management mechanisms to ensure that only authorized personnel can access and use the data set. The steps of data quality assessment include quality assessment and verification of the cleaned and collated data, including checks on the completeness, accuracy, consistency, and availability of the data to ensure that the data set meets the expected quality standards.

[0037] Step S1012: Process the general corpus data into the general vocabulary.

[0038] The Llama series of models have very powerful capabilities. We use tools such as Chinese-Alpaca-2 to expand the Chinese vocabulary of the Llama2 series models and then perform incremental pre-training to generate a general vocabulary based on general corpus data.

[0039] Step S1013: Process the domain corpus data into the domain vocabulary in a format consistent with the general vocabulary.

[0040] In this embodiment, the sentencepiece tool is used to train a domain vocabulary based on the coal mine vertical domain data set, and then the word segmentation results are aligned with the general vocabulary of the Chinese-Alpaca-2 tool. The SentencePiece tool is an unsupervised word segmentation tool that converts text data into a standard word segmentation form by adaptively learning subword units in the text. The SentencePiece tool is used to train the coal mine vertical domain data set, and a domain vocabulary specifically for the coal mine field is generated based on the domain corpus data.

[0041] The trained domain vocabulary and general vocabulary are aligned to ensure that the domain vocabulary can be consistent with the format of the general vocabulary, so that the two can share the same word segmentation and encoding method, thereby avoiding inconsistencies or conflicts during model training. The goal of aligning the word segmentation results is to make the specific terms in the coal mining field have appropriate representation in the final generated coal mining vertical field model, and to be able to seamlessly combine with the word segmentation of the general language when the coal mining vertical field model is inferred.

[0042] Step S102 , weighted fusion is performed on each segmentation in the general vocabulary and the domain vocabulary based on their respective weights to obtain a fused vocabulary, wherein the weights represent the relative frequencies of the segmentation in the domain corpus data and the general corpus data.

[0043] This embodiment performs weighted fusion of the segmentations in the general vocabulary and the domain vocabulary in order to generate a fused vocabulary that accurately characterizes the importance of each segmentation in different corpora. The fused vocabulary not only retains the knowledge specific to the coal mining field, but also maintains the wide applicability of the general language. Specifically, the segmentations in the domain corpus data and the general corpus data may differ in frequency of occurrence, so the weight of each segmentation is determined by its relative frequency in the domain corpus data and the general corpus data to ensure that the domain-specific segmentations are given a higher weight in the fused vocabulary, thereby improving the model's ability to understand domain-related content. In short, the purpose of weighted fusion is to balance the importance of general segmentations and domain segmentations, so that the fused vocabulary finally generated can not only effectively process general texts, but also accurately identify segmentations specific to the coal mining field, thereby improving the performance of the large model in the coal mining vertical field in the professional field.

[0044] Step S103: Based on the fusion vocabulary, an embedding model is trained using the domain corpus data and the general corpus data, wherein the embedding model is used to map word segmentation to corresponding word vectors based on semantic information.

[0045] The embedding model is trained using a fusion vocabulary to map word segments to corresponding word vectors and optimize the word vector representation through semantic information. Specifically, the fusion vocabulary combines word segments from the general corpus data and the domain corpus data, and assigns each word segment a corresponding weight, so that the embedding model can learn how to generate corresponding word vectors based on the contextual information of the word segmentation during the training process. When training the embedding model, the text data of the domain corpus data and the general corpus data are input into the embedding model. The embedding model continuously adjusts the word vectors so that word segments with similar meanings have close vector representations, and then uses semantic information to correctly locate each word segment in the semantic space. The goal of this training process is to enable the word vector to accurately capture the semantic relationship between word segments, so that it can reflect both the meaning in the general context and the knowledge specific to the coal mining field. Ultimately, the trained embedding model will convert word segments (whether general or specific to the coal mining field) into meaningful word vectors.

[0046] Step S104, loading the pre-trained original large language model, and replacing the trained embedding model with the embedding layer of the original large language model to obtain an updated large language model.

[0047] First, select and load a general large language model that has been pre-trained on a large scale as the original large language model, that is, the base model. Common base models include BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-trained Transformer), RoBERTa (Robustly optimized BERT approach), T5 (Text-To-Text Transfer Transformer), etc. For the vertical field of coal mines, the open source Llama-2-7b model using the GPT framework is selected as the base model in this embodiment. The original large language model has been trained with a large amount of general data and has good language understanding ability, but its embedding layer (that is, the part of the word vector) may not be suitable for the special word segmentation and context of a specific field. Therefore, update its embedding layer so that the large language model can better handle the specific terms and languages ​​in the coal mine field. Replace the trained embedding model (that is, the embedding layer corresponding to the fusion vocabulary obtained by training in step S103) with the embedding layer of the original large language model. The role of the embedding layer is to convert the input word segmentation into a vector representation, and these word vectors will serve as the basis for the large language model to process text. Through replacement, the new embedding model can provide the large language model with word vectors that are more suitable for specific fields, allowing the model to understand the professional terms in the field more accurately.

[0048] Step S105, using the domain corpus data and the general corpus data to perform incremental pre-training on the updated large language model to obtain a large model for the coal mine vertical domain.

[0049] The basic idea of ​​incremental pre-training is to perform further training on the existing updated large language model (the embedding layer has been replaced by step S104 at this time) instead of training from scratch. The advantage of incremental training is that it can fine-tune the model through additional domain corpus data on the basis of maintaining most of the learned knowledge, so that its performance in the coal mining field is enhanced. In this way, the model that originally performed well in the general context can better adapt to the special needs and knowledge in the coal mining field. In the incremental pre-training process, further training is performed using domain corpus data and general corpus data. By using these two types of data at the same time, the large language model can retain the ability to process general language while enhancing domain knowledge. After incremental pre-training, a large model of the vertical field of coal mines that can perform more accurately on data in the coal mining field is finally obtained. For example, for texts such as coal mine operating manuals and safety records, the large model of the vertical field of coal mines can understand and generate more professional content. At the same time, since the knowledge of general data is still retained, the large model of the vertical field of coal mines can also maintain a certain degree of versatility in tasks involving other fields.

[0050] The present invention achieves deep learning and precise expression of professional knowledge of the coal mining industry by constructing a large model of the vertical field of coal mines and combining the domain corpus data with the general corpus data. By weighted fusion of the domain vocabulary and the general vocabulary, the model's ability to understand specific terms and workflows in the coal mining field can be effectively improved, avoiding the "hallucination" phenomenon that often occurs in general large language models when dealing with professional problems in coal mines. In addition, the embedded model can fully capture domain semantic information during the training process, thereby improving the performance accuracy of the model in tasks such as coal mine safety management, technical operations, and mine monitoring. Finally, the large model of the vertical field of coal mines obtained through incremental pre-training can provide more professional and accurate coal mine solutions while ensuring the versatility of the model, greatly improving coal mine management, operational efficiency, and safe production levels, and providing strong support for the intelligent and digital transformation of the coal mining industry.

[0051] In an optional implementation, the weighted fusion of each word in the general vocabulary and the domain vocabulary based on their respective weights to obtain a fused vocabulary includes: Step S1031, in the general vocabulary and the domain vocabulary, the frequency and weight of each word segment are calculated.

[0052] For each word segment in the general vocabulary and the domain vocabulary, the number of times it appears in the respective corpus is calculated. Words with high frequency of occurrence are highly representative in the language. The weight is calculated based on the relative importance of the word frequency in the entire corpus. The weight of each word segment reflects its relative importance in the coal mine field and the general context. Through the weight, it is possible to highlight those professional terms or word segments that frequently appear in the domain corpus data, and reduce the influence of words that are common in the general corpus but not very important in the field. The design of the weight usually takes into account the impact of domain-specific word segmentation on the model, so that in the subsequent word table fusion process, the high weight of the domain word segmentation can be retained, thereby improving the effect of the domain task. Through the word frequency and weight calculation of this step, a basis can be provided for the subsequent word table fusion of step S104. During fusion, high-weighted domain word segmentations will receive more attention and optimization, so that the large model of the vertical field of coal mines aimed to be obtained will perform more accurately in the coal mine field task. At the same time, the processing of general word segmentations will also be based on their weights to ensure that the model will not lose its general language processing ability due to excessive emphasis on a certain type of word segmentation during the fusion process.

[0053] Step S1032: Based on the weights, linearly combine the word segments in the general vocabulary and the domain vocabulary to obtain the fused vocabulary.

[0054] In this step, the segmentations in the general vocabulary and the domain vocabulary are not simply superimposed, but weightedly merged according to the weight of each segmentation, which can effectively highlight important professional terms in the field while retaining the wide applicability of the general vocabulary. When performing linear combination, the weight of each segmentation reflects its importance and relevance in the respective corpus. Based on the weights, it is ensured that the segmentations that frequently appear in the domain corpus and are critical to the model task occupy a more important position in the fused vocabulary. On the contrary, those that are less important in the coal mining field but common in the general context are appropriately reduced in their influence. In an optional embodiment, the calculation of the word frequency and weight of each segmentation in the general vocabulary and the domain vocabulary specifically includes the following steps: For each word segment in the general vocabulary and the domain vocabulary, the occurrence frequency of the word segment in the domain corpus data is determined as a first word frequency, or the occurrence frequency of the word segment in the general corpus data is determined as a second word frequency.

[0055] For each word in the general vocabulary and domain vocabulary , calculate their frequency of occurrence in their respective corpora and obtain the first word frequency (field word frequency) , represents the number of occurrences of the segmented word in the domain corpus, and, obtains the second word frequency (general word frequency) , indicating participle The number of occurrences in the general corpus.

[0056] For each word segment in the general vocabulary, the relative frequency of the word segment in the general corpus data is calculated according to the first word frequency, and the relative frequency of the word segment in the general corpus data is determined as a first linear item. .

[0057] Calculate each word Relative frequency in general corpus data, i.e. domain relative frequency . Field relative frequency As the first linear term, it represents the relative importance of the segmentation in the field. The calculation formula is as follows:

[0058] In the formula, is the total frequency of occurrence of all word segments in the domain corpus data.

[0059] For each word segment in the domain vocabulary, the relative frequency of the word segment in the domain corpus data is calculated according to the second word frequency, and the relative frequency of the word segment in the general corpus data is determined as the second linear item. .

[0060] Calculate each word The relative frequency in the general corpus data, also known as the general relative frequency . Universal relative frequency As the second linear term, it represents the relative importance of the segmentation in the general corpus. The calculation formula is as follows:

[0061] In the formula, is the total frequency of occurrence of all tokens in the general corpus data.

[0062] For the same segmented words in the general vocabulary and the domain vocabulary, the logarithm of the ratio of the first word frequency to the second word frequency of the segmented words is determined as a logarithmic term.

[0063] For the same word in the general vocabulary and the domain vocabulary, calculate the logarithm of the ratio of the first word frequency to the second word frequency of the word As a logarithmic term. As the weight of the segmentation in the fusion vocabulary, the segmentation with a relative frequency higher in the domain corpus than in the general corpus is highlighted and the ratio is smoothed by logarithm. The calculation formula is as follows:

[0064] Preset weight adjustment parameters are respectively assigned to the first linear term, the second linear term and the logarithmic term, and the weight of each word segmentation is calculated.

[0065] Assign a preset weight adjustment parameter to each linear term and logarithmic term. The weight adjustment parameters corresponding to the first linear term, the second linear term and the logarithmic term are: , and The final weight By weighted addition of the above linear and logarithmic terms, we get:

[0066] The weight is combined with the original word frequency (first word frequency and second word frequency) through the above formula to obtain a smoother and more effective weight In this embodiment, in order to make the segmentation words in the domain vocabulary more important in a specific task than the segmentation words in the general vocabulary, The value is greater than The value of . Appropriate adjustments can be made based on the relative importance of the segmented words in the domain vocabulary in the general vocabulary. The value should be lower than , but can be The value is equivalent to or slightly higher.

[0067] In an optional implementation, after obtaining the fused vocabulary, the method further includes: checking whether there are unmatched segmented words in the domain vocabulary, the unmatched segmented words being segments that exist in the domain vocabulary but not in the general vocabulary; and if the unmatched segmented words are detected, adding the segmented words to the fused vocabulary.

[0068] When constructing the fusion vocabulary, the domain vocabulary and the general vocabulary are weightedly fused through the weight adjustment parameters. Although the vast majority of segmented words may have been successfully merged, in some specific cases, some segmented words in the domain vocabulary may not appear in the general vocabulary. Unmatched segmented words refer to words that exist in the domain vocabulary but have no corresponding items in the general vocabulary. This may be due to the particularity of the domain vocabulary or the general vocabulary does not cover the professional terms, abbreviations, etc. in this field. By comparing the domain vocabulary and the general vocabulary, check which words only appear in the domain vocabulary but not in the general vocabulary. These words are unmatched segmented words. For example, the coal mining field may involve terms such as "gas leakage" and "mining face", which may not be common in the general corpus data, so they may not appear in the general vocabulary.

[0069] In order to ensure that the final fused vocabulary can fully cover the domain knowledge and does not miss any important domain terms. These unmatched segmented words are extracted from the domain vocabulary and added directly to the final fused vocabulary. In this way, the fused vocabulary contains both segmented words from the general vocabulary and segmented words in the domain vocabulary that are not covered by the general vocabulary. The priority of adding unmatched segmented words can be determined based on the importance or frequency of the segmented words to ensure that the most common and important segmented words are added to the fused vocabulary first. Optionally, after obtaining the fused vocabulary, it is also possible to check whether the vocabulary contains coal mining field-specific terms and common words to ensure that the required content is covered. Based on the verification results and feedback from domain experts, the aligned vocabulary is optimized to ensure that the coal mining field-specific terms and common words of the general language are covered.

[0070] In an optional implementation, the embedding model is trained based on the fused vocabulary using the domain corpus data and the general corpus data, including: mixing the domain corpus data and the general corpus data in a preset ratio to obtain training corpus data, wherein the general corpus data occupies a higher ratio than the domain corpus data; based on the fused vocabulary, performing word segmentation processing on the training corpus data to obtain multiple training word segmentations; based on the multiple training word segmentations, training the embedding model according to preset training parameters, wherein the training parameters include batch size, learning rate and loss function.

[0071] When training the embedding model, we combine domain corpus data and general corpus data. By mixing the two corpus data, we ensure that the training corpus data can reflect domain knowledge while retaining the broad coverage of general language. The ratio of domain corpus data to general corpus data is pre-set, with a relatively small proportion of domain corpus data, which can be 20% or 30%; and a relatively large proportion of general corpus data.

[0072] In order to convert text data into a form that can be understood by the embedding model, the original text of the training corpus data is segmented to obtain the vocabulary form required for embedding model training. Use the fusion vocabulary to perform word segmentation to ensure that the segmentation results cover both domain vocabulary and general vocabulary. Segment the mixed training corpus data according to the fusion vocabulary. Through word segmentation, multiple training word segments are obtained, each of which is a unit that the embedding model needs to recognize. The final training word segmentation will be used as the input of the embedding model.

[0073] The goal of training an embedding model is to let the model learn how to convert the input word segmentation into high-dimensional word vectors so that these word vectors can fully represent the semantic information of the vocabulary, thereby making accurate predictions or inferences in subsequent tasks. When training an embedding model, set specific training parameters to guide the model training process, including: batch size (BatchSize), learning rate (Learning Rate) and loss function (Loss Function). When training an embedding model, you can use contrastive loss (Contrastive Loss) or multiple negative sample contrast loss (Multiple Negatives Sampling Loss) as the loss function to help the embedding model learn the semantic relationship between word embeddings.

[0074] Table 1 shows an example of embedding model training parameter settings, see Table 1: The base model is Dmeta-Embedding-base, which is a pre-trained embedding model that can generate text embedding vectors.

[0075] The train batch size is set to 16, and the amount of training corpus data used in each training is 16 samples.

[0076] Epochs (training cycle) is 20. During the training process, the training corpus data will be traversed 20 times (i.e. 20 Epochs). Each Epoch contains a complete training data transfer.

[0077] Select MultipleNegativesSymmetricRankingLoss for Loss (loss function). The embedding model is trained by comparing the distance between positive and negative samples, with the goal of making the embedding of positive samples closer and the embedding of negative samples farther away.

[0078] The learn rate is set to 2e-05, or 0.00002, which indicates the step size for each parameter update. The learning rate scheduler uses the WarmupLinearSchedule method. In the early stages of training, the learning rate increases gradually, and then decays linearly during training. This helps avoid instability caused by too large a learning rate at the beginning of training, and gradually reduces the learning rate as training progresses, improving training accuracy.

[0079] The number of warmup steps is set to 5384, which means that at the beginning of training, the learning rate will gradually increase from the initial small value until it reaches the target learning rate of 2e-05.

[0080] The Evaluation steps are set to 2692, which means that during the training process, the embedding model is evaluated every 2692 steps (i.e., several small batches in a training process) to ensure that there is no overfitting or performance degradation during the training process.

[0081] Table 1

[0082] In an optional embodiment, the method also includes: obtaining a high-dimensional word vector output by the embedding model, and determining the core semantic information and secondary semantic information represented by the high-dimensional word vector; performing dimensionality reduction processing on the high-dimensional word vector to obtain a corresponding low-dimensional word vector, wherein the core semantic information represented by the high-dimensional word vector is stored in the head dimension of the low-dimensional word vector, and the secondary semantic information represented by the high-dimensional word vector is stored in the tail dimension of the low-dimensional word vector; and storing the low-dimensional word vector in a data structure predefined by the embedding model.

[0083] First, obtain high-dimensional word vectors from the embedding model. The "high-dimensional" here refers to the multi-dimensional representation generated by the embedding model for each word segment, which has a higher dimension (such as 768 dimensions or higher), and is distinguished from the low-dimensional word vectors in the following text. In high-dimensional word vectors, the embedding model represents semantic information at different levels. Core semantic information refers to the most important and representative meaning of the word in a specific context, while secondary semantic information refers to other possible meanings of the word or additional information in the context. By analyzing high-dimensional word vectors, we can identify which dimensions mainly correspond to core semantics and which dimensions correspond to secondary semantics.

[0084] The purpose of this embodiment to reduce the dimensionality of high-dimensional word vectors is to reduce storage space and computational complexity while retaining important semantic information as much as possible. The low-dimensional word vector obtained after dimensionality reduction has a low dimension (such as 256 or 512 dimensions), which is suitable for subsequent large language model training and reasoning. During the dimensionality reduction process, the core semantic information is retained in the head dimension of the low-dimensional word vector, while the secondary semantic information is stored in the tail dimension. Ensure that in the low-dimensional representation, the most important information (core semantics) can be accessed and used first, while the secondary information can be called when needed. This can improve the efficiency and accuracy of the model in specific tasks.

[0085] Finally, the generated low-dimensional word vector is saved in a data structure predefined by the embedding model for subsequent use and calling. The present invention improves the storage process of word vectors, which helps to quickly access and process word vectors and improve the operating efficiency of the final generated large model of the coal mine vertical field.

[0086] Table 2 shows the performance improvement of the trained embedding model, see Table 2: Table 2

[0087] The original embedding model has an embedding dimension of 768 and a FAHR index of 93.48. When the embedding dimension of the embedding model is reduced to 256, the FAHR index drops to 93.02. Despite the reduction in dimension, there is almost no performance drop, only a slight drop of 0.46 percentage points. This shows that after reducing the embedding dimension, the optimized model still maintains strong performance, and the storage and computing resource consumption is significantly reduced. When the dimension is increased to 512, the FAHR value is still 93.02, the same as the case of 256 dimensions. In other words, there is no significant difference in performance between the 256-dimensional and 512-dimensional embedding models. This result shows that within a specific range of embedding dimensions, the performance of the embedding model tends to be stable, and the embedding dimension is further optimized to reduce resource consumption without losing performance. When the embedding dimension is restored to 768, the FAHR value rebounds slightly to 93.13. Although it is still lower than the original embedding model (93.48), the performance gap is small, with only a drop of 0.35 percentage points. This shows that the optimized embedding model can still maintain good performance in high-dimensional cases.

[0088] In an optional implementation, replacing the trained embedding model with the embedding layer of the original large language model to obtain an updated large language model includes: replacing the embedding layer of the original large language model with the trained embedding model to obtain the embedding layer of the updated large language model; and initializing the embedding layer of the updated large language model using the word vector output by the trained embedding model.

[0089] The embedding layer is the first layer of the original large language model, which is used to convert the input text into a vector representation. The embedding layer in the original large language model contains a general vocabulary and embedding method, which is not optimized for the specific coal mining field. Therefore, the embedding layer that originally belonged to the general large language model is replaced with the trained embedding model. In this way, the large language model can obtain word vectors specifically for the coal mining field, which can more accurately represent the unique terms, concepts and contexts of the coal mining field. In the updated large language model, the new embedding layer will be initialized to the word vectors output by the trained coal mining field embedding model. Specifically, these word vectors will be directly applied to the embedding layer of the large language model, replacing the content in the original embedding layer. Using the word vectors output by the trained embedding model to initialize the embedding layer of the updated large language model determines how the updated large language model processes the input text.

[0090] In an optional implementation, the incremental pre-training of the updated large language model using the domain corpus data and the general corpus data to obtain a large model for the vertical domain of coal mines includes: Step S1051 : Mix the domain corpus data and the general corpus data in a preset ratio to obtain training corpus data, wherein the general corpus data occupies a higher ratio than the domain corpus data.

[0091] In the previous article, when training the embedding model, we have explained in detail how to obtain the training corpus data. Based on the same or similar process, we will not repeat it here.

[0092] Step S1052, adding a low-rank matrix and a bias matrix to the weight matrix of the linear layer of the updated large language model.

[0093] This embodiment uses LoRA (Low-Rank Adaptation) technology, which allows the large language model to perform effective adaptive adjustments with a smaller number of parameters by adding a low-rank matrix to the weight matrix of the updated large language model. The addition of a low-rank matrix can help the large language model learn new domain knowledge with less computational and storage overhead during incremental learning. The role of the bias matrix is ​​to further adjust the calculation results of the weight matrix and increase the flexibility of the large language model. By making appropriate adjustments to the weights and biases, the large language model can better adapt to the new data distribution.

[0094] Step S1053, freezing the other layers of the updated large language model except the embedding layer, and training the embedding layer of the updated large language model using the training corpus data.

[0095] Freezing all layers except the embedding layer is to avoid over-adjustment of the weights of other layers in the large language model in the early stages of incremental pre-training. At this time, only the embedding layer (that is, the word vector layer) will be trained, which ensures that the newly added word embeddings focus on learning specific vocabulary and terminology in the coal mining field while maintaining the original semantics. When other layers are frozen, only the embedding layer will be trained based on the mixed training corpus data to help the large language model understand the professional terminology in the coal mining field and ensure that the semantic representation of the large language model is closer to the knowledge of the coal mining industry.

[0096] Step S1054, when the preset iteration condition is met, unfreeze the other layers, use the training corpus data to fine-tune all layers of the updated large language model, and obtain a low-rank adaptation model corresponding to the large language model.

[0097] After the embedding layer is trained, the other layers in the large language model are unfrozen so that they can also participate in the training. By fine-tuning all layers, the large language model can gradually adapt to the special context of the coal mining field while retaining the original general knowledge. The unfreezing operation can be selected after the embedding layer has completed preliminary learning to ensure good model performance during fine-tuning. At this point, the large language model is fully fine-tuned based on the mixed training corpus data. Through this process, the large language model can be further refined on the basis of its original general capabilities to adapt it to the grammar, terminology, and context of the coal mining field. Fine-tuning can improve the large language model's ability to handle domain-specific tasks. After this step, a large language model adjusted by low-rank adaptation (LoRA) is obtained, which can effectively utilize the knowledge in the coal mining field while retaining the advantages of the general large language model.

[0098] Step S1055, merging the low-rank adaptive model with the original large language model to obtain the coal mine vertical domain large model.

[0099] Finally, the low-rank adaptation model after incremental pre-training and fine-tuning is merged with the original large language model. The purpose is to integrate the model results after domain adaptation into a complete large language model so that it can handle general tasks and tasks in the coal mining field at the same time. The merged model has cross-domain knowledge and adaptability, and can be more professional and accurate in the coal mining field. After the merger is completed, the invention aims to construct a large model for the vertical field of coal mining.

[0100] Figure 2 This is a structural block diagram of a device for constructing a large vertical model of a coal mine provided by an embodiment of the present invention. Figure 2 , the device comprises: An acquisition module 201 is used to acquire domain corpus data and general corpus data, and to construct a domain vocabulary and a general vocabulary based on the domain corpus data and the general corpus data respectively; A fusion module 202 is used to perform weighted fusion on each segmentation in the general vocabulary and the domain vocabulary based on their respective weights to obtain a fused vocabulary, wherein the weights represent the relative frequencies of the segmentation in the domain corpus data and the general corpus data; A first training module 203 is used to train an embedding model based on the fusion vocabulary using the domain corpus data and the general corpus data, wherein the embedding model is used to map word segmentation to corresponding word vectors based on semantic information; An updating module 204 is used to load the pre-trained original large language model and replace the trained embedding model with the embedding layer of the original large language model to obtain an updated large language model; The second training module 205 is used to perform incremental pre-training on the updated large language model using the domain corpus data and the general corpus data to obtain a large model for the vertical domain of coal mines.

[0101] In an optional implementation, the fusion module includes: A weight calculation submodule, used for calculating the frequency and weight of each word in the general vocabulary and the field vocabulary; The linear combination submodule is used to linearly combine the respective participles in the general vocabulary and the domain vocabulary based on the weights to obtain the fused vocabulary.

[0102] In an optional implementation, the weight calculation submodule includes: A word frequency determination unit, configured to determine, for each word segment in the general word list and the domain word list, the frequency of occurrence of the word segment in the domain corpus data as a first word frequency, or to determine the frequency of occurrence of the word segment in the general corpus data as a second word frequency; A first relative frequency determination unit is configured to calculate, for each word segment in the general vocabulary, a relative frequency of the word segment in the general corpus data according to the first word frequency, and determine the relative frequency of the word segment in the general corpus data as a first linear item; A second relative frequency determination unit is used to calculate, for each word segment in the domain vocabulary, a relative frequency of the word segment in the domain corpus data according to the second word frequency, and determine the relative frequency of the word segment in the general corpus data as a second linear item; A logarithmic term determining unit, configured to determine, for the same segmented word in the general vocabulary and the domain vocabulary, the logarithm of the ratio of the first word frequency to the second word frequency of the segmented word as a logarithmic term; The linear combination unit is used to respectively assign preset weight adjustment parameters to the first linear term, the second linear term and the logarithmic term, and calculate the weight of each word segment.

[0103] In an optional embodiment, the device further comprises: A checking module, used to check whether there are unmatched segmented words in the domain vocabulary, wherein the unmatched segmented words are segmented words that exist in the domain vocabulary but not in the general vocabulary; The adding module is used to add the segmented word to the fusion word list when the unmatched segmented word is detected.

[0104] In an optional implementation, the first training module includes: A first mixing submodule is used to mix the domain corpus data and the general corpus data according to a preset ratio to obtain training corpus data, wherein the general corpus data occupies a higher proportion than the domain corpus data; A word segmentation processing submodule, used for performing word segmentation processing on the training corpus data based on the fusion word list to obtain a plurality of training word segments; The first training submodule is used to train the embedding model based on the multiple training word segmentations according to preset training parameters, where the training parameters include batch size, learning rate and loss function.

[0105] In an optional embodiment, the device further comprises: A word vector acquisition module, used to acquire the high-dimensional word vector output by the embedding model, and determine the core semantic information and secondary semantic information represented by the high-dimensional word vector; A dimensionality reduction module, used to perform dimensionality reduction processing on the high-dimensional word vector to obtain a corresponding low-dimensional word vector, wherein the core semantic information represented by the high-dimensional word vector is stored in the head dimension of the low-dimensional word vector, and the secondary semantic information represented by the high-dimensional word vector is stored in the tail dimension of the low-dimensional word vector; The word vector saving module is used to save the low-dimensional word vector in a data structure predefined by the embedding model.

[0106] In an optional implementation, the update module includes: A replacement submodule, used to replace the embedding layer of the original large language model with the trained embedding model to obtain the embedding layer of the updated large language model; The initialization submodule is used to initialize the embedding layer of the updated large language model using the word vector output by the trained embedding model.

[0107] In an optional implementation, the second training module includes: A second mixing submodule is used to mix the domain corpus data and the general corpus data according to a preset ratio to obtain training corpus data, wherein the general corpus data occupies a higher ratio than the domain corpus data; A matrix adding submodule, used for adding a low-rank matrix and a bias matrix to the weight matrix of the linear layer of the updated large language model; A freezing submodule, used to freeze the other layers of the updated large language model except the embedding layer, and train the embedding layer of the updated large language model using the training corpus data; An unfreezing submodule, used for unfreezing the other layers when a preset iteration condition is met, and fine-tuning all layers of the updated large language model using the training corpus data to obtain a low-rank adaptation model corresponding to the large language model; The merging submodule is used to merge the low-rank adaptive model with the original large language model to obtain the large model of the coal mine vertical field.

[0108] In an optional implementation, the acquisition module includes: A sorting submodule, used to sort the pre-acquired coal mine vertical field data set into the field corpus data, and to sort the pre-acquired open source data set into the general corpus data, wherein the coal mine vertical field data set includes equipment monitoring data, mine geological exploration report data, forum data and industry website data; A first processing submodule, configured to process the general corpus data into the general vocabulary; The second processing submodule is used to process the domain corpus data into the domain vocabulary in a format consistent with the general vocabulary.

[0109] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, electronic devices and storage media. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0110] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods and apparatuses according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing terminal device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded into a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0111] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0112] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0113] The above is a detailed introduction to the method and device for constructing a large vertical field model of a coal mine provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for constructing a large vertical model of a coal mine, characterized in that: The method comprises: Acquire domain corpus data and general corpus data, and construct a domain vocabulary and a general vocabulary based on the domain corpus data and the general corpus data respectively; Performing weighted fusion on each segmentation in the general vocabulary and the domain vocabulary based on their respective weights to obtain a fused vocabulary, wherein the weights represent the relative frequencies of the segmentation in the domain corpus data and the general corpus data; Based on the fusion vocabulary, an embedding model is trained using the domain corpus data and the general corpus data, wherein the embedding model is used to map word segmentation to corresponding word vectors based on semantic information; Loading the pre-trained original large language model, and replacing the trained embedding model with the embedding layer of the original large language model to obtain an updated large language model; The updated large language model is incrementally pre-trained using the domain corpus data and the general corpus data to obtain a large model for the coal mine vertical domain.

2. The method according to claim 1, characterized in that: The step of weighting and fusing each word in the general vocabulary and the domain vocabulary based on their respective weights to obtain a fused vocabulary includes: In the general vocabulary and the field vocabulary, the frequency and weight of each word segment are calculated; Based on the weights, each word segment in the general vocabulary and the domain vocabulary is linearly combined to obtain the fused vocabulary.

3. The method according to claim 2, characterized in that The step of calculating the frequency and weight of each word segment in the general vocabulary and the domain vocabulary includes: For each word segment in the general vocabulary and the domain vocabulary, determining the occurrence frequency of the word segment in the domain corpus data as a first word frequency, or determining the occurrence frequency of the word segment in the general corpus data as a second word frequency; For each word segment in the general vocabulary, calculate the relative frequency of the word segment in the general corpus data according to the first word frequency, and determine the relative frequency of the word segment in the general corpus data as a first linear item; For each word segment in the domain vocabulary, calculate the relative frequency of the word segment in the domain corpus data according to the second word frequency, and determine the relative frequency of the word segment in the general corpus data as a second linear item; For the same segmented words in the general vocabulary and the domain vocabulary, the logarithm of the ratio of the first word frequency to the second word frequency of the segmented words is determined as a logarithmic term; Preset weight adjustment parameters are respectively assigned to the first linear term, the second linear term and the logarithmic term, and the weight of each word segmentation is calculated.

4. The method according to claim 1 or 2, characterized in that: After obtaining the fused vocabulary, the method further includes: Checking whether there are any unmatched segmented words in the domain vocabulary, wherein the unmatched segmented words are segmented words that exist in the domain vocabulary but not in the general vocabulary; When the unmatched segmented word is detected, the segmented word is added to the fusion word list.

5. The method according to claim 1, characterized in that: The step of training the embedding model based on the fused vocabulary using the domain corpus data and the general corpus data includes: Mixing the domain corpus data and the general corpus data according to a preset ratio to obtain training corpus data, wherein the general corpus data occupies a higher ratio than the domain corpus data; Based on the fusion vocabulary, the training corpus data is segmented to obtain a plurality of training segmented words; Based on the multiple training participles, the embedding model is trained according to preset training parameters, where the training parameters include batch size, learning rate, and loss function.

6. The method according to claim 5, characterized in that The method further comprises: Obtaining a high-dimensional word vector output by the embedding model, and determining the core semantic information and secondary semantic information represented by the high-dimensional word vector; Performing dimensionality reduction processing on the high-dimensional word vector to obtain a corresponding low-dimensional word vector, wherein the core semantic information represented by the high-dimensional word vector is stored in the head dimension of the low-dimensional word vector, and the secondary semantic information represented by the high-dimensional word vector is stored in the tail dimension of the low-dimensional word vector; The low-dimensional word vector is stored in a data structure predefined by the embedding model.

7. The method according to claim 1, characterized in that The step of replacing the trained embedding model with the embedding layer of the original large language model to obtain an updated large language model includes: Replacing the embedding layer of the original large language model with the trained embedding model to obtain the embedded layer of the updated large language model; The embedding layer of the updated large language model is initialized using the word vectors output by the trained embedding model.

8. The method according to claim 7, characterized in that The method of using the domain corpus data and the general corpus data to incrementally pre-train the updated large language model to obtain a large model for the vertical domain of coal mines includes: Mixing the domain corpus data and the general corpus data according to a preset ratio to obtain training corpus data, wherein the general corpus data occupies a higher ratio than the domain corpus data; Adding a low-rank matrix and a bias matrix to the weight matrix of the linear layer of the updated large language model; Freeze the other layers of the updated large language model except the embedding layer, and train the embedding layer of the updated large language model using the training corpus data; When a preset iteration condition is met, unfreeze the other layers, and use the training corpus data to perform fine-tuning training on all layers of the updated large language model to obtain a low-rank adaptation model corresponding to the large language model; The low-rank adaptive model is merged with the original large language model to obtain the large model of the coal mine vertical field.

9. The method according to claim 1, characterized in that: The acquiring of domain corpus data and general corpus data, and constructing a domain vocabulary and a general vocabulary based on the domain corpus data and the general corpus data, respectively, includes: Organizing the pre-acquired coal mine vertical field data set into the field corpus data, and organizing the pre-acquired open source data set into the general corpus data, wherein the coal mine vertical field data set includes equipment monitoring data, mine geological exploration report data, forum data and industry website data; processing the general corpus data into the general vocabulary; The domain corpus data is processed into the domain vocabulary that is unified with the general vocabulary format.

10. A device for constructing a large vertical model of a coal mine, characterized in that: The device comprises: An acquisition module, used for acquiring domain corpus data and general corpus data, and constructing a domain vocabulary and a general vocabulary based on the domain corpus data and the general corpus data respectively; A fusion module, used for weighted fusion of each segmentation in the general vocabulary and the domain vocabulary based on their respective weights to obtain a fused vocabulary, wherein the weights represent the relative frequencies of the segmentation in the domain corpus data and the general corpus data; A first training module is used to train an embedding model based on the fusion vocabulary using the domain corpus data and the general corpus data, wherein the embedding model is used to map word segmentation to corresponding word vectors based on semantic information; An updating module is used to load the pre-trained original large language model and replace the trained embedding model with the embedding layer of the original large language model to obtain an updated large language model; The second training module is used to perform incremental pre-training on the updated large language model using the domain corpus data and the general corpus data to obtain a large model for the coal mine vertical domain.

Images