Training method and device of vertical field large language model and electronic equipment
By extracting frequently occurring adjacent character pairs from the target vertical domain text data, building a vocabulary list and combining general domain data for training, the problem of limited application effects of traditional large language models in specific fields is solved, and good performance in specific fields is achieved model training.
Patent Information
- Application Number
- CN202410880683.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-02
- Publication Date
- 2024-11-29
AI Technical Summary
The application effect of traditional large language models in specific vertical fields is limited, mainly due to the limited number of labeled data in specific fields, high cost and possible sensitive information, which limits the applicability and effectiveness of the model.
By extracting frequently occurring adjacent character pairs from the target vertical domain text data, building a vocabulary list, expanding the vocabulary list of the basic large language model, and continuing pre-training and supervised fine-tuning in combination with general domain data, forming a large number suitable for specific fields Language model.
Effectively utilize limited annotation data to train large language models that perform well in specific vertical fields, improve the model's recognition and understanding ability in specific fields, and enhance the processing ability of the annotation training corpus.
Smart Images

Figure CN119047566A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a training method, device and electronic equipment for a large language model in a vertical field. Background Art
[0002] In the field of artificial intelligence, large language models such as ChatGPT and its peers have become an important driving force for technological development in recent years. These models usually rely on massive parameters and powerful learning capabilities, and can achieve remarkable results in multiple application areas such as natural language processing. However, although large language models excel in versatility and versatility, their application effects in specific vertical fields are still limited.
[0003] Traditional large language model training methods usually include pre-training and supervised fine-tuning. In the pre-training stage, the model learns the basic structure and semantic rules of the language through large-scale unlabeled text data; in the supervised fine-tuning stage, the model is further adjusted through annotated data in specific fields to adapt to specific tasks. Although this process improves the performance of the model on specific tasks, the annotated data in specific fields is often limited in quantity, expensive, and may involve sensitive information, which limits the applicability and effectiveness of the model.
[0004] Therefore, how to make full use of the limited annotated data in a specific vertical field and train a large language model that performs well in the specific vertical field based on the limited annotated data is a technical problem that needs to be solved urgently. Summary of the invention
[0005] In view of the problems existing in the prior art, the present invention provides a training method, device and electronic device for a large language model in a vertical field, so as to make full use of limited annotated data in a specific vertical field, so as to train a large language model that performs well in the specific vertical field based on the limited annotated data.
[0006] The present invention provides a method for training a large language model in a vertical field, comprising the following steps.
[0007] Acquire target vertical field text data; extract frequently occurring adjacent character pairs from the target vertical field text data to obtain a target vertical field vocabulary; use the target vertical field vocabulary to expand the original vocabulary of a basic large language model; continue to train the basic large language model after the original vocabulary is expanded to obtain a target vertical field large language model, so as to use the target vertical field large language model to generate answers to target vertical field questions.
[0008] According to a training method for a vertical field large language model provided by the present invention, the frequently occurring adjacent character pairs are extracted from the target vertical field text data to obtain a target vertical field vocabulary, including: decomposing the target vertical field text data into single characters to obtain a first character sequence; performing character frequency statistics on the first character sequence to filter out frequently occurring adjacent character pairs from the first character sequence; and constructing the target vertical field vocabulary based on all adjacent character pairs filtered out from the first character sequence.
[0009] According to a training method for a vertical domain large language model provided by the present invention, the method performs character frequency statistics on the first character sequence to filter out frequently occurring adjacent character pairs from the first character sequence, including: performing character frequency statistics on the first character sequence, and filtering out the most frequently occurring adjacent character pairs from the first character sequence according to the statistical results; re-decomposing the remaining characters in the first character sequence that have not been filtered out into single characters, continuing to perform character frequency statistics on the re-decomposed single characters, and filtering out the most frequently occurring adjacent character pairs according to the statistical results, and repeating this step until the filtering end condition is met.
[0010] According to a training method for a vertical domain large language model provided by the present invention, the target vertical domain vocabulary is used to expand the original vocabulary of the basic large language model, including: for each word in the target vertical domain vocabulary, the following operations are performed: a plurality of words whose similarity to the word is greater than a preset threshold are obtained from the original vocabulary, and the average value of the initial vector representations of the plurality of words is calculated; the average value of the initial vector representations of the plurality of words is used as the initial vector representation of the word; and the adjacent characters and their initial vector representations are added to the original vocabulary.
[0011] According to a training method for a vertical domain large language model provided by the present invention, the method also includes: acquiring general domain text data; continuing to train the basic large language model after expanding the original vocabulary to obtain a target vertical domain large language model, including: continuing to pre-train the basic large language model after expanding the original vocabulary according to the target vertical domain text data and the general domain text data; and performing supervised fine-tuning on the basic large language model after continued pre-training according to the target vertical domain text data and the general domain text data to obtain the target vertical domain large language model.
[0012] According to a vertical domain large language model training method provided by the present invention, the basic large language model after the original vocabulary is expanded is continued to be pre-trained according to the target vertical domain text data and the general domain text data, including: generating a first training data set according to the target vertical domain text data and the general domain text data; using the first training data set, the basic large language model after the original vocabulary is expanded is continued to be pre-trained; the basic large language model after the continued pre-training is supervised fine-tuned according to the target vertical domain text data and the general domain text data, including: generating a second training data set according to the target vertical domain text data and the general domain text data; using the second training data set, the basic large language model after the continued pre-training is supervised fine-tuned.
[0013] The present invention also provides a training device for a vertical field large language model, comprising the following modules: an acquisition module, used to acquire target vertical field text data; an extraction module, used to extract frequently occurring adjacent character pairs from the target vertical field text data to obtain a target vertical field vocabulary; an expansion module, used to use the target vertical field vocabulary to expand the original vocabulary of a basic large language model; a training module, used to continue training the basic large language model after the original vocabulary is expanded to obtain a target vertical field large language model, so as to use the target vertical field large language model to generate answers to target vertical field questions.
[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the training method of a large vertical field language model as described in any one of the above-mentioned methods is implemented.
[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for training a large vertical field language model as described in any one of the above.
[0016] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for training a large vertical field language model.
[0017] The training method, device and electronic device of the vertical field large language model provided by the present invention obtain a target vertical field vocabulary by extracting frequently occurring adjacent character pairs from the target vertical field text data; using the target vertical field vocabulary, the original vocabulary of the basic large language model is expanded; so that the basic large language model after the expanded vocabulary can fully recognize and understand the annotated training corpus obtained from the target vertical field text data, and then can train a large language model with good performance in a specific vertical field based on limited annotated data. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0019] Figure 1 It is a flowchart of the training method of the vertical field large language model provided by the present invention.
[0020] Figure 2 It is a flowchart of the method for acquiring the target vertical field vocabulary provided by the present invention.
[0021] Figure 3 It is a flowchart of the training method of the target vertical field large language model provided by the present invention.
[0022] Figure 4 It is a structural schematic diagram of the training device for the vertical domain large language model provided by the present invention.
[0023] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0025] Combine the following Figure 1-Figure 3 The present invention describes the training method of the vertical domain large language model.
[0026] Figure 1 It is a flowchart of the training method of the vertical field large language model provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 101: Obtain target vertical field text data.
[0027] The target vertical field may be one of the vertical fields, for example, the environmental protection field, the medical field, etc. As an example only, the target vertical field may be the ecology field.
[0028] The target vertical field text data may include, but are not limited to, web page texts, professional books, and standard specifications in the vertical field. As an example only, the target vertical field text data may include academic papers, technical reports, clinical trial records, and industry standards in the field of ecology.
[0029] In the specific implementation process, text data in vertical fields can be collected from various channels using common methods (for example, crawler technology, API technology, etc.), which is not limited by the description of this specification.
[0030] After collecting text data in the target vertical field, in order to improve the quality and relevance of the data and ensure the effectiveness of subsequent model training, the collected text data needs to be cleaned.
[0031] The data cleaning process can use natural language processing technology, including automated text parsing tools and AI-driven content filtering algorithms. By accurately removing noise information in the text, such as advertisements, irrelevant hyperlinks, and duplicate content, while ensuring that all data complies with data protection regulations, especially when processing text containing sensitive information, strict privacy protection measures are taken. The specific process is as follows: 1) Remove irrelevant characters
[0032] In this step, an automated text parsing tool is used to scan and identify all irrelevant characters in the text data. For example, HTML tags, which are usually formatting elements left over when data is extracted from web page text. Another example is special symbols such as emoticons or non-standard text formatting symbols. Another example is advertising inserts, which are common in data obtained from online media or business documents. Through specially developed algorithms, all irrelevant characters in the target vertical field text data can be identified and removed, thereby improving the purity and usability of the data. For example, HTML is parsed and stripped from the target vertical field text data, retaining the pure text content; advertising text is identified and replaced or deleted.
[0033] 2) Remove sensitive information
[0034] In order to protect personal privacy and comply with data use policies, special attention needs to be paid to identifying and removing sensitive information from the data, including but not limited to personally identifiable information (such as name, address, email address, etc.) and copyrighted text. Using pattern recognition and text matching technology, potential sensitive content can be detected and anonymized or completely deleted to comply with legal and ethical requirements. In addition, a manual review process is introduced to conduct a secondary check on sensitive information that may be missed by the algorithm to ensure data compliance.
[0035] 3) Remove redundant text
[0036] In this step, advanced text comparison algorithms are used to identify and remove duplicate documents or text paragraphs. This is to avoid overfitting the model to a specific expression or style during the training process, thereby improving the generalization ability of the model. By comparing the hash values of the text or using natural language processing technology to evaluate the similarity between texts, duplicate or highly similar content can be effectively identified and removed. For example, using locality sensitive hashing technology, duplicate content can be efficiently detected and processed in large-scale data sets.
[0037] The data cleaning process described above not only improves the quality and relevance of the data, but also ensures the effectiveness and security of subsequent model training. This comprehensive cleaning strategy, combined with the application of automated tools and artificial intelligence algorithms, enables the trained target vertical domain large language model to show higher performance and adaptability when processing real-world data.
[0038] Step 102: extract frequently occurring adjacent character pairs from the target vertical field text data to obtain a target vertical field vocabulary.
[0039] The vocabulary is a tool used in the large language model to map words in the text into a computer-understandable format (such as a digital ID). The vocabulary provides a unified vocabulary index for the large language model, enabling the large language model to process and understand human language. The vocabulary also provides a basis for data annotation, ensuring that the words in the training corpus can be correctly recognized and processed by the large language model.
[0040] In the specific implementation process, frequently occurring adjacent character pairs can be extracted from the target vertical field text data to obtain the target vertical field vocabulary in a variety of ways, which are not limited to the description of this specification.
[0041] For an embodiment of extracting frequently occurring adjacent character pairs from target vertical field text data to obtain a target vertical field vocabulary, see Figure 2 The relevant description in will not be repeated here.
[0042] Step 103: Use the target vertical domain vocabulary to expand the original vocabulary of the basic large language model.
[0043] During the specific implementation process, in order to enable the basic large language model to recognize the words in the target vertical domain vocabulary, enable the newly added words to the basic large language model to quickly adapt and effectively learn in the model, and reduce the invalid annotation data caused by the unknown semantics of the words, it is also necessary to obtain the initial vector representation of the words in the target vertical domain vocabulary.
[0044] In the semantic space, words with similar vector representations can share similar semantic properties. Therefore, in some embodiments, the following operations can be performed for each word in the target vertical domain vocabulary to obtain an initial vector representation of each word, providing a reasonable initial semantic positioning for each word.
[0045] A plurality of words whose similarity with the word is greater than a preset threshold are obtained from the original word list of the basic large language model, and an average value of the initial vector representations of the plurality of words is calculated; and the average value of the initial vector representations of the plurality of words is used as the initial vector representation of the word.
[0046] In the specific implementation process, multiple words whose similarity with any word in the target vertical field word list is greater than a preset threshold can be obtained from the original word list of the basic large language model in a variety of ways, which are not limited to the description of this specification.
[0047] As an example only, the UTF-8 encoding representation of any word in the target vertical domain vocabulary can be calculated. After obtaining the UTF-8 encoding representation corresponding to the word, words with similar UTF-8 representations can be found from the original vocabulary of the basic large language model, and they are regarded as multiple words whose similarity with the word is greater than a preset threshold. In the specific implementation process, since the similarity calculation process involves the weighted sum operation of vectors in high-dimensional space, it is usually performed by a high-performance computing platform to ensure the accuracy and efficiency of the operation.
[0048] After obtaining the initial vector representation of each word in the target vertical domain vocabulary, it is added to the original vocabulary of the basic large language model.
[0049] In the specific implementation process, the target vertical domain vocabulary can be added to the end of the original vocabulary of the basic large language model. After that, the total number of words in the updated vocabulary of the basic large language model needs to be counted to ensure that the structure of the expanded vocabulary of the basic large language model is compatible with the architecture of the basic large language model, especially in the encoding and decoding process involving the basic large language model, to ensure data consistency and processing efficiency.
[0050] In large language models, the size of the vocabulary often needs to be aligned to a specific multiple to optimize computational efficiency, especially when using certain types of neural network processors or performing parallel computing, the size of the vocabulary needs to be divisible by 128. If the number of words counted is not divisible by 128, you need to append a special character "<|endoftext|>" to the end of the vocabulary, which is used for padding until the length of the vocabulary meets a multiple of 128. This special character is usually regarded as padding or blank in model training and application, and will not affect the processing of actual text data by the basic large language model.
[0051] Step 104: Continue training the basic large language model after expanding the original vocabulary to obtain a target vertical domain large language model, so as to use the target vertical domain large language model to generate answers to questions in the target vertical domain.
[0052] In the specific implementation process, the basic large language model after the original vocabulary is expanded can be further trained in a variety of ways to obtain a target vertical field large language model, which is not limited by the description of this specification.
[0053] For an embodiment of continuing to train the basic large language model after expanding the original vocabulary to obtain the target vertical field large language model, see Figure 3 The relevant content in will not be repeated here.
[0054] After training the target vertical domain large language model, the target vertical domain large language model can be used to generate answers to questions in the target vertical domain.
[0055] Just as an example, the question entered by the user is: How to carry out ecological protection?
[0056] Using the big language model of the target vertical field, we can get the following answer: Ecological protection is the key to ensuring harmonious coexistence between humans and nature. It involves protecting natural resources, maintaining biodiversity, and promoting sustainable development. The following are some basic guidelines for ecological protection actions: Adhere to ecological priority and green development: take ecological protection as a prerequisite for development and avoid damaging the ecological environment.
[0057] Implement major projects for ecosystem protection and restoration, such as the protection of the Three Rivers Source and the construction of national parks, to enhance ecological functions.
[0058] Develop eco-friendly industries: Promote green, low-carbon and circular development to achieve a win-win situation of economic and ecological benefits.
[0059] Through these actions, we can jointly protect our home planet and achieve harmonious coexistence between man and nature.
[0060] Figure 2 is a flow chart of the method for acquiring the target vertical domain vocabulary provided by the present invention, such as Figure 2 As shown, the method includes the following: Step 201: Decompose the target vertical field text data into individual characters to obtain a first character sequence.
[0061] In a specific implementation process, the acquired target vertical field text data may be decomposed into single characters, and all the single characters form a first character sequence.
[0062] Step 202: Filter out adjacent character pairs with the highest occurrence frequency from the first character sequence by performing character frequency statistics on the first character sequence.
[0063] In some embodiments, character frequency statistics may be performed on the first character sequence, and based on the statistical results, adjacent characters with the highest occurrence frequency may be filtered out from the first character sequence.
[0064] In the specific implementation process, a complex data mining algorithm, such as an n-gram model, can be used to identify and record the number of occurrences of each pair of adjacent characters in the text; then the recorded data is processed by a sorting algorithm to determine the adjacent character pairs whose number of occurrences is greater than a preset threshold. As an example, in an ecology text, the words "sea" and "ice" often appear together, so the adjacent character pair "sea ice" will be screened out.
[0065] In the specific implementation process, after a round of adjacent character pairs are screened out, the distribution and adjacent relationship of characters in the text of the vertical field will be changed accordingly. Therefore, the remaining characters in the first character sequence that have not been screened out can be re-decomposed into single characters, and the character frequency statistics of the re-decomposed single characters are continued. According to the statistical results, the adjacent character pairs with the highest frequency of occurrence are screened out. This step is repeated, and the adjacent character pairs with the highest frequency of occurrence are screened out in each round of iteration until the screening end condition is reached.
[0066] The end conditions of the screening may include: reaching a preset number of iterations, or until the new merger no longer brings about a significant improvement in the performance of the vocabulary. In the specific implementation process, the preset number of iterations may be determined by comprehensively considering the influence of various factors such as the size of the collected text data in the target vertical field, the professionalism of the target vertical field, and the performance requirements of the large language model in the target vertical field.
[0067] Step 203: construct a target vertical domain vocabulary based on all adjacent character pairs screened from the first character sequence.
[0068] After all adjacent character pairs are screened out from the first character sequence, it is necessary to combine linguistic knowledge and natural language processing technology to merge these adjacent character pairs into a new linguistically meaningful vocabulary.
[0069] As an example only, you can use pre-trained word embedding models (such as Word2Vec, GloVe, FastText, etc.) to obtain the vector representation of each character in the adjacent character pair, and use context-based word embedding models (such as BERT (Bidirectional Encoder Representations from Transformers, Transformer-based bidirectional encoder representation), GPT (Generative Pre-trained Transformer, Generative Pre-trained Transformer model), Transformer, etc.) to generate vector representations of adjacent character pairs. Based on the vector representation of each character in the adjacent character pair and the vector representation of the adjacent character pair, the contextual representation model (Contextual Models) is used to evaluate the semantic consistency between the new vocabulary corresponding to the adjacent character pair and the original character pair. If the two are semantically consistent, the adjacent character pair can be used as a new vocabulary and added to the target vertical domain vocabulary.
[0070] During the specific implementation process, terms that are commonly used in the vertical field but do not appear in the original model vocabulary can also be added to the target vertical field vocabulary.
[0071] In the embodiment provided by the present invention, the target vertical field text data is decomposed into individual characters to obtain a first character sequence; by performing character frequency statistics on the first character sequence, frequently occurring adjacent character pairs are screened out from the first character sequence; and a target vertical field vocabulary is constructed based on all adjacent character pairs screened out from the first character sequence. Since the characteristics of the vertical field text data are fully utilized, a professional target vertical field vocabulary that can accurately reflect the language characteristics of the target vertical field can be obtained. The target vertical field vocabulary can then be used to effectively improve the performance of the target vertical field large language model in the target vertical field.
[0072] Figure 3 It is a flowchart of the training method of the target vertical field large language model provided by the present invention, such as Figure 3 As shown, the method includes the following: 301. Based on the target vertical field text data and the acquired general field text data, continue pre-training the basic large language model after expanding the original vocabulary.
[0073] In the specific implementation process, general domain text data can be obtained from various channels using common methods (for example, crawler technology, API technology, etc.), without being limited by the description of this specification.
[0074] As an example only, general domain text data can be obtained from sources such as news, blogs, social media posts, and various types of literary works on the Internet. The wide range of data sources ensures the universality and comprehensiveness of semantic understanding of the trained large language model in the target vertical domain.
[0075] In the specific implementation process, a first training data set can be generated based on the target vertical field text data and the general field text data; using the first training data set, based on a preset optimization algorithm (e.g., stochastic gradient descent, AdamW, etc.), some parameters of the basic large language model after the original vocabulary is expanded are adjusted until the model training stop condition (e.g., the loss function converges) is reached, and the continued pre-training task of the basic large language model is completed.
[0076] In the specific implementation process, the training corpus in the first training data set can be generated according to the target vertical field text data and the general field text data in a common manner at a preset ratio (for example, 1:1), without being limited by the description of this specification.
[0077] In this step, the core parameters of the basic large language model are frozen, and only local parameters are updated to improve the domain adaptability of the basic large language model.
[0078] 302. Based on the target vertical domain text data and the general domain text data, supervised fine-tuning is performed on the pre-trained basic large language model to obtain the target vertical domain large language model.
[0079] In the specific implementation process, a second training data set can be generated based on the target vertical field text data and the general field text data; using the second training data set, based on the preset optimization algorithm (for example, stochastic gradient descent, AdamW, etc.), some parameters of the basic large language model after continued pre-training are adjusted. Until the model training stop condition is reached (for example, the loss function converges), the continued pre-training task of the basic large language model is completed.
[0080] In the specific implementation process, the training corpus in the second training data set can be generated according to the target vertical field text data and the general field text data in a common manner at a preset ratio (for example, 1:1), without being limited by the description of this specification.
[0081] In this step, the core parameters of the basic large language model are frozen, and the model performance is optimized only by precisely adjusting the specified parts to ensure that the trained large language model for the target vertical field can operate efficiently in the target vertical field.
[0082] The training device for the vertical field large language model provided by the present invention is described below. The training device for the vertical field large language model described below and the training method for the vertical field large language model described above can refer to each other.
[0083] Figure 4 is a schematic diagram of the structure of the training device for the vertical domain large language model provided by the present invention, such as Figure 4 As shown, the device 400 includes the following modules.
[0084] The acquisition module 410 is used to acquire target vertical field text data.
[0085] The extraction module 420 is used to extract frequently occurring adjacent character pairs from the target vertical field text data to obtain a target vertical field vocabulary.
[0086] The expansion module 430 is used to expand the original vocabulary of the basic large language model using the target vertical field vocabulary.
[0087] The training module 440 is used to continue training the basic large language model after the original vocabulary is expanded to obtain a target vertical field large language model, so as to use the target vertical field large language model to generate answers to questions in the target vertical field.
[0088] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the training method of the vertical field large language model, the method comprising: obtaining target vertical field text data; extracting frequently occurring adjacent character pairs from the target vertical field text data to obtain a target vertical field vocabulary; using the target vertical field vocabulary to expand the original vocabulary of the basic large language model; continuing to train the basic large language model after the original vocabulary is expanded to obtain the target vertical field large language model, so as to use the target vertical field large language model to generate answers to the target vertical field questions.
[0089] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0090] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the vertical field large language model training method provided by the above methods, and the method includes: obtaining target vertical field text data; extracting frequently occurring adjacent character pairs from the target vertical field text data to obtain a target vertical field vocabulary; using the target vertical field vocabulary to expand the original vocabulary of the basic large language model; continuing to train the basic large language model after the original vocabulary is expanded to obtain the target vertical field large language model, so as to use the target vertical field large language model to generate answers to target vertical field questions.
[0091] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the vertical field large language model training method provided by the above methods, the method comprising: obtaining target vertical field text data; extracting frequently occurring adjacent character pairs from the target vertical field text data to obtain a target vertical field vocabulary; using the target vertical field vocabulary to expand the original vocabulary of the basic large language model; continuing to train the basic large language model after the original vocabulary is expanded to obtain the target vertical field large language model, so as to use the target vertical field large language model to generate answers to target vertical field questions.
[0092] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0093] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A training method for a large language model in a vertical field, characterized in that: include: Obtain text data in the target vertical field; Extracting frequently occurring adjacent character pairs from the target vertical field text data to obtain a target vertical field vocabulary; Using the target vertical domain vocabulary, expanding the original vocabulary of the basic large language model; The basic large language model after the original vocabulary is expanded is further trained to obtain a target vertical field large language model, so as to use the target vertical field large language model to generate answers to questions in the target vertical field.
2. The method for training a large vertical language model according to claim 1, characterized in that: The step of extracting frequently occurring adjacent character pairs from the target vertical field text data to obtain a target vertical field vocabulary includes: Decomposing the target vertical field text data into individual characters to obtain a first character sequence; By performing character frequency statistics on the first character sequence, frequently occurring adjacent character pairs are screened out from the first character sequence; The target vertical domain vocabulary is constructed according to all adjacent character pairs screened from the first character sequence.
3. The method for training a large vertical language model according to claim 2, characterized in that: The step of performing character frequency statistics on the first character sequence to filter out frequently occurring adjacent character pairs from the first character sequence includes: Performing character frequency statistics on the first character sequence, and selecting adjacent character pairs with the highest occurrence frequency from the first character sequence according to the statistical results; The remaining characters in the first character sequence that have not been filtered out are re-decomposed into single characters, and character frequency statistics are continued for the re-decomposed single characters. Based on the statistical results, the adjacent character pairs with the highest occurrence frequency are filtered out, and this step is repeated until the filtering end condition is met.
4. The method for training a large vertical language model according to claim 1, characterized in that: The method of using the target vertical domain vocabulary to expand the original vocabulary of the basic large language model includes: For each word in the target vertical domain vocabulary, perform the following operations: Acquire multiple words from the original word list whose similarity to the word is greater than a preset threshold, and calculate an average value of the initial vector representations of the multiple words; Taking the average of the initial vector representations of the plurality of words as the initial vector representation of the word; The adjacent characters and their initial vector representations are added to the original vocabulary.
5. The method for training a large vertical domain language model according to any one of claims 1 to 4, characterized in that: The method further comprises: Obtain general domain text data; The step of continuing to train the basic large language model after expanding the original vocabulary to obtain a target vertical domain large language model includes: According to the target vertical field text data and the general field text data, continuing pre-training the basic large language model after expanding the original vocabulary; According to the target vertical field text data and the general field text data, supervised fine-tuning is performed on the basic large language model after continued pre-training to obtain the target vertical field large language model.
6. The method for training a large vertical language model according to claim 5, characterized in that: The step of continuing to pre-train the basic large language model after expanding the original vocabulary according to the target vertical field text data and the general field text data includes: Generate a first training data set according to the target vertical field text data and the general field text data; Using the first training data set, continuing pre-training the basic large language model after expanding the original vocabulary; The supervised fine-tuning of the pre-trained basic large language model according to the target vertical field text data and the general field text data includes: Generate a second training data set according to the target vertical field text data and the general field text data; The second training data set is used to perform supervised fine-tuning on the basic large language model after continued pre-training.
7. A training device for a large language model in a vertical field, characterized in that: include: The acquisition module is used to obtain text data in the target vertical field; An extraction module, used to extract frequently occurring adjacent character pairs from the target vertical field text data to obtain a target vertical field vocabulary; An expansion module, used to expand the original vocabulary of the basic large language model using the target vertical domain vocabulary; The training module is used to continue training the basic large language model after expanding the original vocabulary to obtain a target vertical field large language model, so as to use the target vertical field large language model to generate answers to questions in the target vertical field.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the training method of the vertical field large language model as described in any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the training method of the vertical domain large language model is implemented as described in any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the training method of the vertical domain large language model is implemented as described in any one of claims 1 to 6.
Citation Information
Patent Citations
Method and device for domain-specific vocabulary discovery, and method and device for domain-specific vocabulary classifier training
CN108846033A
Training corpus expansion method and device and intention recognition model training method and device
CN112651238A
Unsupervised adaptive domain term recognition method and system
CN116701628A
Intelligent auxiliary system based on large language model
CN116797417A
Banking industry large language model training method
CN117217289A
Cited By
Construction method and device of coal mine vertical field large model
CN119848553A
A Construction Method and Device for a Large Model in the Vertical Field of Coal Mines
CN119848553B
