Construction method and system of vertical domain large language model, electronic equipment and storage medium

Abstract

The invention discloses a construction method and system of a vertical domain large language model, electronic equipment and a storage medium, and the method comprises the steps: dividing a vertical domain knowledge specimen into background information and foreground information after constructing a limited number of vertical domain knowledge specimens; according to the method, background information and foreground information are acquired, background simulation is carried out based on the background information, foreground simulation is carried out based on the foreground information, a plurality of pieces of background simulation information and a plurality of pieces of foreground simulation information can be output respectively, and pairwise weaving fusion operation is carried out on all the background simulation information and all the foreground simulation information to generate a plurality of vertical domain simulation annotation documents. According to the method, massive vertical domain corpus data can be quickly generated in batches under the condition that the vertical domain corpus is limited so as to be used for fine tuning training of a general large language model, the vertical domain large language model obtained through training has higher knowledge understanding ability in the vertical domain, and the precision of vertical domain knowledge extraction questions and answers can be remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of large language model technology, and in particular, to a method and system for constructing a vertical domain large language model, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Currently, Large Language Models (LLMs) have been widely applied in various fields, such as natural language processing, where they can help computers better understand and generate text. However, current LLMs are mainly used for general question answering. Due to a lack of in-depth understanding and learning of vertical domain (hereinafter referred to as vertical domain) knowledge, their performance in tasks such as knowledge extraction and question answering in vertical domains still has room for improvement.

[0003] Existing technologies train general-purpose large language models by constructing a large corpus of language samples in a specific vertical domain, and then fine-tune these general-purpose large language models to construct vertical-domain large language models. For example, patent CN119938862A discloses a method for constructing an emergency question-answering large model, which trains a large language model by collecting a large amount of relevant corpus data in the emergency domain to obtain an emergency vertical-domain large language model. To ensure that the vertical-domain large language model has strong vertical domain understanding capabilities, a massive amount of vertical domain corpus data is usually required for fine-tuning the general-purpose large language model. However, in practical applications, collecting vertical domain corpus data is often difficult, resulting in poor vertical domain understanding capabilities of the large language model. Summary of the Invention

[0004] This invention provides a method and system for constructing a vertical domain large language model, an electronic device, and a computer-readable storage medium. It can quickly and in batches generate massive amounts of vertical domain corpus data after collecting a limited amount of vertical domain corpus data, providing massive training data for fine-tuning training of general large language models. This is beneficial to improving the training accuracy of general large language models and ensuring that the trained vertical domain large language model has a stronger knowledge understanding ability in the vertical domain.

[0005] According to one aspect of the present invention, a method for constructing a vertical domain large language model is provided, comprising the following:

[0006] Retrieve documents from multiple vertical domains;

[0007] Multiple vertical domain documents are annotated to form multiple vertical domain knowledge specimens. Each vertical domain knowledge specimen includes a text lexical sequence and a set of annotated entity information.

[0008] Knowledge simulation is performed based on each vertical domain knowledge specimen, generating several vertical domain simulation annotation documents;

[0009] The general large language model is trained based on vertical domain simulation labeled documents and vertical domain knowledge specimens to obtain a vertical domain large language model.

[0010] Furthermore, the process of generating several vertical domain simulation annotation documents based on each vertical domain knowledge specimen includes the following:

[0011] Vertical knowledge specimens are divided into foreground information and background information; foreground information refers to the labeled entity information in the vertical knowledge specimens, while background information refers to other information in the vertical knowledge specimens besides the labeled entity information.

[0012] Background simulation based on background information;

[0013] Foreground simulation based on foreground information;

[0014] The outputs of the background simulation and the foreground simulation are woven and merged to generate several vertical domain simulation annotation documents.

[0015] Furthermore, the process of performing background simulation based on background information specifically includes:

[0016] Randomly select an operation type from the preset operation types to perform background knowledge simulation. After each background knowledge simulation, output one background simulation message. Perform multiple background knowledge simulation operations on a vertical domain knowledge sample and output several background simulation messages.

[0017] Furthermore, the preset operation types include modification, addition, and deletion, and the background knowledge simulation process specifically includes:

[0018] Construct a probability distribution function for the background modification location of vertical domain knowledge specimens;

[0019] A target position is sampled from a text word sequence based on a background-modified position probability distribution function;

[0020] Background simulation is performed based on the selected operation type. When the selected operation type is "modify", a word is randomly sampled from the experience background knowledge base, and this word is used to replace the word at the target position. The target position and the replaced word are then output as background simulation information. When the selected operation type is "add", a word is randomly sampled from the experience background knowledge base, and this word is added to the word at the target position. The target position and the newly added word are then output as background simulation information. When the selected operation type is "delete", the word at the target position is deleted, and the target position is output as background simulation information.

[0021] Furthermore, the expression for the background modification location probability distribution function is:

[0022] ;

[0023] ;

[0024] in, This represents the probability that position t will be modified by the simulation, and Prob() represents the probabilistic output function. This represents the weight corresponding to the j-th labeled entity information. Let J represent the probability of modification of the j-th labeled entity information with respect to position t, where J represents the number of labeled entity information in the vertical domain knowledge sample. Indicates a normal distribution. This indicates the location center of the source segment for the j-th labeled entity information. It represents the standard deviation.

[0025] Furthermore, the process of performing foreground simulation based on foreground information includes the following:

[0026] Randomly select a labeled entity from the set of labeled entity information, regard the source fragment word position of the labeled entity information as the target position of the foreground simulation, randomly sample a similar entity from the empirical foreground knowledge base, and output the target position and the similar entity as foreground simulation information; repeat the above operation to output several pieces of foreground simulation information.

[0027] Furthermore, the process of weaving and fusing the outputs of the background simulation and the foreground simulation to generate several vertical domain simulation annotation documents includes the following:

[0028] Obtain one background simulation information from the background simulation output and one foreground simulation information from the foreground simulation output. Perform a weaving and fusion operation based on the two simulation information to generate a vertical simulation annotation document. Repeat the above process until all background simulation information and foreground simulation information have completed pairwise weaving and fusion operations to generate several vertical simulation annotation documents.

[0029] In addition, the present invention also provides a system for constructing a large vertical language model, comprising:

[0030] The vertical domain document retrieval module is used to retrieve multiple vertical domain documents;

[0031] The vertical domain knowledge specimen generation module is used to annotate multiple vertical domain documents to form multiple vertical domain knowledge specimens. Each vertical domain knowledge specimen includes a text word sequence and a set of annotated entity information.

[0032] The knowledge simulation module is used to perform knowledge simulation based on each vertical domain knowledge specimen and generate several vertical domain simulation annotation documents.

[0033] The vertical domain large language model training module is used to train a general large language model based on vertical domain simulation labeled documents and vertical domain knowledge specimens to obtain a vertical domain large language model.

[0034] In addition, the present invention also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and the processor executes the steps of the method described above by calling the computer program stored in the memory.

[0035] In addition, the present invention also provides a computer-readable storage medium for storing a computer program for constructing a large language model of a vertical domain, wherein the computer program executes the steps of the method described above when running on a computer.

[0036] The present invention has the following beneficial effects:

[0037] The method for constructing a vertical domain large language model in this invention, after constructing a limited number of vertical domain knowledge specimens, splits a vertical domain knowledge specimen into background information and foreground information, and then performs background simulation based on the background information and foreground simulation based on the foreground information respectively. This can output several background simulation information and several foreground simulation information respectively. Then, all the background simulation information and foreground simulation information are fused in pairs to generate several vertical domain simulation annotation documents. This method can quickly and in batches generate massive amounts of vertical domain corpus data when the vertical domain corpus is limited, so as to provide fine-tuning training for a general large language model. The vertical domain large language model obtained through training has a stronger knowledge understanding ability in the vertical domain and can significantly improve the accuracy of vertical domain knowledge extraction and question answering.

[0038] In addition, the system for constructing a vertical domain large language model of the present invention also has the above-mentioned advantages.

[0039] In addition to the objectives, features, and advantages described above, the present invention has other objectives, features, and advantages. The invention will now be described in further detail with reference to the figures. Attached Figure Description

[0040] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0041] Figure 1 This is a flowchart illustrating the method for constructing a vertical domain large language model according to a preferred embodiment of this application;

[0042] Figure 2 yes Figure 1 A schematic diagram of the sub-process of step S3;

[0043] Figure 3This is a schematic diagram of the module structure of a vertical domain large language model construction system according to another embodiment of this application. Detailed Implementation

[0044] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0045] The main application scenario of this application is: when a general large language model or other models perform poorly in a vertical domain knowledge analysis task, this technical solution can be used to construct a large vertical domain model to obtain a large model with stronger vertical domain understanding capabilities.

[0046] Reference Figure 1 As shown, a preferred embodiment of this application provides a method for constructing a large-scale language model for a vertical domain, including the following:

[0047] Step S1: Obtain multiple vertical domain documents.

[0048] It is understood that in step S1, multiple text document data of the target domain (i.e., vertical domain) are obtained. A vertical domain document consists of multiple lexical units and can be regarded as a lexical unit sequence. For example, the lexical unit sequence of a very short document describing a pet at home might be [I, home, have, only, shepherd, sheep, dog], where square brackets represent a sequence and commas represent lexical unit intervals. This document has a total of 7 lexical units. The specific method of obtaining vertical domain documents is existing technology and will not be elaborated here. For example, it can be obtained through manual collection or automatic crawling technology.

[0049] Step S2: Annotate multiple vertical domain documents to form multiple vertical domain knowledge specimens. Each vertical domain knowledge specimen includes a text lexical sequence and a set of annotated entity information.

[0050] It is understood that in step S2, the main purpose of this process is to annotate the acquired multiple vertical domain documents to form multiple annotated documents (i.e., vertical domain knowledge specimens). The input of this process is the acquired multiple documents, and the output is the corresponding annotated documents. The key information of a vertical domain knowledge specimen includes a text lexical sequence and an annotated entity information set. The text lexical sequence is the sequence of lexical elements that constitute the document. The annotated entity information set contains multiple annotated entity information. An annotated entity information set is a collection consisting of a source segment and the entity type information of that segment. A segment refers to a continuous and uninterrupted subsequence in the lexical sequence, which can be represented by the position tuple of the first and last lexical elements of the subsequence in the text lexical sequence. The entity type information is a preset enumerable type information, and one segment corresponds to one entity type. Taking the phrase [I, home, have, only, shepherd, sheep, dog] as an example, "I" is a segment with a start-end position tuple of (1,1), and the entity type is the preset "person". The three lexical units "shepherd, sheep, dog" can form a segment with a start-end position tuple of (5,7), and the corresponding entity type is the preset "pet". The labeled entity set is {(I, (1,1), person), (shepherd dog, (5,7), pet)}. Therefore, by labeling each vertical domain document, a vertical domain knowledge specimen can be obtained, and each vertical domain knowledge specimen includes a text lexical sequence and a set of labeled entity information. Vertical knowledge specimens are the vertical corpus data used to fine-tune the training of the general large language model. However, the amount of vertical corpus data at this time is relatively small, resulting in poor training accuracy of the model and poor knowledge understanding ability of the trained vertical large language model in the vertical domain. Therefore, it is necessary to generate massive amounts of vertical corpus data based on the labeled vertical knowledge specimens to improve the training accuracy of the general large language model and ensure that the trained vertical large language model has a strong knowledge understanding ability in the vertical domain.

[0051] Step S3: Perform knowledge simulation based on each vertical domain knowledge specimen to generate several vertical domain simulation annotation documents.

[0052] It can be understood that in step S3, knowledge simulation is the process of generating several vertical domain simulation annotation documents using vertical domain knowledge specimens. The knowledge simulation process mainly includes three sub-processes: background simulation, foreground simulation, and dual-scene weaving. For example... Figure 2 As shown, the process of generating several vertical domain simulation annotation documents based on each vertical domain knowledge specimen specifically includes:

[0053] Step S31: Divide the vertical domain knowledge specimen into foreground information and background information. Foreground information refers to the labeled entity information in the vertical domain knowledge specimen, and background information refers to other information in the vertical domain knowledge specimen besides the labeled entity information.

[0054] Step S32: Perform background simulation based on background information;

[0055] Step S33: Perform foreground simulation based on foreground information;

[0056] Step S34: Weave and fuse the outputs of the background simulation and the foreground simulation to generate several vertical domain simulation annotation documents.

[0057] Specifically, for a vertical domain knowledge specimen, the length of its text token sequence is denoted as T, and the text token sequence can be expressed as D = [w1, w2, …, w t , …, w T , where w t is a token, the number of labeled entity information is denoted as J, and the labeled entity information set can be expressed as: [E1, E2, …, E j , …, E J . The start and end positions of the source segment of a labeled entity information E j are denoted as , , and the entity type is denoted as .

[0058] First, in the step S31, the vertical domain knowledge specimen is divided into background information and foreground information. Among them, the foreground information refers to the labeled entity information in the vertical domain knowledge specimen, and the background information refers to the remaining information in the vertical domain knowledge specimen that does not contain labeled entity information. Taking [I, have, a, shepherd, dog] as an example, the token subsequence [shepherd, dog], the start and end position pair (5, 7), and the entity type "pet" are the foreground information, while the token "have" and the start and end position pair (3, 3) are the background information.

[0059] Secondly, in the step S32, background simulation is performed based on the divided background information. Among them, the process of performing background simulation based on background information includes the following content:

[0060] Randomly select an operation type from the preset operation types to perform background knowledge simulation. After each background knowledge simulation, output a background simulation information, and perform multiple background knowledge simulation operations on a vertical domain knowledge specimen to output several background simulation information.

[0061] Optionally, the preset operation types include modification, addition, and deletion. The process of background knowledge simulation specifically includes:

[0062] First, construct the background modification position probability distribution function of the vertical domain knowledge specimen, and its expression is:

[0063] ;

[0064] ;

[0065] Where t represents the position of a document term in the text term sequence. For a document with T terms, the value of t ranges from t=1,2,3,...,T. This indicates the probability that position t will be modified by the simulation; Prob() represents the probabilistic output function, for example, X represents the domain of f(x); This represents the weight corresponding to the j-th labeled entity information, which is a preset value and takes a value greater than or equal to 0; This represents the modification probability function based on the information of the j-th labeled entity with respect to position t. The modification probability function follows a normal distribution. It is derived from elementary function composite transformations based on the foundation; J represents the number of labeled entity information in the vertical domain knowledge specimen; Represents a normal distribution; This represents the location center (i.e., the midpoint between the start and end positions) of the source segment of the j-th labeled entity information. This represents the standard deviation, which is a preset value, and The first derivative must be ≥0, meaning that t moves further and further away from the first derivative. The location is at the center.

[0066] Understandably, in the knowledge simulation process of a large-scale vertical model, the purpose of constructing the probability distribution function of the background modification location for vertical knowledge samples is to specifically simulate the possibility of background modification at different locations in the text, thereby generating training samples that better conform to the characteristics of the domain and improving the knowledge understanding and generalization ability of the large model in the vertical domain. Specifically, the background modification location probability distribution function, by combining the positional information of labeled entities with weight control, constructs a modification preference model centered on entities and with decreasing probability, thus achieving guided and structured text simulation. Based on this function, high-quality data can be generated efficiently, thereby training a more robust, more generalized large-scale vertical model that truly understands core knowledge rather than simply memorizing text.

[0067] Then, a target position is sampled from the text word sequence based on the background modification position probability distribution function. Next, background simulation is performed according to the selected operation type; when the selected operation type is modification, a word is randomly sampled from the empirical background knowledge base, and this word is used to replace the word at the target position, outputting the target position and the replaced word as background simulation information; when the selected operation type is addition, a word is randomly sampled from the empirical background knowledge base, and this word is added to the word at the target position, outputting the target position and the newly added word as background simulation information; when the selected operation type is deletion, the word at the target position is deleted, and the target position is output as background simulation information.

[0068] Specifically, when the randomly selected operation type is modification, the probability distribution function of the modified position is first modified based on the background. Sample a target position from the text word sequence. Then, the probability distribution of the experiential background knowledge base is expressed as follows: A word is randomly sampled from the middle. , where the probability distribution represents This is obtained through statistical learning. Next, the lexical units at the target positions in the text lexical sequence of the vertical domain knowledge sample are replaced with new lexical units. This completes the modification operation, targeting the desired location. and alternative word elements As an output of the modification operation simulation process, it will be used in the subsequent weaving and fusion process.

[0069] Specifically, when the randomly selected operation type is "increase", the position probability distribution function is first modified based on the background. Sample a target position from the text word sequence. Then, the probability distribution of the experiential background knowledge base is expressed as follows: A word is randomly sampled from the middle. , will new word elements Add to target location The addition operation is completed by placing the word after the target word, thus achieving the desired position. and new words As an output of the enhanced operational simulation process, it will be used in subsequent weaving and fusion processes.

[0070] Specifically, when the randomly selected operation type is deletion, the position probability distribution function is first modified based on the background. Sample a target position from the text word sequence. Delete text word sequence The word at the target location is used to complete the deletion operation. As an output of the deletion operation simulation process, it will be used in the subsequent weaving and fusion process.

[0071] It is understandable that performing the above background knowledge simulation operation multiple times on a vertical domain knowledge sample can output several pieces of background simulation information. Optionally, the number of times the background knowledge simulation operation is performed on a vertical domain knowledge sample is determined by the number of lexical units in the background information, specifically based on the following formula: K=r bgs ×N bgt Where K represents the number of simulations for background knowledge; r bgs N represents the correction factor, which is a preset value; bgt This indicates the number of tokens in the background information.

[0072] Furthermore, in step S33, the process of performing foreground simulation based on foreground information includes the following:

[0073] Randomly select a labeled entity from the set of labeled entity information, regard the source fragment word position of the labeled entity information as the target position of the foreground simulation, randomly sample a similar entity from the empirical foreground knowledge base, and output the target position and the similar entity as foreground simulation information; repeat the above operation to output several pieces of foreground simulation information.

[0074] Specifically, firstly, a labeled entity is randomly selected from the set of labeled entity information, denoted as . The source segment word position is considered as the target position for foreground simulation; then, the sampling function from the empirical foreground knowledge base... A similar entity is randomly sampled from the sample, denoted as . , where the sampling function Based on statistical learning, this function takes a sample entity as input and outputs a similar entity, including its corresponding word sequence and entity type. Finally, the target position and the similar entity are output together. This serves as foreground simulation information. Repeating the above operation will output several foreground simulation information entries.

[0075] Finally, in step S34, the process of weaving and fusing the outputs of the background simulation and the foreground simulation to generate several vertical domain simulation annotation documents includes the following:

[0076] Obtain one background simulation information from the background simulation output and one foreground simulation information from the foreground simulation output. Perform a weaving and fusion operation based on the two simulation information to generate a vertical simulation annotation document. Repeat the above process until all background simulation information and foreground simulation information have completed pairwise weaving and fusion operations to generate several vertical simulation annotation documents.

[0077] Specifically, both background and foreground simulations output several simulation information entries. A single weaving process combines one background simulation entry with one foreground simulation entry to form a vertical domain simulation annotation document. The weaving and fusion process is as follows: The two target positions in the background and foreground simulation information are executed sequentially. Positions in the original document (i.e., the vertical domain knowledge specimen) that do not correspond to the two target positions are considered retained positions. For retained positions, the corresponding lexical units are directly copied to the new text lexical sequence. For the target positions in the background simulation information, if the background simulation operation randomly selects a modification operation, the target positions in the new text lexical sequence are replaced with alternative lexical units. If the background simulation operation is randomly selected as the add operation, then the word at the target position in the original text word sequence is copied to the new text word sequence and the new word is added after it. If the background simulation operation randomly selects a deletion operation, then the word at the target position in the original text word sequence will not be copied to the new text word sequence; however, for the target position in the foreground simulation information, the word subsequence at the target position in the original text word sequence will be replaced with a similar entity. The word sequence is replaced and added to the new text word sequence, and the entity type at the target position is replaced with a similar entity. The entity type is determined; after one weaving and fusion process, a new text lexical sequence and a new set of labeled entity information are output, i.e., a vertical domain simulation labeling document is output. The above weaving and fusion process is repeated until all background simulation information and foreground simulation information have completed pairwise weaving and fusion operations, thereby generating several vertical domain simulation labeling documents.

[0078] Step S4: Train the general large language model based on the vertical domain simulation labeled documents and vertical domain knowledge specimens to obtain the vertical domain large language model.

[0079] Specifically, by using the limited number of vertical domain knowledge samples obtained from step S2 and the massive number of vertical domain simulation labeled documents generated through knowledge simulation in step S3, the text word sequence and labeled entity information of the documents are converted into questions and answers of the large model, respectively. Using these as training data, a general large language model can be trained, resulting in a vertical domain large language model with stronger knowledge understanding capabilities in the vertical domain.

[0080] It is understood that the method for constructing a vertical domain large language model in this invention, after constructing a limited number of vertical domain knowledge specimens, splits a vertical domain knowledge specimen into background information and foreground information, and then performs background simulation based on the background information and foreground simulation based on the foreground information respectively. This can output several background simulation information and several foreground simulation information respectively. Then, the background simulation information and foreground simulation information are fused together in pairs to generate several vertical domain simulation annotation documents. This method can quickly and in batches generate massive amounts of vertical domain corpus data when the vertical domain corpus is limited, so as to provide fine-tuning training for a general large language model. The vertical domain large language model obtained through training has a stronger knowledge understanding ability in the vertical domain and can significantly improve the accuracy of vertical domain knowledge extraction and question answering.

[0081] In addition, such as Figure 3 As shown, another embodiment of the present invention also provides a system for constructing a vertical domain large language model, preferably employing the method for constructing a vertical domain large language model as described above, including:

[0082] The vertical domain document retrieval module is used to retrieve multiple vertical domain documents;

[0083] The vertical domain knowledge specimen generation module is used to annotate multiple vertical domain documents to form multiple vertical domain knowledge specimens. Each vertical domain knowledge specimen includes a text word sequence and a set of annotated entity information.

[0084] The knowledge simulation module is used to perform knowledge simulation based on each vertical domain knowledge specimen and generate several vertical domain simulation annotation documents.

[0085] The vertical domain large language model training module is used to train a general large language model based on vertical domain simulation labeled documents and vertical domain knowledge specimens to obtain a vertical domain large language model.

[0086] It is understood that the vertical domain large language model construction system of the present invention, after constructing a limited number of vertical domain knowledge specimens, splits a vertical domain knowledge specimen into background information and foreground information, and then performs background simulation based on the background information and foreground simulation based on the foreground information respectively. It can output several background simulation information and several foreground simulation information respectively. Then, it performs pairwise weaving and fusion operations on all the background simulation information and foreground simulation information to generate several vertical domain simulation annotation documents. It can quickly and in batches generate massive amounts of vertical domain corpus data when the vertical domain corpus is limited, so as to provide fine-tuning training for the general large language model. The vertical domain large language model obtained through training has a stronger knowledge understanding ability in the vertical domain and can significantly improve the accuracy of vertical domain knowledge extraction and question answering.

[0087] In addition, another embodiment of the present invention provides an electronic device including a processor and a memory, wherein the memory stores a computer program, and the processor executes the steps of the method described above by calling the computer program stored in the memory.

[0088] In addition, another embodiment of the present invention provides a computer-readable storage medium for storing a computer program for constructing a large language model of a vertical domain, wherein the computer program performs the steps of the method described above when running on a computer.

[0089] Common computer-readable storage media include: floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tape, any other physical media with perforated patterns, random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), flash erasable programmable read-only memory (FLASH-EPROM), any other memory chips or cartridges, or any other media readable by a computer. Instructions may further be transmitted or received by a transmission medium. The term transmission medium can include any tangible or intangible medium used to store, encode, or carry instructions for execution by a machine, and includes digital or analog communication signals or intangible media that facilitate communication of such instructions. Transmission media include coaxial cables, copper wires, and optical fibers, which contain conductors for transmitting a bus of computer data signals.

[0090] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0091] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0092] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0093] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0094] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0095] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

[0096] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for constructing a large-scale vertical language model, characterized in that, Includes the following: Retrieve documents from multiple vertical domains; Multiple vertical domain documents are annotated to form multiple vertical domain knowledge specimens. Each vertical domain knowledge specimen includes a text lexical sequence and a set of annotated entity information. Knowledge simulation is performed based on each vertical domain knowledge specimen, generating several vertical domain simulation annotation documents; The general large language model is trained based on vertical domain simulation labeled documents and vertical domain knowledge specimens to obtain a vertical domain large language model.

2. The method for constructing a large vertical language model as described in claim 1, characterized in that, The process of generating several vertical domain simulation annotation documents based on each vertical domain knowledge specimen includes the following: Vertical knowledge specimens are divided into foreground information and background information; foreground information refers to the labeled entity information in the vertical knowledge specimens, while background information refers to other information in the vertical knowledge specimens besides the labeled entity information. Background simulation based on background information; Foreground simulation based on foreground information; The outputs of the background simulation and the foreground simulation are woven and merged to generate several vertical domain simulation annotation documents.

3. The method for constructing a vertical domain large language model as described in claim 2, characterized in that, The process of background simulation based on background information specifically includes: Randomly select an operation type from the preset operation types to perform background knowledge simulation. After each background knowledge simulation, output one background simulation message. Perform multiple background knowledge simulation operations on a vertical domain knowledge sample and output several background simulation messages.

4. The method for constructing a vertical domain large language model as described in claim 3, characterized in that, The preset operation types include modification, addition, and deletion. The background knowledge simulation process specifically includes: Construct a probability distribution function for the background modification location of vertical domain knowledge specimens; A target position is sampled from a text word sequence based on a background-modified position probability distribution function; Background simulation is performed based on the selected operation type. When the selected operation type is "modify", a word is randomly sampled from the experience background knowledge base, and this word is used to replace the word at the target position. The target position and the replaced word are then output as background simulation information. When the selected operation type is "add", a word is randomly sampled from the experience background knowledge base, and this word is added to the word at the target position. The target position and the newly added word are then output as background simulation information. When the selected operation type is "delete", the word at the target position is deleted, and the target position is output as background simulation information.

5. The method for constructing a vertical domain large language model as described in claim 4, characterized in that, The expression for the probability distribution function of the background modification location is: ； ； in, This represents the probability that position t will be modified by the simulation, and Prob() represents the probabilistic output function. This represents the weight corresponding to the j-th labeled entity information. Let J represent the probability of modification of the j-th labeled entity information with respect to position t, where J represents the number of labeled entity information in the vertical domain knowledge sample. Indicates a normal distribution. This indicates the location center of the source segment for the j-th labeled entity information. It represents the standard deviation.

6. The method for constructing a large vertical language model as described in claim 2, characterized in that, The process of performing foreground simulation based on foreground information includes the following: Randomly select a labeled entity from the set of labeled entity information, regard the source fragment word position of the labeled entity information as the target position of the foreground simulation, randomly sample a similar entity from the empirical foreground knowledge base, and output the target position and the similar entity as foreground simulation information; repeat the above operation to output several pieces of foreground simulation information.

7. The method for constructing a large vertical language model as described in claim 2, characterized in that, The process of weaving and fusing the outputs of the background simulation and the foreground simulation to generate several vertical domain simulation annotation documents includes the following: Obtain one background simulation information from the background simulation output and one foreground simulation information from the foreground simulation output. Perform a weaving and fusion operation based on the two simulation information to generate a vertical simulation annotation document. Repeat the above process until all background simulation information and foreground simulation information have completed pairwise weaving and fusion operations to generate several vertical simulation annotation documents.

8. A system for constructing a large-scale vertical language model, characterized in that, include: The vertical domain document retrieval module is used to retrieve multiple vertical domain documents; The vertical domain knowledge specimen generation module is used to annotate multiple vertical domain documents to form multiple vertical domain knowledge specimens. Each vertical domain knowledge specimen includes a text word sequence and a set of annotated entity information. The knowledge simulation module is used to perform knowledge simulation based on each vertical domain knowledge specimen and generate several vertical domain simulation annotation documents. The vertical domain large language model training module is used to train a general large language model based on vertical domain simulation labeled documents and vertical domain knowledge specimens to obtain a vertical domain large language model.

9. An electronic device, characterized in that, The method includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the steps of the method as described in any one of claims 1 to 7 by calling the computer program stored in the memory.

10. A computer-readable storage medium for storing a computer program for constructing a large-scale language model of a vertical domain, characterized in that, The computer program, when run on a computer, performs the steps of the method as described in any one of claims 1 to 7.

Images