Semantic large model increment pre-training method and related device

Abstract

The invention belongs to the field of artificial intelligence, and discloses a large semantic model increment pre-training method and a related device, and the method comprises the steps: obtaining a professional field increment training sample, a large semantic model and a large semantic model pre-training sample; taking the large semantic model as a generator and a discriminator to construct a generative adversarial network, and training the generative adversarial network based on the professional field incremental training sample and the large semantic model pre-training sample; and obtaining a generator in the trained generative adversarial network to obtain a professional field increment pre-training large model. The risk of catastrophic forgetting when the large semantic model fits the incremental training sample in the professional field can be effectively avoided, and the problem of model collapse is also effectively avoided. Incremental training can be completed by using small computing power, samples and time cost on the basis of data in the professional field, professional terms, knowledge structures and business logic in the professional field can be better learned and adapted by means of the understanding ability of the large semantic model, and the application effect in the professional field is improved.

Description

Technical Field

[0001] The invention belongs to the field of artificial intelligence and relates to a semantic large model incremental pre-training method and related devices. Background Art

[0002] At present, although the mainstream open source big models are pre-trained on tens of trillions of characters, they are usually general-purpose and not specifically targeted at specific industries, such as the power industry. The power industry has its own unique terminology, processes, equipment, regulations, and business scenarios, which may not be fully covered in open source big models. Specifically, a typical application of big models in the power sector is intelligent customer service. Users may ask: "My meter shows a fault, what should I do?" In order to answer such questions, intelligent customer service based on big models needs to have certain knowledge of the power industry, which requires further professional field incremental pre-training of the big model, that is, training on text data from the power industry on the basis of general pre-training.

[0003] There are three mainstream methods for introducing professional knowledge and private domain data into semantic big models, namely re-pre-training, incremental pre-training, and external knowledge base based on retrieval enhancement. Among them, the cost of re-pre-training is huge. Pre-training a big model requires extremely high computing resources, manpower and time costs, and requires an investment of tens of billions or even hundreds of billions of yuan. In addition, the timeliness is low. As time goes by, the private domain data of the industry will change. The training cycle of the big model is too long and does not support high-frequency updates of knowledge and information. There is error propagation in the external knowledge base based on retrieval enhancement. The performance of this solution is highly dependent on the quality of retrieval recall. If the retrieved information is inaccurate or irrelevant, it will directly lead to a decline in the quality of generated content. The big model does not actually grasp the industry knowledge structure, nor does it understand the business logic, resulting in limited improvement capabilities in some logical reasoning business scenarios.

[0004] Incremental pre-training is widely used as a relatively good implementation method. However, incremental pre-training also has the following two problems. The first is catastrophic forgetting. Directly starting incremental training cannot effectively combine the newly learned knowledge with the original knowledge, and may even cause the large model to directly forget the original knowledge. The second is model collapse. If the large model is directly fitted with professional knowledge and private domain data, the performance of the large model will drop sharply during the training process, or it will be unable to continue to learn new information. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a semantic large model incremental pre-training method and related devices.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a semantic large model incremental pre-training method, comprising:

[0008] Obtain incremental training samples, semantic large models, and semantic large model pre-training samples in professional fields;

[0009] The semantic big model is used as the generator and discriminator to build a generative adversarial network, and the generative adversarial network is trained based on incremental training samples in professional fields and pre-trained samples of the semantic big model.

[0010] Get the generator in the trained generative adversarial network and obtain a large incremental pre-training model for professional fields.

[0011] Optionally, the training of the generative adversarial network based on the incremental training samples in the professional field and the pre-trained samples of the semantic large model includes:

[0012] The discriminator training steps and the generator training steps are performed alternately, based on the incremental training samples of professional fields and the pre-trained samples of the semantic large model, and the adversarial generative network is trained according to the optimized objective function;

[0013] Among them, the optimization objective function is:

[0014]

[0015] Among them, SGD is the discriminator, SGG is the generator, L(L SGD , L SGG ) is the adversarial function to measure the adversarial nature between the generator and the discriminator; L SGD is the discriminator loss function, L SGG is the generator loss function;

[0016] The discriminator training steps include: fixing the generator parameters, based on the incremental training samples in the professional field and the pre-training samples of the semantic large model, by maximizing L(L SGD , L SGG ) to train the discriminator;

[0017] The generator training steps include: fixing the discriminator parameters, based on the incremental training samples of the professional field and the pre-training samples of the semantic large model, by minimizing L(L SGD , L SGG ) to train the discriminator.

[0018] Optionally, the discriminator loss function L SGD for:

[0019]

[0020] Where E represents the expected value, SGD(x) is the predicted probability of the discriminator for the true sample x, SGD(SGG(z)) is the predicted probability of the discriminator for the generator's generated sample SGG(z), z is the input noise of the generator, and p data (x) is the distribution of the real samples, p z (z) is the distribution of input noise; the real sample x includes incremental training samples in professional fields and pre-training samples of semantic large models;

[0021] The generator loss function L SGG for:

[0022]

[0023] in, To generate adversarial loss; is the next sample tag prediction loss of the semantic large model, T is the sample length, x t is the t-th sample label, P is the conditional probability, θ is the generator parameter; λ is the balance and Hyperparameters of .

[0024] Optionally, the number of the professional field incremental training samples and the number of the semantic large model pre-training samples are the same.

[0025] Optionally, it also includes: obtaining the discriminator in the trained adversarial generative network to obtain a detection model of the professional field incremental pre-trained large model; wherein the detection model is used to obtain the confidence of the output of the professional field incremental pre-trained large model based on the output of the professional field incremental pre-trained large model.

[0026] Optionally, the professional field incremental training samples are incremental training samples in the power field; the incremental training samples in the power field are obtained through one or more of the following: technical manuals, operating procedures, maintenance guides, customer service records and legal and regulatory texts in the power industry; the professional field incremental pre-training big model is a power semantic big model.

[0027] A second aspect of the present invention provides a semantic large model incremental pre-training system, comprising:

[0028] Data acquisition module, used to obtain professional field incremental training samples, semantic large models and semantic large model pre-training samples;

[0029] The incremental training module is used to construct a generative adversarial network using the semantic big model as a generator and a discriminator, and to train the generative adversarial network based on incremental training samples in professional fields and pre-trained samples of the semantic big model;

[0030] The result output module is used to obtain the generator in the trained adversarial generative network and obtain a large incremental pre-training model in a professional field.

[0031] Optionally, the result output module is also used to obtain the discriminator in the trained adversarial generative network to obtain a detection model of the professional field incremental pre-trained large model; wherein the detection model is used to obtain the confidence of the output of the professional field incremental pre-trained large model based on the output of the professional field incremental pre-trained large model.

[0032] According to a third aspect of the present invention, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned semantic large model incremental pre-training method when executing the computer program.

[0033] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned semantic large model incremental pre-training method are implemented.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] The semantic big model incremental pre-training method of the present invention realizes the incremental pre-training of the semantic big model by constructing a generative adversarial network using the semantic big model as a generator and a discriminator, and training the generative adversarial network based on incremental training samples in professional fields and pre-trained samples of the semantic big model. Training by constructing a generative adversarial network can effectively avoid the risk of catastrophic forgetting of the semantic big model when fitting incremental training samples in professional fields; at the same time, training based on incremental training samples in professional fields and pre-trained samples of the semantic big model can also effectively avoid the problem of model collapse. This method does not require retraining of the semantic big model, and can be directly based on the open source semantic big model, using less computing power, samples and time costs to complete incremental training based on data in professional fields. It can use the understanding ability of the semantic big model itself to better learn and adapt to professional terminology, knowledge structure and business logic in professional fields, and improve the application effect in professional fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of the incremental pre-training method for a large semantic model according to an embodiment of the present invention.

[0037] Figure 2 Schematic diagram of using real samples to train a discriminator in an embodiment of the present invention.

[0038] Figure 3 A schematic diagram of a discriminator for training using generated samples according to an embodiment of the present invention.

[0039] Figure 4 Schematic diagram of training a generator using feedback results of a discriminator according to an embodiment of the present invention.

[0040] Figure 5 It is a schematic diagram of the application of the electric power semantic big model in the electric power technical standard question and answer system based on the retrieval enhancement of the electric power semantic big model according to an embodiment of the present invention.

[0041] Figure 6 This is a structural block diagram of the semantic large model incremental pre-training system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0043] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0044] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0045] See also Figure 1 In one embodiment of the present invention, a semantic big model incremental pre-training method is provided, which can effectively overcome the catastrophic forgetting problem and model collapse problem of incremental pre-training, and can realize the incremental pre-training of the open source general semantic big model in the power field to obtain a power semantic big model that can adapt to the power field.

[0046] Specifically, the semantic large model incremental pre-training method of the present invention comprises the following steps:

[0047] S1: Obtain professional domain incremental training samples, semantic big models, and semantic big model pre-training samples.

[0048] S2: Use the semantic big model as the generator and discriminator to build a generative adversarial network, and train the generative adversarial network based on incremental training samples in professional fields and pre-trained samples of the semantic big model.

[0049] S3: Get the generator in the trained generative adversarial network to obtain a large incremental pre-training model in a professional field.

[0050] As described in the background technology, there are three mainstream methods for introducing professional knowledge and private domain data into semantic big models, namely, re-pre-training, incremental pre-training, and retrieval-enhanced plug-in knowledge base.

[0051] Among them, the steps of re-pretraining are roughly as follows: 1. Mix the incremental samples (2 billion characters) with the original pre-trained samples (20 trillion characters) in a certain ratio. 2. On the Wanka cluster, start the re-pretraining of the semantic big model for 2-4 months. 3. Complete the evaluation of general knowledge and professional knowledge. 4. Deploy the application. The advantages of this solution are: the semantic big model can better understand and adapt to the professional terminology, knowledge structure and business logic of a specific industry, thereby improving the application effect in the industry. In addition, the industry private domain data is often closely related to the business goals. Incorporating these data into the pre-training process helps the semantic big model learn features that are more relevant to the business goals, and improves the practicality and accuracy of the semantic big model. However, the cost of this solution is huge. Pre-training a new semantic big model requires extremely high computing resources, manpower and time costs, requiring an investment of tens of billions or even hundreds of billions of yuan. In addition, the timeliness is low. As time goes by, the industry private domain data will change. The training cycle of the semantic big model is too long, and it does not support high-frequency updates of knowledge and information.

[0052] The steps of incremental pre-training are roughly as follows: 1. Use incremental samples (2 billion characters) as data for incremental training. 2. On the Baika cluster, start incremental training of the large model for 2-4 days. 3. Complete the evaluation of general knowledge and professional knowledge. 4. Deploy the application. The advantages of this solution are: Since the semantic large model has been pre-trained on trillions of characters and has language understanding and knowledge foundation, compared to training a brand new semantic large model from scratch, this solution can quickly adapt to new data, does not require a large amount of industry data to learn the domain knowledge structure and business logic, and requires less data volume, computing resources and time cost. However, this solution has the problems of catastrophic forgetting and model collapse. Among them, catastrophic forgetting means that directly starting incremental training cannot effectively combine the newly learned knowledge with the original knowledge, and even directly forgets the original knowledge. Model collapse means that if the semantic large model is directly fitted to the industry private domain data, the performance of the semantic large model will drop sharply during the training process, or it will be unable to continue to learn new information.

[0053] The steps of the plug-in knowledge base based on retrieval enhancement are roughly as follows: 1. Establish an inverted index for incremental samples and import them into the vector retrieval library. 2. Establish a text recall and fine ranking model to facilitate the recall of relevant context from the retrieval library. 3. Establish a tool chain: user input → retrieval library recall → recall result fine ranking → jointly input the large model. 4. Deployment and application. The advantages of this solution are: allowing the semantic large model to refer to a large amount of domain expertise and private domain data when generating content, which can significantly improve the quality and accuracy of the generated content. Over time, the solution can flexibly update the knowledge base by updating the knowledge base without retraining the entire model. In addition, the interpretability of the model can be improved by displaying the retrieved information used to generate content, so that users can understand the basis of the model's decision. However, there is error propagation in this solution, and the performance of this solution is highly dependent on the quality of retrieval recall. If the retrieved information is inaccurate or irrelevant, it will directly lead to a decrease in the quality of the generated content. The semantic large model does not actually master the industry knowledge structure, nor does it understand the business logic, resulting in limited improvement capabilities in some business scenarios of logical reasoning.

[0054] Based on the above analysis, the incremental pre-training method is finally selected in this embodiment to introduce professional knowledge and private domain data into the semantic large model, which requires solving the problems of catastrophic forgetting and model collapse. In this regard, the two mainstream methods: partial training and LORA (Low-Rank Adaptation) training, are both aimed at reducing the amount of training parameters and improving training efficiency. Among them, partial training only updates part of the parameters of the model, while LORA training achieves effective update of parameters by adding low-rank adapters to specific layers of the model. However, the incremental pre-training effect of the model will decrease as the amount of training parameters decreases.

[0055] In order to enable the semantic large model to better understand and use professional terms in professional fields, it is a better way to use the full amount of parameters in professional fields for incremental pre-training. Based on this, the semantic large model incremental pre-training method provided by the present invention does not need to reduce the amount of training parameters, and can also overcome catastrophic forgetting and model collapse.

[0056] Specifically, the incremental pre-training method of the semantic big model of the present invention realizes the incremental pre-training of the semantic big model by constructing a generative adversarial network using the semantic big model as a generator and a discriminator, and training the generative adversarial network based on incremental training samples in professional fields and pre-trained samples of the semantic big model. Training by constructing a generative adversarial network can effectively avoid the risk of catastrophic forgetting of the semantic big model when fitting incremental training samples in professional fields; at the same time, training based on incremental training samples in professional fields and pre-trained samples of the semantic big model can also effectively avoid the problem of model collapse. This method does not require retraining of the semantic big model, and can be directly based on the open source semantic big model, using less computing power, samples and time costs to complete incremental training based on data in professional fields. It can use the understanding ability of the semantic big model itself to better learn and adapt to professional terminology, knowledge structure and business logic in professional fields, and improve the application effect in professional fields.

[0057] In a possible implementation, the training of the adversarial generative network based on the professional domain incremental training samples and the semantic large model pre-training samples includes: using the discriminator training step and the generator training step to alternately perform the discriminator training step, based on the professional domain incremental training samples and the semantic large model pre-training samples, and training the adversarial generative network according to the optimization objective function; wherein the optimization objective function is:

[0058]

[0059] Among them, SGD is the discriminator, SGG is the generator, L(L SGD , L SGG ) is the adversarial function to measure the adversarial nature between the generator and the discriminator; L SGD is the discriminator loss function, L SGG is the generator loss function.

[0060] The discriminator training steps include: fixing the generator parameters, based on the incremental training samples in the professional field and the pre-training samples of the semantic large model, by maximizing L(L SGD , L SGG ) to train the discriminator.

[0061] The generator training steps include: fixing the discriminator parameters, based on the incremental training samples of the professional field and the pre-training samples of the semantic large model, by minimizing L(L SGD , L SGG ) to train the discriminator.

[0062] Specifically, the semantic big model incremental pre-training method of the present invention can realize the incremental pre-training of any semantic big model or general big model in any professional field. For the convenience of description, in this embodiment, the open source generative big model LLaMa3 is used as an example of the incremental pre-training of the semantic big model in the power field.

[0063] First, we use the open-source generative model LLaMa3 as the generator and discriminator. The generator is responsible for generating text, and the training goal is to deceive the discriminator while generating fluent text. The discriminator is responsible for judging the reliability of the generated text, and the training goal is to distinguish between real text and generated text. The results of the discriminator can also be used to evaluate the generator's mastery of new and old knowledge. We build an adversarial generative network based on the generator and the discriminator.

[0064] Then, the semantic big model pre-training samples of the generative big model LLaMa3 are prepared, and the incremental training samples in the power field are prepared according to the quality standard of the semantic big model pre-training samples.

[0065] Optionally, the power field incremental training samples are obtained through one or more of the following: technical manuals, operating procedures, maintenance guides, customer service records, and legal and regulatory texts of the power industry.

[0066] In incremental pre-training, the generator needs to continuously perform prediction tasks based on the input for training. Half of the generator's input comes from professional domain incremental training samples, that is, domain expertise and private domain data, with the goal of learning professional domain knowledge; the other half comes from semantic large model pre-training samples, that is, general data, with the goal of consolidating learned knowledge. Optionally, the number of professional domain incremental training samples and semantic large model pre-training samples is the same. The discriminator distinguishes between the generator's generated samples and real samples, and the discriminator's feedback is used to guide the generator's training to make its generated samples more realistic.

[0067] In each round of training, the generator tries to generate samples that can deceive the discriminator, while the discriminator trains to improve its ability to distinguish. That is, in adversarial training, the loss functions of the generator and the discriminator are mutually adversarial, the discriminator tries to maximize its loss function to better distinguish between real samples and generated samples, and the generator tries to minimize its loss function to generate more realistic generated samples.

[0068] This adversarial process can be formalized as a minimum-maximum problem, or a zero-sum game problem, in which the discriminator maximizes the ability to distinguish between real samples and generated samples, while the generator minimizes the discriminator's ability to distinguish. Based on this, the optimization objective function is constructed in combination with the zero-sum game idea:

[0069]

[0070] Among them, L(L SGD , L SGG ) is the adversarial function that measures the adversarial nature between the generator and the discriminator.

[0071] During the training process, the discriminator training steps and the generator training steps are alternated to gradually improve the performance of the semantic large model through multiple iterations.

[0072] See also Figures 2 to 4 , where the discriminator training step is to maximize L(L SGD , L SGG ) to train the discriminator. The generator training step is to minimize L(L SGD , L SGG ) to train the discriminator.

[0073] Specifically, both the discriminator training step and the generator training step include: using real samples such as professional field incremental training samples and semantic large model pre-training samples to obtain the discriminator's prediction situation, and generating some generated samples through the generator, and obtaining the discriminator's prediction situation through these generated samples, and finally integrating the specific values ​​of the two prediction situations, which are used to calculate the discriminator loss function value in the discriminator training step, and update the discriminator according to the discriminator loss function value; in the generator training step, they are used to calculate the generator loss function value, and update the generator according to the generator loss function value.

[0074] In adversarial training, the loss function is a key indicator to measure model performance, which guides the update of model parameters. The definition of loss function is different for generator and discriminator.

[0075] The goal of the discriminator is to distinguish between real samples and generated samples, so the discriminator loss function is designed to maximize the recognition probability of real samples and the rejection probability of generated samples.

[0076] Optionally, the discriminator loss function L SGD for:

[0077]

[0078] Where E represents the expected value, SGD(x) is the predicted probability of the discriminator for the true sample x, SGD(SGG(z)) is the predicted probability of the discriminator for the generator's generated sample SGG(z), z is the input noise of the generator, and p data (x) is the distribution of the real samples, p z(z) is the distribution of input noise; the real samples x include incremental training samples in professional fields and pre-training samples of semantic large models.

[0079] The training goal of the discriminator is to minimize the discriminator loss function L SGD .

[0080] The goal of the generator is to generate samples that can deceive the discriminator. Optionally, the generator loss function L SGG It includes two parts, namely:

[0081]

[0082] in, To generate adversarial loss; is the next sample token prediction loss of the semantic large model, and λ is the balance and Hyperparameters of .

[0083] Generative Adversarial Loss It aims to minimize the logarithmic probability of the discriminator identifying the probability that the generated sample is a real sample, as shown in the following formula:

[0084]

[0085] Next sample label prediction loss for semantic large models The purpose is to minimize the error of the language model in predicting the next token, as shown in the following formula:

[0086]

[0087] Where T is the sample length, x t is the tth sample label, P is the conditional probability, and θ is the generator parameter.

[0088] The generator must not only generate samples that can deceive the discriminator, but also accurately predict the next token to ensure that the generated samples are grammatically and semantically coherent. By combining adversarial loss and the next token prediction loss of the semantic large model, the generator can generate more natural and coherent text.

[0089] Finally, after completing the training of the generative adversarial network, the generator in the trained generative adversarial network is used as the incremental pre-training result of the semantic large model, that is, the incremental pre-training large model in the professional field.

[0090] In a possible implementation, the semantic large model incremental pre-training method further includes: obtaining the discriminator in the trained adversarial generative network to obtain a detection model of the professional field incremental pre-training large model; wherein the detection model is used to obtain the confidence of the output of the professional field incremental pre-training large model based on the output of the professional field incremental pre-training large model.

[0091] Specifically, the discriminator in the trained generative adversarial network is used as the detection model of the incremental pre-trained large model in the professional field, and the confidence of the output results of the incremental pre-trained large model in the professional field is scored, providing users or model designers with a reference for the model's prediction accuracy and avoiding outputting hallucinatory answers to users.

[0092] In a possible implementation manner, the application of the power semantic big model obtained by the above method in the power technical standard question-answering system based on the power semantic big model retrieval enhancement is further described.

[0093] See also Figure 5 , such as asking "What is the standard for chromatographic acetylene in the main transformer?" Among them, "main transformer" is an "industry term" in the power field. This term has no clear academic definition, but it is generally recognized in the power field that it refers to transformers with voltage levels of 110kv and above. First, the incrementally trained power semantic big model can recognize the user's intentions and accurately generate search terms to retrieve technical standard clauses. Secondly, the user's questions and search content are spliced ​​into the preset prompt template of the power semantic big model and input into the power semantic big model. Thirdly, after obtaining the inference results of the power semantic big model, the detection model of the power semantic big model is used to detect the inference results. If it passes, it is returned. If it fails, the power semantic big model is asked to infer again until a reliable result is obtained.

[0094] The following are device embodiments of the present invention, which can be used to implement the method embodiments of the present invention. For details not disclosed in the device embodiments, please refer to the method embodiments of the present invention.

[0095] See also Figure 6 In another embodiment of the present invention, a semantic large model incremental pre-training system is provided, which can be used to implement the above-mentioned semantic large model incremental pre-training method. Specifically, the semantic large model incremental pre-training system includes a data acquisition module, an incremental training module and a result output module.

[0096] Among them, the data acquisition module is used to obtain incremental training samples, semantic big models and semantic big model pre-training samples in professional fields; the incremental training module is used to construct an adversarial generative network using the semantic big model as a generator and a discriminator, and to train the adversarial generative network based on the incremental training samples in professional fields and the semantic big model pre-training samples; the result output module is used to obtain the generator in the trained adversarial generative network to obtain the incremental pre-trained big model in the professional field.

[0097] In a possible implementation, the result output module is also used to obtain the discriminator in the trained adversarial generative network to obtain a detection model of the professional field incremental pre-trained large model; wherein the detection model is used to obtain the confidence of the output of the professional field incremental pre-trained large model based on the output of the professional field incremental pre-trained large model.

[0098] All relevant contents of each step involved in the embodiment of the aforementioned semantic large model incremental pre-training method can be referred to the functional description of the functional modules corresponding to the semantic large model incremental pre-training system in the embodiment of the present invention, and will not be repeated here.

[0099] The division of modules in the embodiments of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present invention may be integrated into one processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0100] In another embodiment of the present invention, a computer device is provided, the computer device including a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, which are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions in a computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the semantic large model incremental pre-training method.

[0101] In another embodiment of the present invention, the present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in a computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space that stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by a processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the incremental pre-training method for a semantic large model in the above embodiment.

[0102] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0103] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0104] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0105] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A semantic large model incremental pre-training method, characterized in that: include: Obtain incremental training samples, semantic large models, and semantic large model pre-training samples in professional fields; The semantic big model is used as the generator and discriminator to build a generative adversarial network, and the generative adversarial network is trained based on incremental training samples in professional fields and pre-trained samples of the semantic big model. Get the generator in the trained generative adversarial network and obtain a large incremental pre-training model for professional fields.

2. The semantic large model incremental pre-training method according to claim 1 is characterized in that: The training of the adversarial generative network based on the professional field incremental training samples and the semantic large model pre-training samples includes: The discriminator training steps and the generator training steps are performed alternately, based on the incremental training samples of professional fields and the pre-trained samples of the semantic large model, and the adversarial generative network is trained according to the optimized objective function; Among them, the optimization objective function is: Among them, SGD is the discriminator, SGG is the generator, L(L SGD , L SGG ) is the adversarial function to measure the adversarial nature between the generator and the discriminator; L SGD is the discriminator loss function, L SGG is the generator loss function; The discriminator training steps include: fixing the generator parameters, based on the incremental training samples in the professional field and the pre-training samples of the semantic large model, by maximizing L(L SGD , L SGG ) to train the discriminator; The generator training steps include: fixing the discriminator parameters, based on the incremental training samples of the professional field and the pre-training samples of the semantic large model, by minimizing L(L SGD , L SGG ) to train the discriminator.

3. The semantic large model incremental pre-training method according to claim 2 is characterized in that: The discriminator loss function L SGD for: Where E represents the expected value, SGD(x) is the predicted probability of the discriminator for the true sample x, SGD(SGG(z)) is the predicted probability of the discriminator for the generator's generated sample SGG(z), z is the input noise of the generator, and p data (x) is the distribution of the real samples, p z (z) is the distribution of input noise; the real sample x includes incremental training samples in professional fields and pre-training samples of semantic large models; The generator loss function L SGG for: in, To generate adversarial loss; is the next sample tag prediction loss of the semantic large model, T is the sample length, x t is the t-th sample label, P is the conditional probability, θ is the generator parameter; λ is the balance and Hyperparameters of .

4. The semantic large model incremental pre-training method according to claim 1, characterized in that: The number of the professional field incremental training samples and the number of the semantic large model pre-training samples are the same.

5. The semantic large model incremental pre-training method according to claim 1, characterized in that: Also includes: The discriminator in the trained adversarial generative network is obtained to obtain a detection model of the professional field incremental pre-trained large model; wherein the detection model is used to obtain the confidence of the output of the professional field incremental pre-trained large model based on the output of the professional field incremental pre-trained large model.

6. The semantic large model incremental pre-training method according to claim 1, characterized in that: The professional field incremental training samples are electric power field incremental training samples; the electric power field incremental training samples are obtained through one or more of the following: technical manuals, operating procedures, maintenance guides, customer service records and legal and regulatory texts of the electric power industry; The professional field incremental pre-trained big model is a power semantic big model.

7. A semantic large model incremental pre-training system, characterized in that: include: Data acquisition module, used to obtain professional domain incremental training samples, semantic large models and semantic large model pre-training samples; The incremental training module is used to construct a generative adversarial network using the semantic big model as a generator and discriminator, and train the generative adversarial network based on incremental training samples in professional fields and pre-trained samples of the semantic big model; The result output module is used to obtain the generator in the trained adversarial generative network and obtain a large incremental pre-training model in a professional field.

8. The semantic large model incremental pre-training system according to claim 7, characterized in that: The result output module is also used to obtain the discriminator in the trained adversarial generative network to obtain the detection model of the professional field incremental pre-trained large model; wherein the detection model is used to obtain the confidence of the output of the professional field incremental pre-trained large model based on the output of the professional field incremental pre-trained large model.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the semantic large model incremental pre-training method as described in any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the semantic large model incremental pre-training method as described in any one of claims 1 to 6 are implemented.

Images