Large language model pre-training and active fine tuning method for neural network prediction proxy

Abstract

The invention relates to the technical field of language processing, in particular to a large language model pre-training and active fine tuning method for a neural network prediction proxy, and the method comprises the steps: obtaining a training data set of a large language model, extracting the low-dimensional data representation of each training sample in the training data set, inputting the low-dimensional data representation into a proxy neural network, obtaining a predicted loss function value of each training sample as an approximately evaluated loss value of each training sample in the large language model; and actively sampling and marking a plurality of target samples from the plurality of training samples based on the loss function value, forming a current training batch according to the plurality of target samples, pre-training or actively fine-tuning the large language model based on the current training batch, and using an obtained latest batch loss function data set to optimize and update the proxy neural network. Therefore, the problems of low sample efficiency and calculation efficiency, poor adaptive robustness and the like in the pre-training or large-scale active fine tuning process of the large language model are solved.

Description

Technical Field

[0001] The present application relates to the field of language processing technology, and in particular to a large language model pre-training and active fine-tuning method for a neural network prediction agent. Background Art

[0002] LLM (Large Language Model) is a natural language processing system based on deep learning, usually using a transformer architecture. Its core feature is to learn the grammar, semantics, and contextual patterns of a language through pre-training with a large amount of text data. They rely on massive parameters and large-scale computing resources and have a powerful ability to generate and understand language.

[0003] Pre-training of large language models is of great significance, as it provides the model with rich language knowledge and contextual understanding capabilities. During the pre-training phase, through unsupervised learning of massive text data, the model can capture the structure of the language, grammatical rules, and the relationship between words, thereby forming a deep semantic understanding. This process not only improves the generalization ability of the model, enabling it to learn transferable task representations and perform well in a variety of downstream tasks.

[0004] Fine-tuning is a technique for adapting a pre-trained language model for a specific task. In the pre-training phase, the model usually learns general language knowledge, but in specific tasks, such as sentiment analysis, machine translation, or question-answering systems, fine-tuning is required to adapt the model to the task requirements. There are usually two ways to fine-tune: Full parameter fine-tuning: train and adjust all parameters of the model. This method is flexible but computationally expensive. PEFT (Parameter-efficient fine-tuning): For example, using techniques such as LoRA (Low-Rank Adaptation), only a small part of the model's parameters are adjusted or new parameter modules are added, thereby achieving task adaptation while keeping computational and storage costs low. Together, these technologies have promoted the widespread application of large language models in practical scenarios such as dialogue, content generation, and information retrieval.

[0005] However, the prior art still has some defects, mainly in the following two points:

[0006] (1) Sample efficiency and computational efficiency issues in pre-training and active fine-tuning. The data quality of large language models is often uneven, resulting in low data utilization of large language models during uniform distribution pre-training. Although large-scale training data can help the model learn rich language features, in actual training, the model cannot fully utilize this data. First, in the early stages of training, the model usually requires a lot of time for the warm-up phase, during which the model's ability to learn data is relatively weak. The batch training method commonly used by language models may result in a high rate of data reuse and low learning efficiency for new data. When processing large-scale data sets, the model may be exposed to the same batch of data multiple times, resulting in information redundancy rather than gain. Relatively speaking, the model's ability to learn newly introduced samples is insufficient, which limits its absorption of more samples and information, and thus affects its performance in a variety of application scenarios. Similar problems also occur during active fine-tuning. The model is usually large in scale and requires more computing resources. How to select high-quality samples to label and optimize becomes a key factor limiting the fine-tuning performance of large models.

[0007] (2) The robustness problem of pre-training and active fine-tuning. The poor adaptive robustness of large language models is also a problem that needs to be solved urgently. In practical applications, changes in the environment and input data may cause significant fluctuations in model performance. For example, when the model encounters input that is inconsistent with the distribution of the training data, its performance may drop significantly. This lack of robustness often makes the model unable to effectively deal with noise, ambiguity, and diversity in the real world. Especially in some application scenarios with extremely high accuracy requirements, such as medical and legal fields, the lack of model adaptability may lead to serious consequences. In addition, as user needs change and new language phenomena emerge, models need to be continuously updated and retrained to maintain their effectiveness. However, current training methods are often not adaptable to new data, resulting in the model's inability to quickly adapt and maintain good performance when faced with new tasks or new fields. Summary of the invention

[0008] The present application provides a large language model pre-training and active fine-tuning method for a neural network prediction agent to solve the problems of low sample efficiency and computational efficiency, poor adaptive robustness, etc. in the pre-training process of the large language model.

[0009] In a first aspect, the present application provides a large language model pre-training and active fine-tuning method for a neural network prediction agent, comprising the following steps: obtaining a training data set of a large language model, the training data set including multiple training samples; extracting a low-dimensional data representation of each training sample in the training data set, inputting the low-dimensional data representation into a proxy neural network of the large language model, and the proxy neural network outputting a loss function value for each training sample; actively sampling multiple target samples from the multiple training samples based on the loss function value, forming a current training batch based on the multiple target samples, and pre-training or actively fine-tuning the large language model based on the current training batch.

[0010] Optionally, extracting a low-dimensional data representation of each training sample in the training data set includes: inputting each training sample in the training data set into a feature encoding network, and the feature encoding network outputting the low-dimensional data representation of each training sample.

[0011] Optionally, the feature encoding network is a language encoder in a pre-trained text-image multimodal model CLIP, and the language encoder is used to encode each training sample to obtain a low-dimensional data representation.

[0012] Here, the agent neural network uses maximum likelihood as the optimization objective:

[0013]

[0014] in, represents the optimization objective; θ is the proxy neural network parameter; p θ represents the output probability distribution of the proxy neural network; φ represents the pre-trained feature encoding network; K is the training batch size; When the data is selected for the tth training step Afterwards, the actual evaluation loss function value is obtained by training the large language model once.

[0015] Optionally, multiple target samples are actively sampled from multiple training samples based on the loss function value, including: screening multiple training samples according to the loss function value and a rejection sampling criterion; using the screened training samples for the next iterative update of pre-training or fine-tuning of the large language model according to the loss function value and a worst-case criterion; storing a pair of low-dimensional data representation of the current real batch samples and the real loss function value evaluated in the large language model as proxy neural network training data, and optimizing and updating the proxy neural network.

[0016] Optionally, the rejection sampling criterion includes: discarding training samples whose loss function values ​​among multiple training samples are greater than a first preset value.

[0017] Optionally, the worst case criterion includes: selecting training samples whose loss function values ​​are greater than a second preset value from the screened training samples.

[0018] The second aspect of the present application provides a pre-training and active fine-tuning device for a large language model based on a neural network prediction agent, comprising: an acquisition module, used to acquire a training data set of a large language model, the training data set including multiple training samples; a processing module, used to extract a low-dimensional data representation of each training sample in the training data set, input the low-dimensional data representation into a proxy neural network of the large language model, and the proxy neural network outputs a loss function value for each training sample; a training fine-tuning module, used to actively sample multiple target samples from multiple training samples based on the loss function value, form a current training batch based on the multiple target samples, and pre-train or actively fine-tune the large language model based on the current training batch.

[0019] The third aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the large language model pre-training and active fine-tuning method of the neural network prediction agent of the first aspect.

[0020] A fourth aspect of the present application provides a computer-readable storage medium having a computer program or instruction stored thereon, which, when executed, implements the large language model pre-training and active fine-tuning method of the neural network prediction agent of the first aspect.

[0021] Therefore, this application includes the following beneficial effects:

[0022] The embodiment of the present application obtains a training data set of a large language model, extracts a low-dimensional data representation of each training sample in the training data set, and inputs it into a lightweight proxy neural network of the large language model to obtain a predicted loss function value for each training sample, actively samples multiple target samples from multiple training samples based on the loss function value to form a current training batch, uses the current training batch to pre-train and actively fine-tune the large language model, uses a robust and data-efficient active sampling strategy and uses a proxy neural network to reduce computational overhead, shorten training time, and improve training efficiency and robustness. Thus, the process of forwarding evaluation of a large amount of data in the large model during the active screening process is represented by a small model prediction, which solves the problems of low sample efficiency and computational efficiency, poor adaptive robustness, and the like in the pre-training process of the large language model and active fine-tuning.

[0023] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0025] Figure 1 A flowchart of a large language model pre-training and active fine-tuning method for a neural network prediction agent provided according to an embodiment of the present application;

[0026] Figure 2 This is an example diagram of a large language model pre-training and active fine-tuning device based on a neural network prediction agent according to an embodiment of the present application;

[0027] Figure 3 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0029] The following describes the large language model pre-training and active fine-tuning method of the neural network prediction agent of the embodiment of the present application with reference to the accompanying drawings. In view of the problems of low sample efficiency and computational efficiency, poor adaptive robustness, etc. in the pre-training process of the large language model mentioned in the above background technology, the present application provides a large language model pre-training and active fine-tuning method of a neural network prediction agent, in which a training data set of a large language model is obtained, a low-dimensional data representation of each training sample in the training data set is extracted, and it is input into the agent neural network of the large language model to obtain the loss function value of each training sample, and multiple target samples are actively sampled from multiple training samples based on the loss function value to form a current training batch, and the current training batch is used to pre-train or actively fine-tune the large language model, and a robust and data-efficient active sampling strategy and an agent neural network are used to reduce the computational overhead, shorten the time consumption, and improve the efficiency and robustness of training. Thus, the problems of low sample efficiency and computational efficiency, poor adaptive robustness, etc. in the pre-training process of the large language model are solved.

[0030] Specifically, Figure 1 A flowchart of a large language model pre-training and active fine-tuning method for a neural network prediction agent provided in an embodiment of the present application.

[0031] like Figure 1 As shown, the large language model pre-training and active fine-tuning method of the neural network prediction agent includes the following steps:

[0032] In step S101, a training data set of a large language model is obtained, where the training data set includes a plurality of training samples.

[0033] Among them, the big language model is a deep learning model that can gain an understanding of language structure, grammar, and semantics by training on a large amount of text data; a training dataset refers to a specific set of data used to train the model, which is usually composed of a large number of text fragments, which can be sentences, paragraphs, or documents; a training sample refers to a single text entry or instance.

[0034] In step S102, a low-dimensional data representation of each training sample in the training data set is extracted, and the low-dimensional data representation is input into the proxy neural network of the large language model, and the proxy neural network outputs the loss function value of each training sample.

[0035] Among them, low-dimensional data representation refers to converting the original high-dimensional training samples into a representation of lower dimensions but still retaining key features and information, which can simplify the data structure while maintaining its core properties for more efficient analysis; the method of extracting low-dimensional data representation of each training sample in the training data set will be described in detail below and will not be repeated here; the proxy neural network is a small and efficient neural network that can replace the actual large language model to evaluate the quality of training samples and reduce the computational cost required to directly access large models; the loss function value is used to measure the degree of difference between the model prediction and the true value.

[0036] It can be understood that the embodiment of the present application first extracts a low-dimensional data representation of each training sample in the training data set, and provides the obtained low-dimensional data representation as input to the proxy neural network, which can accurately estimate the loss function value corresponding to each training sample without directly relying on an expensive large language model.

[0037] In an embodiment of the present application, extracting a low-dimensional data representation of each training sample in a training data set includes: inputting each training sample in the training data set into a feature encoding network, and the feature encoding network outputs a low-dimensional data representation of each training sample.

[0038] Among them, the feature encoding network will be described in detail below and will not be repeated here.

[0039] It can be understood that the embodiment of the present application processes the training data set through a feature encoding network to obtain a low-dimensional data representation of each training sample in the training data set.

[0040] In an embodiment of the present application, the feature encoding network is a language encoder in a pre-trained text-image multimodal model CLIP, and the language encoder is used to encode each training sample to obtain a low-dimensional data representation.

[0041] Among them, the pre-trained image-text multimodal model CLIP is a multimodal model that has been pre-trained on a large-scale image-text pair dataset. The language encoder used to process text is based on the Transformer architecture and can effectively capture the structural information and semantic content of the text.

[0042] It can be understood that the embodiment of the present application uses the language encoder of the pre-trained graphic multimodal model CLIP to encode each training sample of the training data set to obtain a low-dimensional data representation, reduce the complexity of the data, and retain sufficient contextual information.

[0043] In the embodiment of the present application, the agent neural network uses maximum likelihood as the optimization objective:

[0044]

[0045] in, represents the optimization objective; θ is the proxy neural network parameter; p θ represents the output probability distribution of the proxy neural network; φ represents the pre-trained feature encoding network; K is the training batch size; When the data is selected for the tth training step Afterwards, the corresponding true evaluation loss function value is obtained by training the large language model once.

[0046] In step S103, multiple target samples are actively sampled from multiple training samples based on the loss function value, a current training batch is formed according to the multiple target samples, and a large language model is pre-trained or actively fine-tuned based on the current training batch.

[0047] Among them, based on the loss function value, multiple target samples are actively sampled from multiple training samples. The active sampling based on the worst-case criterion is described in detail below and will not be repeated here. After obtaining multiple target samples, the formula Get the estimated loss function value corresponding to each sample According to the estimated loss function value, the worst K samples, that is, the ones with the highest loss function value, are selected to form the current training batch, that is,

[0048]

[0049] It can be understood that the embodiment of the present application performs active sampling, first collects samples larger than the batch size required for training as target samples, and calculates the estimated loss function value corresponding to each target sample, selects the worst K samples (with the highest loss function value) to form the current training batch, and pre-trains or actively fine-tunes the large language model based on the current training batch.

[0050] In an embodiment of the present application, multiple target samples are actively sampled from multiple training samples based on the loss function value, including: screening multiple training samples according to the loss function value and the rejection sampling criterion; using the training samples screened according to the loss function value and the worst-case criterion for the next iterative update of pre-training or fine-tuning of the large language model; storing the low-dimensional data representation of the current real batch samples and the real loss function value pair evaluated in the large language model as the proxy neural network training data, and optimizing and updating the proxy neural network.

[0051] Among them, the rejection sampling criterion is a strategy used to exclude samples that continuously show high losses and are difficult to train, so as to avoid wasting computing resources. It will be described in detail below and will not be repeated here.

[0052] It can be understood that the embodiment of the present application screens multiple training samples through rejection sampling criteria and loss function values, selectively discards samples that are difficult to train, and avoids waste of training resources. After the screening is completed, multiple target samples are actively sampled according to the loss function value and the worst-case criterion, and the low-dimensional data representations of these samples and the true loss function values ​​evaluated in the large language model are used as the proxy neural network training data, and these data are used to optimize and update the proxy neural network.

[0053] In an embodiment of the present application, the rejection sampling criterion includes: discarding training samples whose loss function values ​​among multiple training samples are greater than a first preset value.

[0054] The first preset value is set according to actual conditions and is not specifically limited here.

[0055] It can be understood that the rejection sampling criterion of the embodiment of the present application is to compare the loss function values ​​in multiple training samples with the first preset value, and discard the training samples whose loss function values ​​in multiple training samples are greater than the first preset value to avoid wasting training resources.

[0056] In the embodiment of the present application, the worst case criterion includes: selecting training samples whose loss function values ​​are greater than a second preset value from the screened training samples.

[0057] Among them, the second preset value is the batch size required for training, which is set according to actual conditions and is not specifically limited here.

[0058] It can be understood that the worst-case criterion of the embodiment of the present application is to obtain the batch size required for training as the second preset value, screen out training samples that are larger than the batch size required for training, and use them as target samples for active sampling.

[0059] According to the large language model pre-training and active fine-tuning method of the neural network prediction agent proposed in the embodiment of the present application, by obtaining a training data set of the large language model, extracting the low-dimensional data representation of each training sample in the training data set, and inputting it into the agent neural network of the large language model, the loss function value of each training sample is obtained, and based on the loss function value, multiple target samples are actively sampled from multiple training samples to form a current training batch, and the current training batch is used to pre-train and actively fine-tune the large language model. A robust and data-efficient active sampling strategy is used and a agent neural network is used to reduce computational overhead, shorten training time, and improve training efficiency and robustness.

[0060] The following further describes the large language model pre-training and active fine-tuning method of the neural network prediction agent through a specific embodiment.

[0061] The main idea of ​​this embodiment is to use the worst case criterion to actively select samples with larger loss function values ​​in the batch for training at each step of large language model pre-training and active fine-tuning to improve the robustness of fine-tuning. In order to avoid the huge computational overhead caused by accessing the large language model when selecting samples, it is proposed to use a small neural network to learn the loss function value corresponding to the sample.

[0062] (1) Low-dimensional data representation

[0063] The data used for fine-tuning the large language model is in the form of natural language, which may contain multiple sentences / paragraphs and is difficult to be directly used for neural network input. In this embodiment, it is proposed to use a pre-trained feature encoding network φ to process it and convert it into low-dimensional features to adapt to the neural network input.

[0064] Specifically, we use the language encoder (Text Encoder) in a pre-trained text-image multimodal model CLIP proposed in recent years and widely used to encode the fine-tuning data of the large language model to obtain the corresponding low-dimensional data representation φ CLIP (x).

[0065] (2) Proxy Neural Network

[0066] A core idea of ​​this embodiment is to use a small-scale neural network as a proxy for the large language model M, amortizing the huge computational overhead of obtaining the corresponding loss function value for each sample. As mentioned above, a pre-trained representation model is used to obtain a low-dimensional data representation, and there are no special requirements for the proxy neural network. When the data is selected in the tth training step, After that, the large language model will be trained once to obtain the corresponding loss function value Use this data to train a proxy neural network, using maximum likelihood as the optimization objective:

[0067]

[0068] in represents the optimization objective, θ is the proxy neural network parameter, and p θ represents the output probability distribution of the proxy neural network, φ represents the pre-trained feature encoding network, and K is the training batch size.

[0069] (3) Active sampling based on the worst-case criterion

[0070] In order to improve the robustness of large language model pre-training and active fine-tuning, a worst-case criterion is adopted for active sampling.

[0071] Specifically, first collect data from the dataset that is larger than the batch size required for training Afterwards, use the proxy neural network to obtain the estimated loss function value corresponding to each sample

[0072]

[0073] Afterwards, based on the estimated loss function value, the worst K samples (those with the highest loss function value) are selected to form the current training batch:

[0074]

[0075] (4) Rejection sampling criteria

[0076] In the pre-training and active fine-tuning of large language models, some data may contain errors or low quality, which may make these data difficult to train and always maintain high loss function values. In this case, these data should be selectively discarded to avoid wasting training resources. Therefore, a rejection sampling criterion is introduced to avoid the interference of these biased data. Specifically, first discard The samples with the largest loss function value are selected, and then active sampling is performed according to the worst-case criterion introduced in (3).

[0077] Next, a large language model pre-training and active fine-tuning device based on a neural network prediction agent proposed in an embodiment of the present application is described with reference to the accompanying drawings.

[0078] Figure 2 It is a block diagram of a large language model pre-training and active fine-tuning device of a neural network prediction agent in an embodiment of the present application.

[0079] like Figure 2As shown, the large language model pre-training and active fine-tuning device 10 based on neural network prediction agent includes: an acquisition module 201, a processing module 202 and a training fine-tuning module 203.

[0080] Among them, the acquisition module 201 is used to obtain a training data set of a large language model, and the training data set includes multiple training samples; the processing module 202 is used to extract a low-dimensional data representation of each training sample in the training data set, and input the low-dimensional data representation into the proxy neural network of the large language model, and the proxy neural network outputs the loss function value of each training sample; the training fine-tuning module 203 is used to actively sample multiple target samples from multiple training samples based on the loss function value, form a current training batch according to the multiple target samples, and pre-train or actively fine-tune the large language model based on the current training batch.

[0081] In an embodiment of the present application, the processing module 202 is further used to: extract a low-dimensional data representation of each training sample in the training data set, input each training sample in the training data set into a feature encoding network, and the feature encoding network outputs a low-dimensional data representation of each training sample.

[0082] In an embodiment of the present application, the feature encoding network is a language encoder in a pre-trained text-image multimodal model CLIP, and the language encoder is used to encode each training sample to obtain a low-dimensional data representation.

[0083] In the embodiment of the present application, the agent neural network uses maximum likelihood as the optimization objective:

[0084]

[0085] in, represents the optimization objective; θ is the proxy neural network parameter; p θ represents the output probability distribution of the proxy neural network; φ represents the pre-trained feature encoding network; K is the training batch size; When the data is selected for the tth training step After that, the corresponding true loss function value is obtained by training the large language model once; the low-dimensional representation of the current real batch sample and the pair of the true loss function value evaluated in the large language model are stored As a proxy neural network model p θ The online training data of (l|φ(x)) is used to optimize and update the proxy neural network.

[0086] In an embodiment of the present application, the training fine-tuning module 203 is further used to: actively sample multiple target samples from multiple training samples based on the loss function value, and screen the multiple training samples according to the loss function value and the rejection sampling criterion; actively sample multiple target samples from the screened training samples according to the loss function value and the worst case criterion.

[0087] In an embodiment of the present application, the rejection sampling criterion includes: discarding training samples whose loss function values ​​among multiple training samples are greater than a first preset value.

[0088] In the embodiment of the present application, the worst case criterion includes: selecting training samples whose loss function values ​​are greater than a second preset value from the screened training samples.

[0089] It should be noted that the aforementioned explanation of the large language model pre-training and active fine-tuning method embodiment of the neural network prediction agent is also applicable to the large language model pre-training and active fine-tuning device of the neural network prediction agent of this embodiment, and will not be repeated here.

[0090] According to the pre-training and active fine-tuning device of a large language model based on a neural network prediction agent proposed in an embodiment of the present application, a training data set of the large language model is obtained, a low-dimensional data representation of each training sample in the training data set is extracted, and the low-dimensional data representation is input into the agent neural network of the large language model to obtain the loss function value of each training sample. Based on the loss function value, multiple target samples are actively sampled from multiple training samples to form a current training batch. The current training batch is used to pre-train and actively fine-tune the large language model. A robust and data-efficient active sampling strategy is used and a agent neural network is used to reduce computational overhead, shorten training time, and improve training efficiency and robustness.

[0091] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0092] A memory 301 , a processor 302 , and a computer program stored in the memory 301 and executable on the processor 302 .

[0093] When the processor 302 executes the program, the large language model pre-training and active fine-tuning method of the neural network prediction agent provided in the above embodiment is implemented.

[0094] Furthermore, the electronic device further comprises:

[0095] The communication interface 303 is used for communication between the memory 301 and the processor 302 .

[0096] The memory 301 is used to store computer programs that can be run on the processor 302 .

[0097] The memory 301 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0098] If the memory 301, the processor 302 and the communication interface 303 are implemented independently, the communication interface 303, the memory 301 and the processor 302 can be connected to each other through a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0099] Optionally, in a specific implementation, if the memory 301, the processor 302 and the communication interface 303 are integrated on a chip, the memory 301, the processor 302 and the communication interface 303 can communicate with each other through an internal interface.

[0100] The processor 302 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0101] An embodiment of the present application also provides a computer-readable storage medium having a computer program or instruction stored thereon. When the computer program or instruction is executed, the large language model pre-training and active fine-tuning method of the above-mentioned neural network prediction agent is implemented.

[0102] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0103] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0104] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0105] It should be understood that the various parts of the present application can be implemented in hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, the steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0106] A person of ordinary skill in the art may understand that all or part of the steps carried by the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the above-mentioned program may be stored in a computer-readable storage medium, which, when executed, includes one of the steps of the method embodiment or a combination thereof.

[0107] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for pre-training and active fine-tuning of a large language model for a neural network prediction agent, characterized in that: The following steps are involved: Obtaining a training data set for a large language model, wherein the training data set includes a plurality of training samples; Extracting a low-dimensional data representation of each training sample in the training data set, inputting the low-dimensional data representation into a proxy neural network of the large language model, and the proxy neural network outputting a loss function value of each training sample; Actively sample multiple target samples from the multiple training samples based on the loss function value, form a current training batch according to the multiple target samples, and pre-train or actively fine-tune the large language model based on the current training batch.

2. The large language model pre-training and active fine-tuning method of the neural network prediction agent according to claim 1, characterized in that: The step of extracting a low-dimensional data representation of each training sample in the training data set includes: Each training sample in the training data set is input into a feature encoding network, and the feature encoding network outputs a low-dimensional data representation of each training sample.

3. The large language model pre-training and active fine-tuning method of the neural network prediction agent according to claim 2, characterized in that: The feature encoding network is a language encoder in the pre-trained text-image multimodal model CLIP, and the language encoder is used to encode each training sample to obtain a low-dimensional data representation.

4. The large language model pre-training and active fine-tuning method of the neural network prediction agent according to claim 1, characterized in that: The agent neural network uses maximum likelihood as the optimization objective: in, represents the optimization objective; θ is the proxy neural network parameter; p θ represents the output probability distribution of the proxy neural network; φ represents the pre-trained feature encoding network; K is the training batch size; When the training sample is selected for the tth training step Afterwards, the actual evaluation loss function value is obtained by training the large language model once.

5. The large language model pre-training and active fine-tuning method of the neural network prediction agent according to claim 1, characterized in that: The actively sampling a plurality of target samples from the plurality of training samples based on the loss function value comprises: Screening the plurality of training samples according to the loss function value and the rejection sampling criterion; The training samples screened according to the loss function value and the worst-case criterion are used for the next iterative update of the large language model pre-training or fine-tuning; the low-dimensional data representation of the current real batch samples and the real loss function value pair evaluated in the large language model are stored as the proxy neural network training data, and the proxy neural network is optimized and updated.

6. The method for pre-training and actively fine-tuning a large language model of a neural network prediction agent according to claim 5, characterized in that: The rejection sampling criterion includes: discarding training samples whose loss function values ​​among the multiple training samples are greater than a first preset value.

7. The method for pre-training and actively fine-tuning a large language model of a neural network prediction agent according to claim 5, characterized in that: The worst case criterion includes: selecting training samples whose loss function values ​​are greater than a second preset value from the screened training samples.

8. A large language model pre-training and active fine-tuning device for a neural network prediction agent, characterized in that: include: An acquisition module, used to acquire a training data set of a large language model, wherein the training data set includes a plurality of training samples; A processing module, configured to extract a low-dimensional data representation of each training sample in the training data set, input the low-dimensional data representation into a proxy neural network of the large language model, and the proxy neural network outputs a loss function value of each training sample; A training fine-tuning module is used to actively sample multiple target samples from the multiple training samples based on the loss function value, form a current training batch according to the multiple target samples, and pre-train or actively fine-tune the large language model based on the current training batch.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the large language model pre-training and active fine-tuning method of the neural network prediction agent according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed, the large language model pre-training and active fine-tuning method of the neural network prediction agent according to any one of claims 1 to 7 is implemented.

Images