Large language model text generation continuous traceability model training method and device

Abstract

The application provides a large language model generated text continuous tracing model training method and device, relates to the field of computer systems based on specific computing models, and the method comprises the following steps: extracting feature vectors of text data in each training sample in a current training stage, wherein the training sample further comprises a label of a type to which a large language model belongs; the large language model in the current training stage is released later than a model in a historical training stage; obtaining initial prototypes of each large language model and text feature correlation data; performing global and local decorrelation processing on the initial prototypes obtained in each historical and current training stage to obtain respective decorrelated prototypes of each large language model to generate a large language model generated text continuous tracing model. The application can solve the problem of frequent retraining caused by a fixed label set in a traditional tracing method, effectively improve the model training efficiency and reduce resource consumption, and improve the reliability and effectiveness of the tracing result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer systems based on specific computing models, and in particular to a large language model generated text continuous tracing model training method and device. BACKGROUND

[0002] With the rapid growth of large language model (LLM) generated content, its potential risk of false information dissemination has also increased. Accurately identifying and tracing the source of large language model generated text is of great significance to ensure responsibility division, enhance content verification process, and improve the transparency of information dissemination. Traditional large language model generated text tracing research mainly formalizes this task as a binary classification problem, aiming to distinguish whether a given text is generated by a large language model or written by a human. With the deepening of research, the focus has shifted from simple detection to tracing the large language model that generated the text, which is usually formalized as a multi-classification problem, making the task more practical but also more challenging.

[0003] Large language model generated text source tracing aims to determine whether a piece of text is generated by a large language model and further identify which specific large language model generated the text. In existing large language model generated text tracing methods, this task is usually framed as a multi-classification problem with a fixed label set, consisting of several predefined large language models and a human class.

[0004] However, existing large language model generated text tracing methods usually rely on a static classifier with a fixed label set. Since the label set is fixed, the classifier cannot automatically identify text generated by newly emerging large language models. Therefore, whenever a new large language model emerges, the classifier must be reconfigured with a fixed label set containing the new large language model and retrained using existing and new class-related data. This repeated retraining process not only consumes time and computational resources, but also cannot keep up with the rapid pace of large language model development. SUMMARY

[0005] In view of this, the embodiments of the present application provide a large language model generated text continuous tracing model training method and device to eliminate or improve one or more defects in the prior art.

[0006] One aspect of the present application provides a large language model generated text continuous tracing model training method, comprising:

[0007] The phase feature extraction step: input each training sample in the data set corresponding to the current training phase in the current model training process into the feature extraction unit, so that the feature extraction unit extracts the feature vector corresponding to the text data in each training sample; wherein each training sample also contains a label indicating the type of large language model generating the text data; the release time of the large language model corresponding to the current training phase is later than the release time of the large language model corresponding to each historical training phase; and according to the feature vector of each text data and the label corresponding to each text data, the initial prototype and text feature correlation data of each large language model are obtained respectively;

[0008] If the current training phase is the last training phase in the current model training process, the text feature correlation data of each large language model obtained in sequence according to each historical training phase and the current training phase is used to perform global decorrelation processing and local decorrelation processing on each initial prototype obtained in sequence according to each historical training phase and the current training phase, to obtain the decorrelated prototype corresponding to each large language model, so as to generate a large language model text continuous tracing model for predicting the type of large language model generating text data and containing the feature extraction unit based on the current decorrelated prototype and the label group containing the label corresponding to each decorrelated prototype.

[0009] In some embodiments of the present application, the phase feature extraction step further comprises:

[0010] Collecting each text data written by human and each text data generated by each large language model currently released;

[0011] Taking each text data written by human as the initial text data of the large language model, and setting a label indicating the type of large language model for the initial large language model and each large language model released;

[0012] Taking the label of the initial large language model as the first label, and sorting the labels of each large language model released in sequence after the label of the initial large language model in the order of release time from early to late to obtain the corresponding label set;

[0013] According to the order of each label in the label set, each data set corresponding to each label is constructed in sequence, the data set contains each text data corresponding to the label, and each text data in the data set and the corresponding label form different training samples.

[0014] In some embodiments of the present application, before the stage feature extraction step, the method further includes:

[0015] If a large language model that has not been applied in the historical training period is detected or received, a label is set to uniquely represent the type of the large language model, and multiple text data generated using the large language model are obtained to generate a data set for the current training stage; wherein the data set for the current training stage includes multiple training samples.

[0016] In some embodiments of the present application, the feature extraction unit includes: a pre-trained language model with fixed parameters and a random up-projection layer with fixed parameters;

[0017] Correspondingly, the process of inputting each training sample in the data set corresponding to the current training stage in the model training process into the feature extraction unit, so that the feature extraction unit extracts the feature vector corresponding to each text data in each training sample, including:

[0018] The text data corresponding to each training sample in the data set corresponding to the current training stage in this model training process are respectively input into the pre-trained language model, so that the pre-trained language model outputs the initial feature vector corresponding to each of the text data, and then the random upward projection layer performs upward projection processing on each of the initial feature vectors to obtain the feature vector corresponding to each of the text data.

[0019] In some embodiments of the present application, obtaining the initial prototype and text feature correlation data of each of the large language models according to the feature vector of each of the text data and the label corresponding to each of the text data includes:

[0020] According to the labels corresponding to the respective text data, the feature vectors of the respective text data belonging to the same large language model are summed up to obtain the initial prototypes of the respective large language models;

[0021] Furthermore, according to the labels corresponding to the respective text data, the feature vectors of the respective text data belonging to the same large language model are calculated using a Gram matrix to obtain the Gram matrix corresponding to each of the large language models as the text feature correlation number.

[0022] In some embodiments of the present application, the text feature correlation data of each of the large language models sequentially obtained in each of the historical training stages and the current training stage are sequentially obtained, and global decorrelation processing and local decorrelation processing are performed on each of the initial prototypes sequentially obtained in each of the historical training stages and the current training stage to obtain decorrelation prototypes corresponding to each of the large language models, including:

[0023] Performing global decorrelation processing on each of the initial prototypes sequentially obtained in each of the historical training stages and the current training stage based on the text feature correlation data of each of the large language models sequentially obtained in each of the historical training stages and the current training stage to obtain prototypes corresponding to each of the large language models from which global correlation has been eliminated;

[0024] Furthermore, based on the text feature correlation data of each of the large language models sequentially obtained in each of the historical training stages and the current training stage, each of the two initial prototypes is subjected to local decorrelation processing to obtain prototypes corresponding to each of the large language models from which local correlation has been eliminated;

[0025] The prototypes from which global correlation has been eliminated and the prototypes from which local correlation has been eliminated, which correspond to each of the large language models, are respectively used as decorrelation prototypes which correspond to each of the large language models.

[0026] A second aspect of the present application provides a method for continuously tracing the source of text generated by a large language model, comprising:

[0027] The target text data is input into the large language model-generated text continuous tracing model obtained by training the large language model-generated text continuous tracing model training method described in the first aspect above, so that the feature extraction unit in the large language model-generated text continuous tracing model extracts the target feature vector corresponding to the target text data, so that the large language model-generated text continuous tracing model matches the target feature vector with each of the decorrelated prototypes respectively to obtain the target prototype corresponding to the target text data, so that the large language model-generated text continuous tracing model searches for the label corresponding to the target prototype in the label group and outputs the label as the tracing prediction result of the target text data.

[0028] In some embodiments of the present application, the decorrelated prototype includes: a prototype from which global correlation has been eliminated and a prototype from which local correlation has been eliminated;

[0029] Matching the target feature vector with each of the decorrelation prototypes to obtain a target prototype corresponding to the target text data includes:

[0030] Matching the target feature vector with each of the prototypes whose global correlation has been eliminated, respectively, to obtain a global matching score corresponding to each of the prototypes whose global correlation has been eliminated;

[0031] sorting the prototypes whose global correlations have been eliminated in descending order of the global matching scores, and taking the first two prototypes whose global correlations have been eliminated as candidate prototypes;

[0032] Matching the target feature vector with the prototypes from which local correlations have been eliminated, respectively, corresponding to each of the candidate prototypes, to obtain local matching scores corresponding to each of the two candidate prototypes;

[0033] The candidate prototype with the highest local matching score is determined as the target prototype corresponding to the target text data.

[0034] A third aspect of the present application provides a large language model-generated text continuous tracing model training device, comprising:

[0035] A stage feature extraction model, used to perform the stage feature extraction step: inputting each training sample in the data set corresponding to the current training stage in this model training process into a feature extraction unit, so that the feature extraction unit extracts the feature vector corresponding to each text data in each training sample; wherein each training sample also includes a label for indicating the type of the large language model that generated the text data; the release time of the large language model corresponding to the current training stage is later than the release time of the large language model corresponding to each historical training stage; and according to the feature vector of each text data and the label corresponding to each text data, respectively obtaining the initial prototype and text feature correlation data of each large language model;

[0036] The global and local decorrelation module is used to, if the current training stage is the last training stage in this model training process, perform global decorrelation processing and local decorrelation processing on each of the initial prototypes obtained in sequence in each of the historical training stages and the current training stage according to the text feature correlation data of each of the large language models obtained in sequence in each of the historical training stages and the current training stage, so as to obtain decorrelation prototypes corresponding to each of the large language models, so as to generate a large language model-generated text continuous tracing model for predicting the type of the large language model to which the generated text data belongs and including the feature extraction unit based on the current decorrelation prototypes and the label group including the labels corresponding to each of the decorrelation prototypes.

[0037] A fourth aspect of the present application provides a device for continuously tracing the source of text generated by a large language model, comprising:

[0038] The model prediction module is used to input the target text data into the large language model generated text continuous tracing model trained by the large language model generated text continuous tracing model training method, so that the feature extraction unit in the large language model generated text continuous tracing model extracts the target feature vector corresponding to the target text data, so that the large language model generated text continuous tracing model matches the target feature vector with each of the decorrelation prototypes respectively to obtain the target prototype corresponding to the target text data, so that the large language model generated text continuous tracing model searches for the label corresponding to the target prototype in the label group and outputs the label as the tracing prediction result of the target text data.

[0039] The fifth aspect of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for training a continuous tracing model for generating text using a large language model is implemented, and / or the method for continuously tracing text generated using a large language model is implemented.

[0040] The sixth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements the method for training a model for continuously tracing text generated by a large language model, and / or implements the method for continuously tracing text generated by a large language model.

[0041] The seventh aspect of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method for training a model for continuously tracing text generated by a large language model, and / or implements the method for continuously tracing text generated by a large language model.

[0042] The present application provides a large language model to generate text continuous traceability model training method, through the stage feature extraction step: each training sample in the data set corresponding to the current training stage in this model training process is input into the feature extraction unit respectively, so that the feature extraction unit extracts the feature vector corresponding to each text data in each training sample; wherein each training sample also includes a label for indicating the type of the large language model that generates the text data; the release time of the large language model corresponding to the current training stage is later than the release time of the large language model corresponding to each historical training stage; and, according to the feature vector of each text data and the label corresponding to each text data, the initial prototype and text feature correlation data of each large language model are obtained respectively; if the current training stage is the current model training process In the last training stage, according to the text feature correlation data of each of the large language models obtained in sequence in each of the historical training stages and the current training stage, global decorrelation processing and local decorrelation processing are performed on each of the initial prototypes obtained in sequence in the historical training stages and the current training stage to obtain the decorrelation prototypes corresponding to each of the large language models. Based on the current decorrelation prototypes and the label group containing the labels corresponding to each of the decorrelation prototypes, a large language model-generated text continuous tracing model for predicting the type of large language model to which generated text data belongs and containing the feature extraction unit is generated. This can solve the problem of frequent retraining caused by a fixed label set in traditional tracing methods, effectively improve model training efficiency and reduce resource consumption, and improve the reliability and effectiveness of tracing results.

[0043] Additional advantages, purposes, and features of the present application will be described in part in the following description and will become apparent to those skilled in the art upon study of the following or may be learned from practice of the present application. The purposes and other advantages of the present application may be achieved and obtained by the structures specifically pointed out in the specification and drawings.

[0044] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present application are not limited to the above specific description, and the above and other purposes that can be achieved by the present application will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings described herein are intended to provide a further understanding of the present application, constitute a part of the present application, and do not constitute a limitation of the present application. The components in the drawings are not drawn to scale, but are only for the purpose of illustrating the principles of the present application. In order to facilitate the illustration and description of some parts of the present application, the corresponding parts in the drawings may be enlarged, that is, they may become larger than other components in the exemplary device actually manufactured according to the present application. In the drawings:

[0046] Figure 1 This is a first flow chart of a method for training a large language model to generate a continuous text tracing model in one embodiment of the present application.

[0047] Figure 2 This is a second flow chart of the method for training a large language model to generate a continuous text tracing model in one embodiment of the present application.

[0048] Figure 3 This is a third flow chart of the method for training a continuous text tracing model using a large language model in one embodiment of the present application.

[0049] Figure 4 A flowchart of a method for continuously tracing text generated by a large language model in one embodiment of the present application.

[0050] Figure 5 The figure is a flowchart of the prototype matching process in the method for continuous text tracing generated by a large language model in one embodiment of the present application.

[0051] Figure 6 This is a schematic diagram of an example of the sequential learning process corresponding to the large language model generation text continuous tracing task paradigm provided in an application example of this application.

[0052] FIG7( a ) is a schematic diagram of the learning phase corresponding to the global and local prototype decorrelation continuous traceability method provided in an application example of the present application.

[0053] FIG7( b ) is a schematic diagram of the reasoning phase corresponding to the global and local prototype decorrelation continuous tracing method provided in an application example of the present application.

[0054] Figure 8 A schematic diagram of the GLPD algorithm provided for an application example of this application.

[0055] Figure 9 Provided for this application example Figure 9 A heat map comparing the pairwise prototype similarities of the ten large language models provided in an application example of this application after global and local decorrelation.

[0056] Figure 10 This is a structural diagram of a large language model-generated text continuous tracing model training device in one embodiment of the present application.

[0057] Figure 11 A schematic diagram of the structure of a device for continuously tracing text generated by a large language model in one embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail in conjunction with the embodiments and drawings. Here, the illustrative embodiments of this application and their descriptions are used to explain this application, but are not intended to limit this application.

[0059] It should also be noted here that in order to avoid obscuring the present application due to unnecessary details, the accompanying drawings only show structures and / or processing steps that are closely related to the scheme according to the present application, while other details that are not closely related to the present application are omitted.

[0060] It should be emphasized that the term "include / comprises" when used herein refers to the existence of features, elements, steps or components, but does not exclude the existence or addition of one or more other features, elements, steps or components.

[0061] It should also be noted that, unless otherwise specified, the term "connection" herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0062] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0063] It should be noted that the researchers explored a variety of text features to distinguish large language models from different sources using supervised learning. First, a simple approach is to directly use the pre-trained language model RoBERTa to extract text features to track the source of the text. Second, by utilizing the perplexity distribution of the text in different large language models and recording the next tag probabilities of significant language model n-grams, it can help identify the specific model that generated the text. In addition, by fine-tuning the text-to-text transfer transformer model T5, it can implicitly identify the source of the text when predicting the next word, thereby improving classification accuracy. In addition, using writing style representation is also an effective method that can help distinguish the source of text generation through style features. These technical solutions provide new ideas and solutions for tracing the source of text generated by large language models.

[0064] However, while existing methods have achieved some success in attribution tasks, they typically rely on static classifiers with a fixed label set. Because the label set is fixed, the classifier cannot automatically identify text generated by newly emerging large language models. Therefore, whenever a new large language model emerges, the classifier must be re-set to include the fixed label set of the new large language model and retrained using data related to both existing and new categories. This repeated retraining process is not only time-consuming and computationally resource-intensive, but also unable to keep up with the rapid development of large language models.

[0065] Based on this, in order to solve the problem of frequent retraining caused by fixed label sets in traditional tracing methods, the embodiments of the present application respectively provide a large language model to generate text continuous tracing model training method, a large language model to generate text continuous tracing model training device, electronic device, computer-readable storage medium and computer program product for executing the large language model to generate text continuous tracing model training method, which can effectively improve model training efficiency and reduce resource consumption, and can improve the reliability and effectiveness of tracing results.

[0066] The details are described in detail through the following examples.

[0067] Based on this, the embodiment of the present application provides a large language model to generate text continuous traceability model training method that can be implemented by a large language model to generate text continuous traceability model training device, see Figure 1 The method for training a large language model to generate a continuous text traceability model specifically includes the following contents:

[0068] Step 100: Stage feature extraction step: input each training sample in the data set corresponding to the current training stage in this model training process into the feature extraction unit, so that the feature extraction unit extracts the feature vector corresponding to each text data in each training sample; wherein each training sample also includes a label for indicating the type of the large language model that generates the text data; the release time of the large language model corresponding to the current training stage is later than the release time of the large language model corresponding to each historical training stage; and, according to the feature vector of each text data and the label corresponding to each text data, respectively obtain the initial prototype and text feature correlation data of each large language model.

[0069] First of all, it should be explained that the present application provides a task paradigm for continuous tracing of text generated by a large language model. In this task paradigm, it is required to train the feature extraction unit in the continuous tracing model of the text generated by the large language model in stages. In each training stage, learning is carried out from each data sample in a data set, and each training stage is independent of each other and cannot access the data of the previous historical training stage. At the same time, when a large language model that has not been applied in the historical training period is detected or received, a data set corresponding to the large language model can also be generated, and then stage training can be performed on the data set. Then, the model parameters of the continuous tracing model of the text generated by the large language model can be updated, and there is no need to reset the classifier containing the fixed label set of the new large language model, and retrain using data related to the existing categories and the new categories.

[0070] Therefore, for the above-mentioned task paradigm, step 100 of the present application is applicable to the initial training of the continuous tracing model for text generated by the large language model, and is also applicable to the incremental training of the continuous tracing model for text generated by the large language model. It can further improve the model training efficiency and reduce resource consumption on the basis of ensuring the effectiveness and reliability of the continuous tracing results of text generated by the large language model.

[0071] In one or more embodiments of the present application, this model training process refers to the process of training a traceability model for the currently acquired data set. The current training stage refers to the training stage corresponding to the data set corresponding to the execution of the stage feature extraction step. After completing the stage feature extraction step, the training stage is changed to the historical training stage. Each of the training stages uniquely corresponds to a data set, and each of the data sets contains multiple training samples, each of the training samples contains one of the text data and a label for indicating the type of the large language model that generates the text data; wherein the label corresponds one to one with the type of the large language model. Therefore, in the present application, the label can represent the large language model, that is, the text data corresponding to the label is the text data generated by the large language model uniquely specified by the label.

[0072] It can be understood that the text feature correlation data refers to data used to represent the correlation between the feature vectors corresponding to each of the text data generated based on the same large language model.

[0073] Therefore, in response to the limitations of traditional traceability methods and the challenges of new large models, this application uses a continuous traceability method to address the frequent retraining problem caused by the fixed label set of traditional traceability methods. It is a category-incremental paradigm method. This is the first time that this application has used a continuous learning paradigm to solve the problem of text traceability in the field of large-scale model generation.

[0074] It should be noted that continuous learning (CL), also known as incremental learning (IL), enables the model to continuously accumulate knowledge in a series of tasks without catastrophic forgetting. There are three different scenarios for continuous learning: task incremental learning, domain incremental learning, and category incremental learning. Among these three scenarios, category incremental learning requires the model to retain the knowledge of old categories and effectively distinguish them while gradually learning new category objects, so it is considered to be the most challenging scenario. The continuous tracing task of generating text from a large language model provided in an embodiment of the present application is a category incremental learning paradigm. The tracing model requires that the knowledge of the old large language model be retained while gradually learning the new large language model to generate text.

[0075] However, continuous learning methods also have certain limitations in tracing tasks. Existing continuous learning methods can be roughly divided into four categories: (1) replay-based methods, which recover previous knowledge by replaying samples of old categories; (2) regularization-based methods, which use knowledge distillation to consolidate the knowledge of previous tasks; (3) architecture-based methods, which construct task-specific parameters to reduce inter-task interference in sequential learning; (4) prototype-based methods, which use frozen pre-trained models to extract category prototypes that are independent of task order.

[0076] Because replay-based methods require preserving old samples, which shares the same drawback as traditional attribution methods, replay-based continuous learning methods are not considered. Preliminary experiments have shown that prototype-based continuous learning methods offer significant performance advantages in continuous attribution tasks for the other three continuous learning methods. However, the prototypes of large language models extracted directly from pre-trained language models are highly similar. While global decorrelation techniques can eliminate collinearity between prototypes of some large language models, they cannot account for the fine-grained collinearity (similarity) between prototypes. In particular, as the classification model continuously learns an increasing number of new large language models, global decorrelation techniques can only capture overall collinearity, but not the subtle collinearity between prototypes of two large language models.

[0077] Therefore, based on the incremental learning paradigm, the embodiment of the present application also proposes a method based on global and local prototype decorrelation to further solve the problem that the similarity between the prototypes of the large language model is too high and the correlation between the prototypes of the large language model cannot be removed in a fine-grained manner. The specific process is described in step 200 below.

[0078] Step 200: If the current training stage is the last training stage in this model training process, then according to the text feature correlation data of each of the large language models obtained in sequence in each of the historical training stages and the current training stage, global decorrelation processing and local decorrelation processing are performed on each of the initial prototypes obtained in sequence in the historical training stages and the current training stage to obtain the decorrelation prototypes corresponding to each of the large language models, so as to generate a large language model-generated text continuous tracing model for predicting the type of the large language model to which the generated text data belongs and including the feature extraction unit based on the current decorrelation prototypes and the label group including the labels corresponding to each of the decorrelation prototypes.

[0079] From the above description, it can be seen that the large language model provided in the embodiment of the present application generates a continuous tracing model training method for text, which can not only solve the problem of frequent retraining of traditional tracing methods due to a fixed label set, but also solve the problem of excessive similarity between prototypes of the large language model and the inability to remove the correlation between prototypes in a fine-grained manner. It can solve the problem of frequent retraining of traditional tracing methods due to a fixed label set, effectively improve model training efficiency and reduce resource consumption, and improve the reliability and effectiveness of tracing results.

[0080] In order to further improve the effectiveness and reliability of the continuous tracing model training of the large language model generated text, in the large language model generated text continuous tracing model training method provided in the embodiment of the present application, see Figure 2 The method for training a large language model to generate a text continuous tracing model includes the following steps before step 100:

[0081] Step 010: Collect various text data written by humans and various text data generated by various currently released large language models.

[0082] Step 020: The manually written text data are used as the text data of the initial large language model, and labels indicating the types of the large language models are set for the initial large language model and the published large language models.

[0083] Step 030: The label of the initial large language model is used as the first label, and the labels of each of the large language models that have been published are sorted in order of publication time from earliest to latest after the label of the initial large language model to obtain a corresponding label set.

[0084] Step 040: sequentially constructing a data set corresponding to each of the labels in the order of the labels in the label set, wherein each of the data sets contains the text data corresponding to the label, and each of the text data in the data set and the corresponding label forms a different training sample.

[0085] That is, one of the triggering conditions of the stage feature extraction step can be the triggering of each text data written by a human and each text data generated by the currently published large language model, and steps 100 to 300 executed after step 040 can be an initial full training process of the large language model text generation traceability model to ensure the reliability and effectiveness of the application of the large language model text generation traceability model.

[0086] In order to further improve the efficiency of the large language model text generation traceability model training and reduce the resource consumption of the training device, in a large language model text generation traceability model training method provided in an embodiment of the present application, referring to Figure 3 , the step 100 in the large language model text generation traceability model training method further specifically contains the following content before the step 100:

[0087] Step 050: if a large language model not applied in the historical training period is detected or received, a label for uniquely representing the type of the large language model is set, and a plurality of text data generated by the large language model is obtained to generate a data set of the current training stage; wherein the data set of the current training stage contains a plurality of training samples.

[0088] That is, the second triggering condition of the stage feature extraction step can be triggered by detecting the existence of a newly published large language model, and steps 100 to 300 executed after step 050 can be an incremental training process of the large language model text generation traceability model, which improves the efficiency of the large language model text generation traceability model training and reduces the resource consumption of the training device based on the historical stage trained de-correlation prototypes.

[0089] In order to further improve the effectiveness and reliability of feature extraction in the large language model text generation traceability model training process, in a large language model text generation traceability model training method provided in an embodiment of the present application, the feature extraction unit includes a parameter-fixed pre-training language model and a parameter-fixed random upward projection layer, wherein the parameter-fixed pre-training language model can also be referred to as a frozen pre-training language model; and the parameter-fixed random upward projection layer can also be referred to as a frozen random upward projection layer.

[0090] Correspondingly, referring to Figure 2 or Figure 3 Step 100 of the method for training a large language model to generate a text continuous traceability model specifically includes the following:

[0091] Step 110: Input the text data corresponding to each training sample in the data set corresponding to the current training stage in this model training process into the pre-trained language model respectively, so that the pre-trained language model outputs the initial feature vector corresponding to each of the text data respectively, and then make the random upward projection layer perform upward projection processing on each of the initial feature vectors to obtain the feature vector corresponding to each of the text data.

[0092] Specifically, the calculation formula for the feature vector corresponding to each of the text data is shown in formula (1):

[0093]

[0094] in, Indicates the tth data set corresponding to the tth training stage (current stage) The feature vector corresponding to the k-th text data in ; Represents the tth data set The kth text data corresponding to the kth training sample is unique, and the dataset Contains training samples, represents the up-projection layer that maps the encoded output to a higher-dimensional space (M>L), The dimension of the feature vector representing text data is M; , Indicates that the dimension of the projection layer is L×M; represents the average pooling of the last hidden state obtained from the frozen pre-trained model, , Indicates that the dimension after pooling is L, is an element-wise nonlinear activation function.

[0095] In order to further improve the effectiveness and reliability of obtaining the initial prototype and text feature correlation data during the training process of the continuous tracing model for generating text from a large language model, in a continuous tracing model training method for generating text from a large language model provided in an embodiment of the present application, see Figure 2 or Figure 3 Step 100 in the method for training a large language model to generate a continuous text traceability model further includes the following steps performed after step 110:

[0096] Step 120: According to the labels corresponding to the respective text data, the feature vectors of the respective text data belonging to the same large language model are summed up to obtain the initial prototypes of the respective large language models.

[0097] Specifically, the calculation formula of the initial prototype of the large language model is shown in formula (2):

[0098]

[0099] in, represents the initial prototype; is the indicator function; Indicates the tth data set corresponding to the tth training stage (current stage) The label corresponding to the k-th text data in ; Indicates the label value.

[0100] And, step 130: according to the labels corresponding to each of the text data, the feature vectors of each of the text data belonging to the same large language model are calculated with a Gram matrix to obtain the Gram matrix corresponding to each of the large language models as the text feature correlation number.

[0101] It is understood that the Gram matrix refers to the Gram matrix

[0102] Specifically, the formula for calculating the Gram matrix of the feature vectors of each of the text data belonging to the same large language model is shown in formula (3):

[0103]

[0104] in, Represents the Gram matrix as the correlation number of text features; The dimension of the Gram is an M×M matrix; It is the outer product operation.

[0105] In order to effectively solve the problem that the similarity between prototypes of a large language model is too high and the correlation between prototypes cannot be removed in a fine-grained manner, in a large language model-generated text continuous traceability model training method provided in an embodiment of the present application, if the current training stage is not the last training stage in this model training process, then return to step 100 for the next data set to continue executing the stage feature extraction step; if the current training stage is the last training stage in this model training process, then refer to Figure 2 or Figure 3Step 200 of the method for training a large language model to generate a continuous text traceability model specifically includes the following:

[0106] Step 210: Based on the text feature correlation data of each of the large language models obtained in sequence in each of the historical training stages and the current training stage, global decorrelation processing is performed on each of the initial prototypes obtained in sequence in each of the historical training stages and the current training stage to obtain prototypes corresponding to each of the large language models with global correlation eliminated.

[0107] It can be understood that if the current training stage is the last training stage in this model training process, then the initial prototypes of each of the large language models obtained in sequence in each of the historical training stages and the current training stage are recorded as .

[0108] Specifically, the concatenated data consisting of prototypes whose global correlations have been eliminated corresponding to each of the large language models The calculation formula is shown in formula (4):

[0109]

[0110] in, , are prototypes corresponding to each of the large language models, from which global correlations have been eliminated; Indicates that the dimensions of the prototype are M× ; Representation dataset The corresponding label set; represents the regularization term used to ensure numerical stability during the inverse operation of formula (4); is the splicing data of each initial prototype, , Indicates that the dimensions of the prototype are M× .

[0111] Formula (4) is based on the long-established least squares error prediction theory, or more precisely, the ridge regression algorithm. This global decorrelation process enhances the distinguishability between prototypes of all large language models.

[0112] And, step 220: based on the text feature correlation data of each of the large language models obtained in sequence in each of the historical training stages and the current training stage, each two of the initial prototypes are locally decorrelated to obtain prototypes corresponding to each of the large language models with local correlation eliminated.

[0113] Specifically, each of the large language models corresponds to spliced data constituted by a prototype with eliminated local correlation The calculation formula is shown in formula (5):

[0114]

[0115] Wherein, , wherein is the i-th initial prototype corresponding to a prototype with eliminated local correlation; is the j-th initial prototype corresponding to a prototype with eliminated local correlation. denotes the spliced data of the i-th initial prototype and the j-th initial prototype; denotes a regularization term for ensuring numerical stability during inverse operation of formula (5).

[0116] Step 230: taking each of the large language models respectively as a de-correlated prototype corresponding to each of the large language models.

[0117] Based on the above-mentioned large language model generation text continuous traceability model training method, the present application also provides a large language model generation text continuous traceability method, which is described in detail in Figure 4 , and the large language model generation text continuous traceability method specifically includes the following contents:

[0118] Step 300: input the target text data into the large language model generation text continuous traceability model trained by the large language model generation text continuous traceability model training method, so that the feature extraction unit in the large language model generation text continuous traceability model extracts the target feature vector corresponding to the target text data, so that the large language model generation text continuous traceability model matches the target feature vector with each of the de-correlated prototypes respectively to obtain the target prototype corresponding to the target text data, so that the large language model generation text continuous traceability model finds the label corresponding to the target prototype in the label group and outputs the label as the traceability prediction result of the target text data.

[0119] In step 300, the feature extraction unit extracts the target text data corresponding to the target feature vector .

[0120] As can be seen from the above description, the large language model generation text continuous traceability method provided by the embodiments of the present application can improve the reliability and effectiveness of the traceability result.

[0121] In order to effectively solve the problem that the similarity between prototypes of a large language model is too high and the correlation between prototypes cannot be removed in a fine-grained manner, in a large language model-generated text continuous tracing model training method provided in an embodiment of the present application, the decorrelated prototypes include: prototypes with eliminated global correlation and prototypes with eliminated local correlation; see Figure 5 The method for continuously tracing text generated by a large language model, wherein the target feature vector is matched with each of the decorrelated prototypes to obtain a target prototype corresponding to the target text data (hereinafter referred to as the prototype matching process) specifically includes the following contents:

[0122] Step 310: Match the target feature vector with each of the prototypes whose global correlations have been eliminated, to obtain a global matching score corresponding to each of the prototypes whose global correlations have been eliminated.

[0123] The global matching scores corresponding to the prototypes whose global correlations have been eliminated The calculation formula is shown in formula (6):

[0124]

[0125] in, Represents the target text data The corresponding target feature vector; for , indicating that The global matching score of a large language model, ; Represents the prototype corresponding to the y-th large language model with global correlation eliminated.

[0126] Step 320: Sort the prototypes whose global correlations have been eliminated in descending order of the global matching scores, and use the first two prototypes whose global correlations have been eliminated as candidate prototypes.

[0127] The present application selects the two globally de-correlated prototypes that result in the highest global matching scores as candidate prototypes, representing the two most likely sources of the globally decorrelated prototype predictions.

[0128] Step 330: Match the target feature vector with the prototype from which local correlation has been eliminated, corresponding to each of the candidate prototypes, to obtain local matching scores corresponding to each of the two candidate prototypes.

[0129] The calculation formula for the local matching scores corresponding to the two candidate prototypes is shown in formula (7):

[0130]

[0131] in, Indicates that the local matching scores corresponding to the two candidate prototypes include , among which represents the local matching score corresponding to the i-th candidate prototype; represents the local matching score corresponding to the j-th candidate prototype; represents the prototype corresponding to the i-th candidate prototype with eliminated local correlation; Represents the prototype corresponding to the j-th candidate prototype with eliminated local correlation.

[0132] Step 340: Determine the candidate prototype with the highest local matching score as the target prototype corresponding to the target text data.

[0133] Among them, the target prototype is recorded as .

[0134] To further illustrate the above embodiment, the present application also provides a specific application example of a method for training a continuous tracing model for text generated by a large language model and a method for continuous tracing of text generated by a large language model, which relates to the field of natural language processing and can also be referred to as a global and local prototype decorrelation (GLPD) continuous tracing method, specifically including the following contents:

[0135] 1. Paradigm for Continuous Text Tracing Using Large Language Models

[0136] Specifically, assuming there is A large language model, namely: , sorted chronologically by release order, e.g. Each large language model has a set of texts generated by it In addition, this application example also has a set of texts written by humans. Unless otherwise specified, this application example treats human identity (Human) as a special large language model, denoted as , whose text collection is denoted as .this A sequence of large language models (and ) was subsequently divided into disjoint label sets , where the i-th label set The large language model in is always better than the j-th label set The large language model in the early release (when ), and ( ).

[0137] This application example assumes that human identity belongs to the first label set, i.e. For each label set , this application example builds its instance set by merging the text collection related to the large language model in the tag set ,Right now , Represents the k-th large language model. After that, the application example of this application obtains a A stream data sequence that defines the learning and training phase , They are different data sets, each training stage has a data set ,in It's a text. is the large language model that generated the text.

[0138] This task requires the traceability model to Sequential learning. During the training phase When , the traceability model only uses the dataset of the current training phase After learning, given any unseen text , the traceability model must predict a label , to track text Source. Figure 6 An example of this sequential learning process is shown in Figure 6In the example, stage 1, stage 2, and stage 3 represent different training stages that are performed in sequence; "Human" represents the label corresponding to the initial language model (i.e., human identity); "LLM1" represents the label of the type of the first language model; "LLM2" represents the label of the type of the second language model; "LLM3" represents the label of the type of the third language model; "LLM4" represents the label of the type of the fourth language model; "LLM5" represents the label of the type of the fifth language model; "Human-Written" represents human-written text data; "LLM1-Generated" represents the language model labeled LLM1. "LLM2-Generated" refers to text data generated by the language model labeled LLM2; "LLM3-Generated" refers to text data generated by the language model labeled LLM3; "LLM4-Generated" refers to text data generated by the language model labeled LLM4; and "LLM5-Generated" refers to text data generated by the language model labeled LLM5. Label set 1 refers to the label set corresponding to stage 1; label set 2 refers to the label set corresponding to stage 2; and label set 3 refers to the label set corresponding to stage 4. "Model" is the abbreviation for the model for continuous provenance of text generated by large language models. The task of continuous provenance of text generated by large language models well simulates real-world scenarios, in which new large language models are constantly emerging and the model needs to quickly adapt to and recognize these new large language models without frequently retraining historical data.

[0139] 2. Global and local prototype decorrelation

[0140] The application example of this application proposes a global and local prototype decorrelation continuous tracing method. This method is a prototype-based method, and its core idea is to learn the prototype of a large language model and track the source through prototype matching. Figure 7 (a) and Figure 7 (b) respectively show the overall process of the learning stage and the reasoning stage corresponding to the global and local prototype decorrelation continuous tracing method, where C0, C1, C2 and C3 represent different prototypes respectively; X represents the target text data. In the learning and training stage, a frozen pre-trained language model and a frozen upward projection layer are used to continuously extract the prototypes of the newly emerging large language model. Subsequently, global and local decorrelation mechanisms are introduced to eliminate the overall and pairwise correlations between prototypes. In the reasoning training stage, a two-step prototype matching mechanism is used to track the source of a given text. First, the input is matched with the globally decorrelated prototype to track the two most likely sources, and then matched with the corresponding locally decorrelated prototypes to make the final decision. The following application example of this application describes these two training stages in detail:

[0141] (1) Build a prototype

[0142] During the learning and training phase, GLPD gradually learns and decorrelates the prototypes of the emerging large language model. For example, the training phase provides a A dataset of training samples Each sample consists of a piece of text (represented as a sequence of tokens) and a label (referring to the large language model that generates text). This application example first uses a frozen pre-trained language model, such as LLaMA, and a frozen random upward projection layer, from Extract the features of all training samples and calculate the features of each according to the above formula (1) Get a feature vector .

[0143] Then, for each large language model , the application example of this application generates the feature vector of the training sample of the large language model by summing the above formula (2) to construct its initial prototype ; This initial prototype captures the overall characteristics of the text generated by the large language model and serves as a representative embedding of the large language model.

[0144] After prototype extraction, all large language models observed so far obtain Prototype . However, there is often a high correlation between the prototypes extracted directly through the frozen pre-trained language model, which brings challenges to accurate prototype matching, thereby compromising the accuracy of source tracking. Therefore, the application example of this application introduces global and local decorrelation mechanisms to further decorrelate these prototypes. The application example of this application eliminates the correlation between prototypes by inverse operation of the Gram matrix, which is similar to incremental linear discriminant analysis, but simpler. Specifically, for each large language model , calculate the Gram matrix of the features of all training samples generated by it according to the above formula (3) , the Gram matrix captures the feature correlation of text generated based on the large language model.

[0145] (2) Global prototype decorrelation

[0146] The application example of this application performs global decorrelation based on the aforementioned formula (4) to eliminate the prototypes associated with all large language models observed so far The overall correlation between .

[0147] (3) Local prototype decorrelation

[0148] In addition to global decorrelation, the application example of this application also performs local decorrelation based on the above formula (5) to further eliminate the pairwise correlation between prototypes related to any two specific large language models. and its prototype ,Local decorrelation is performed in a similar way to global decorrelation, and this local decorrelation process further enhances the distinguishability of the large language model to the prototype.

[0149] (4) Prototype matching in two training stages

[0150] In the inference training phase, given the global and local decorrelated prototypes, the goal of this application example is to predict any unseen text Tags , in order to track the source of the text. To achieve this goal, this application example designs a two-step prototype matching process. Specifically, given , this application example first extracts the feature vector , and based on the above formula (6) With all Global Decorrelation Prototype for Large Language Models Matching,The two globally de-correlated prototypes that result in the highest global matching,scores are selected as candidate prototypes, representing the two most likely,sources of the globally decorrelated prototype predictions.

[0151] Afterwards, the application example of this application is based on the above formula (7) The locally decorrelated prototypes corresponding to these two candidate prototypes are matched to further distinguish them.

[0152] Then, this application example selects the candidate prototype with a higher local matching score as the final prediction (i.e., the target prototype): This two-step matching process achieves coarse-to-fine prototype selection, thereby improving the accuracy of prototype-based source tracking. Figure 8 , where "for...do" represents a loop structure and "end for" indicates the end of the loop structure. GLPD is training-agnostic; the pre-trained language model and random up-projection layer are frozen and shared across all learning stages.

[0153] That is, the application example of the present application first solves the problem of frequent retraining caused by the fixed label set of the traditional traceability method, and in view of the limitation of the continuous learning method in the traceability task, the application example of the present application proposes a generated text continuous traceability method based on global and local prototype decorrelation, which eliminates the overall correlation between all prototypes by using a global decorrelation mechanism, and further separates the subtle interference between pairs of prototypes by using a local decorrelation mechanism, thereby significantly improving the discrimination ability of different large language model generated texts, and after learning ten large language models, Figure 9 The similarity between the prototypes of the large language models after local decorrelation, where the darker the green square color represents the higher the similarity, and it can be seen that after overall decorrelation of the prototypes of the ten large language models, the similarity between the prototypes of the large language models is still high. Among them, Human represents human identity and also serves as a type of large language model, GPT-2, GPT-3, Cohere, ChatGPT, Cohere-Chat, GPT-4, MPT, MPT-Chat and LLaMA-Chat represent different types of large language models.

[0154] Therefore, by combining global and local decorrelation, the large language model generated text can be well traced. At the same time, the method uses an incremental learning mechanism, so that the model can quickly adapt to and identify newly emerging large language model generated texts while effectively preserving the existing large language model category knowledge, without frequent retraining.

[0155] The application example of the present application based on global and local prototype decorrelation of generated text continuous traceability method can solve the problem of frequent retraining caused by the fixed label set of the traditional traceability method, and can also solve the problem of high similarity between the prototypes of the large language models and the inability to remove the correlation between the prototypes of the large language models in fine granularity based on the prototype of the continuous learning method.

[0156] In summary, the application example of the present application effectively deals with the problem of large language model generated text traceability that people are increasingly concerned about. Unlike previous research that relies on fixed label set classifiers, the application example of the present application sets this task as a class incremental learning process in continuous learning, so that the model can quickly adapt to the emerging new large language model without frequent retraining. The application example of the present application designs a method that does not need to be trained, continuously extracts new large language model prototypes using a frozen pre-trained language model, and uses global and local prototype decorrelation technology to improve the accuracy of large language model generated text traceability.

[0157] In addition, this application example constructs a COT-Bench benchmark for evaluating language model performance. It includes text data generated by 19 large language models from 12 vendors in eight different fields. This application example labels the 19 large language models in chronological order according to their release dates and lets the models learn sequentially according to the labeling order. This simulates the situation where large language models appear one after another over time, providing a realistic test environment for evaluation.

[0158] This application example evaluates the proposed method on the COT-Bench benchmark and compares it with several existing continuous learning methods. The experimental results are shown in Table 1. LwF, Ease, SimpleCIL, Aper, and RanPAC are continuous learning methods, while T5-Sentinel, LeCNN, DeTeCTive, and JointLearning are non-continuous learning methods. Non-continuous learning methods, which use historical and current data to retrain the classifier at each training stage, can be considered the upper limit of the performance of continuous learning methods.

[0159] The comparison table between the above methods is used to show the final accuracy of different methods across 8 fields on COT-Bench. F and the average accuracy Ā. "Avg." represents the arithmetic mean across the eight domains. The best performance is indicated in bold. Due to the large number of columns in the comparison table, it is split into Table 1-1 and Table 1-2. In the complete comparison table, the recipe column in Table 1-2 is to the right of the poem column in Table 1-1.

[0160] Table 1-1

[0161]

[0162] Table 1-2

[0163]

[0164] Among them, continuous learning methods include:

[0165] 1. LwF is a regularization-based method that injects prior knowledge by distilling the output of the previous model when learning new data.

[0166] 2. Ease is an architecture-based approach that learns a lightweight adapter for each training stage to adapt to the new data distribution. By concatenating the features of the pre-trained model and each adapter, joint decision-making can be made in multiple subspaces.

[0167] 3. SimpleCIL is a prototype-based method that uses a frozen pre-trained model to extract large language model prototypes and traces them through simple prototype matching.

[0168] 4. Aper is a prototype-based method that fine-tunes the adapter of a pre-trained model in the first training phase and then freezes the adapter in subsequent training phases to enhance the representation power of the extracted prototypes.

[0169] 5. RanPAC is a prototype-based method that enhances the linear separability of prototypes by applying global prototype decorrelation.

[0170] Non-continuous learning methods include:

[0171] 1. T5-Sentinel fine-tunes the T5-large model and leverages its next-word prediction capability for source tracking.

[0172] 2. LeCNN uses LLaMA-2-7B to extract text embeddings and trains a CNN classifier on these embeddings for multi-class classification.

[0173] 3. DeTeCtive uses the BERT-based pre-trained language model RoBERTa-base model with a multi-level contrastive learning and dense retrieval framework to identify the source of the text. BERT is a bidirectional encoding representation based on Transformer, and Transformer is a neural network model based on the self-attention mechanism.

[0174] 4. JointLearning directly fine-tunes the LLaMA-3.2-1B model to trace the source of the large language model to generate text.

[0175] Experimental results demonstrate that GLPD consistently outperforms all baselines on datasets across all eight domains, and outperforms the state-of-the-art RanPAC continual learning method. Across all eight domains, GLPD achieves superior average performance compared to state-of-the-art provenance methods that follow a non-continuous learning paradigm. Furthermore, as a training-free method, GLPD approaches the performance of JointLearning (with an overall average gap of only 2.02% A_F and 1.17% Ā), also known as joint learning. Joint learning, serving as an upper bound for continual learning methods and employing the same backbone network as GLPD, further demonstrates its strong performance and demonstrates GLPD's significant potential for new tasks. These results highlight the importance and feasibility of continuous provenance for large language models and provide a promising direction for future research.

[0176] From the software level, this application also provides a large language model to generate text continuous traceability model training device for executing all or part of the large language model to generate text continuous traceability model training method, see Figure 10 The large language model-generated text continuous tracing model training device specifically includes the following contents:

[0177] The stage feature extraction model 10 is used to perform the stage feature extraction step: inputting each training sample in the data set corresponding to the current training stage in this model training process into the feature extraction unit, so that the feature extraction unit extracts the feature vector corresponding to each text data in each training sample; wherein each training sample also includes a label for indicating the type of the large language model that generated the text data; the release time of the large language model corresponding to the current training stage is later than the release time of the large language model corresponding to each historical training stage; and according to the feature vector of each text data and the label corresponding to each text data, respectively obtaining the initial prototype and text feature correlation data of each large language model;

[0178] The global and local decorrelation module 20 is used to, if the current training stage is the last training stage in this model training process, perform global decorrelation processing and local decorrelation processing on each of the initial prototypes obtained in sequence in each of the historical training stages and the current training stage according to the text feature correlation data of each of the large language models obtained in sequence in each of the historical training stages and the current training stage, so as to obtain decorrelation prototypes corresponding to each of the large language models, so as to generate a large language model-generated text continuous tracing model for predicting the type of the large language model to which the generated text data belongs and including the feature extraction unit based on the current decorrelation prototypes and the label group including the labels corresponding to each of the decorrelation prototypes.

[0179] The embodiment of the device for training a continuous tracing model for text generation from a large language model provided in this application can be specifically used to execute the processing flow of the embodiment of the method for training a continuous tracing model for text generation from a large language model in the above-mentioned embodiment. Its functions will not be described in detail here, and reference can be made to the detailed description of the embodiment of the method for training a continuous tracing model for text generation from a large language model.

[0180] The part of the large language model generated text continuous tracing model training device that performs the large language model generated text continuous tracing model training can be completed in the server or client device. The specific selection can be based on the processing power of the client device, the limitations of the user's usage scenario, etc. This application is not limited to this. If all operations are completed in the client device, the client device may also include a processor for the specific processing of the large language model generated text continuous tracing model training.

[0181] The aforementioned client device may include a communication module (i.e., a communication unit) capable of establishing a communication connection with a remote server to facilitate data transmission with the server. The server may include a server at the task scheduling center or, in other implementation scenarios, a server on an intermediate platform, such as a server on a third-party server platform that is communicatively linked to the task scheduling center server. The server may comprise a single computer device, a server cluster consisting of multiple servers, or a distributed server configuration.

[0182] The server and the client device may communicate using any suitable network protocol, including network protocols that have not yet been developed as of the filing date of this application. Examples of such network protocols include TCP / IP, UDP / IP, HTTP, and HTTPS. Furthermore, examples of such network protocols include RPC (Remote Procedure Call Protocol) and REST (Representational State Transfer) protocols, which are used on top of the aforementioned protocols.

[0183] From the above description, it can be seen that the large language model provided in the embodiment of the present application generates a continuous tracing model training device for text, which can not only solve the problem of frequent retraining caused by the fixed label set of the traditional tracing method, but also solve the problem that the similarity between the prototypes of the large language model is too high and the correlation between the prototypes cannot be removed in a fine-grained manner.

[0184] From the software level, this application also provides a large language model generated text continuous tracing device for executing all or part of the large language model generated text continuous tracing method, see Figure 11 The large language model-generated text continuous tracing device specifically includes the following contents:

[0185] The model prediction module 30 is used to input the target text data into the large language model generated text continuous tracing model obtained by the large language model generated text continuous tracing model training method, so that the feature extraction unit in the large language model generated text continuous tracing model extracts the target feature vector corresponding to the target text data, so that the large language model generated text continuous tracing model matches the target feature vector with each of the decorrelation prototypes respectively to obtain the target prototype corresponding to the target text data, so that the large language model generated text continuous tracing model searches for the label corresponding to the target prototype in the label group and outputs the label as the tracing prediction result of the target text data.

[0186] The embodiment of the device for continuously tracing the source of text generated by a large language model provided in this application can be specifically used to execute the processing flow of the embodiment of the method for continuously tracing the source of text generated by a large language model in the above-mentioned embodiment. Its functions will not be described in detail here, and reference can be made to the detailed description of the embodiment of the method for continuously tracing the source of text generated by a large language model in the above-mentioned embodiment.

[0187] The part of the large language model generated text continuous tracing device that performs the large language model generated text continuous tracing can be completed in the server or client device. The specific selection can be based on the processing power of the client device, the limitations of the user's usage scenario, etc. This application is not limited to this. If all operations are completed in the client device, the client device may also include a processor for the specific processing of the large language model generated text continuous tracing.

[0188] An embodiment of the present application further provides an electronic device, which may include a processor, a memory, a receiver, and a transmitter. The processor is used to execute the large language model-generated text continuous tracing model training method and / or the large language model-generated text continuous tracing method mentioned in the above embodiments, wherein the processor and the memory may be connected via a bus or other means, with bus connection being an example. The receiver may be connected to the processor and the memory via a wired or wireless manner.

[0189] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0190] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs and modules, such as program instructions / modules corresponding to the large language model generated text continuous tracing model training method and / or the large language model generated text continuous tracing method in the embodiments of the present application. The processor performs various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, implements the large language model generated text continuous tracing model training method and / or the large language model generated text continuous tracing method in the above method embodiments.

[0191] The memory can include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required by at least one function; and the data storage area can store data created by the processor and the like. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged with respect to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0192] The one or more modules are stored in the memory and, when executed by the processor, perform the large language model generated text continuous tracing model training method and / or the large language model generated text continuous tracing method in the embodiments.

[0193] In some embodiments of the present application, the user equipment can include a processor, a memory and a transceiver unit which can include a receiver and a transmitter, the processor, the memory, the receiver and the transmitter can be connected through a bus system, the memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to transceive signals.

[0194] As an implementation manner, the functions of the receiver and the transmitter in the present application can be implemented by considering a transceiver circuit or a transceiver dedicated chip, and the processor can be implemented by considering a dedicated processing chip, a processing circuit or a general-purpose chip.

[0195] As another implementation manner, the server provided by the embodiments of the present application can be implemented by considering using a general-purpose computer. That is, program codes for implementing the functions of the processor, the receiver and the transmitter are stored in the memory, and the general-purpose processor implements the functions of the processor, the receiver and the transmitter by executing the codes in the memory.

[0196] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements the steps of the aforementioned large language model-generated text continuous traceability model training method and / or large language model-generated text continuous traceability method. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

[0197] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the aforementioned large language model-generated text continuous tracing model training method and / or large language model-generated text continuous tracing method.

[0198] It should be understood by those skilled in the art that the various exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether it is implemented in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium or transmitted on a transmission medium or communication link via a data signal carried in a carrier.

[0199] It should be understood that the present application is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present application is not limited to the specific steps described and illustrated. Those skilled in the art can make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present application.

[0200] In this application, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or replace features of other embodiments.

[0201] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Those skilled in the art will appreciate that various modifications and variations of the present embodiment are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A method for training a continuous tracing model for text generated by a large language model, characterized in that: include: Stage feature extraction step: inputting each training sample in the data set corresponding to the current training stage in this model training process into a feature extraction unit, so that the feature extraction unit extracts the feature vector corresponding to each text data in each training sample; wherein each training sample also includes a label for indicating the type of the large language model that generated the text data; the release time of the large language model corresponding to the current training stage is later than the release time of the large language model corresponding to each historical training stage; and, based on the feature vector of each text data and the label corresponding to each text data, respectively obtaining the initial prototype and text feature correlation data of each large language model; If the current training stage is the last training stage in this model training process, then according to the text feature correlation data of each of the large language models obtained in sequence in each of the historical training stages and the current training stage, global decorrelation processing and local decorrelation processing are performed on each of the initial prototypes obtained in sequence in the historical training stages and the current training stage to obtain decorrelation prototypes corresponding to each of the large language models, so as to generate a large language model-generated text continuous tracing model for predicting the type of the large language model to which the generated text data belongs and including the feature extraction unit based on the current decorrelation prototypes and the label group including the labels corresponding to each of the decorrelation prototypes; The step of obtaining the initial prototype and text feature correlation data of each of the large language models according to the feature vectors of each of the text data and the labels corresponding to each of the text data includes: According to the labels corresponding to the respective text data, the feature vectors of the respective text data belonging to the same large language model are summed up to obtain the initial prototypes of the respective large language models; Furthermore, according to the labels corresponding to the respective text data, the feature vectors of the respective text data belonging to the same large language model are calculated using a Gram matrix to obtain the Gram matrix corresponding to each of the large language models as the text feature correlation number.

2. The method for training a large language model to generate text continuous tracing model according to claim 1 is characterized in that: Before the feature extraction step, the stage also includes: Collect human-written text data and text data generated by various large language models that have been released; Using each manually written text data as each text data of the initial large language model, and setting a label for indicating the type of the large language model for the initial large language model and each of the published large language models; The label of the initial large language model is used as the first label, and the labels of each of the published large language models are sorted in descending order of release time after the label of the initial large language model to obtain a corresponding label set; According to the order of each label in the label set, the data sets corresponding to each label are constructed in sequence, wherein the data sets contain the text data corresponding to the labels, and each text data in the data sets and the corresponding labels constitute different training samples.

3. The method for training a large language model to generate text continuous tracing model according to claim 1, characterized in that: Before the feature extraction step, the stage also includes: If a large language model that has not been applied in the historical training period is detected or received, a label is set to uniquely represent the type of the large language model, and multiple text data generated using the large language model are obtained to generate a data set for the current training stage; wherein the data set for the current training stage includes multiple training samples.

4. The method for training a large language model to generate text continuous tracing model according to claim 1, characterized in that: The feature extraction unit includes: a pre-trained language model with fixed parameters and a random upward projection layer with fixed parameters; Correspondingly, the process of inputting each training sample in the data set corresponding to the current training stage in the model training process into the feature extraction unit, so that the feature extraction unit extracts the feature vector corresponding to each text data in each training sample, including: The text data corresponding to each training sample in the data set corresponding to the current training stage in this model training process are respectively input into the pre-trained language model, so that the pre-trained language model outputs the initial feature vector corresponding to each of the text data, and then the random upward projection layer performs upward projection processing on each of the initial feature vectors to obtain the feature vector corresponding to each of the text data.

5. The method for training a large language model to generate text continuous tracing model according to claim 1, characterized in that: The method of performing global decorrelation processing and local decorrelation processing on each of the initial prototypes sequentially obtained in each of the historical training stages and the current training stage based on the text feature correlation data of each of the large language models sequentially obtained in each of the historical training stages and the current training stage to obtain decorrelation prototypes corresponding to each of the large language models includes: Performing global decorrelation processing on each of the initial prototypes sequentially obtained in each of the historical training stages and the current training stage based on the text feature correlation data of each of the large language models sequentially obtained in each of the historical training stages and the current training stage to obtain prototypes corresponding to each of the large language models from which global correlation has been eliminated; Furthermore, based on the text feature correlation data of each of the large language models sequentially obtained in each of the historical training stages and the current training stage, each of the two initial prototypes is subjected to local decorrelation processing to obtain prototypes corresponding to each of the large language models from which local correlation has been eliminated; The prototypes from which global correlation has been eliminated and the prototypes from which local correlation has been eliminated, which correspond to each of the large language models, are respectively used as decorrelation prototypes which correspond to each of the large language models.

6. A method for continuously tracing text generated by a large language model, characterized in that: include: The target text data is input into the large language model generated text continuous tracing model obtained by training the large language model generated text continuous tracing model training method according to any one of claims 1 to 5, so that the feature extraction unit in the large language model generated text continuous tracing model extracts the target feature vector corresponding to the target text data, so that the large language model generated text continuous tracing model matches the target feature vector with each of the decorrelated prototypes respectively to obtain the target prototype corresponding to the target text data, so that the large language model generated text continuous tracing model searches for the label corresponding to the target prototype in the label group and outputs the label as the tracing prediction result of the target text data.

7. The method for continuously tracing text generated by a large language model according to claim 6 is characterized in that: The decorrelation prototype includes: a prototype having eliminated global correlation and a prototype having eliminated local correlation; Matching the target feature vector with each of the decorrelation prototypes to obtain a target prototype corresponding to the target text data includes: Matching the target feature vector with each of the prototypes whose global correlation has been eliminated, respectively, to obtain a global matching score corresponding to each of the prototypes whose global correlation has been eliminated; sorting the prototypes whose global correlations have been eliminated in descending order of the global matching scores, and taking the first two prototypes whose global correlations have been eliminated as candidate prototypes; Matching the target feature vector with the prototypes from which local correlations have been eliminated, respectively, corresponding to each of the candidate prototypes, to obtain local matching scores corresponding to each of the two candidate prototypes; The candidate prototype with the highest local matching score is determined as the target prototype corresponding to the target text data.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements the method for training a large language model for continuously tracing text generated by the large language model as described in any one of claims 1 to 5, and / or implements the method for continuously tracing text generated by the large language model as described in claim 6 or 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the method for training a model for continuously tracing text generated by a large language model as described in any one of claims 1 to 5, and / or implements the method for continuously tracing text generated by a large language model as described in claim 6 or 7.

Images