Method and system for learning implicit mechanism of Train DLenGap to assist large language model development

Abstract

According to the method, a Train DLenGap problem is identified and named according to data loading function calling of a Transform submodule, the reason of the problem is found, and the Train DLenGap problem is named as an implicit mechanism; according to the method, a FuncPLP data set is constructed to represent a Train DLenGap problem, and a TransfClsf classifier is created to learn the characteristics of the FuncPLP data set; experimental results show that the accuracy rate of the TransfClsf classifier is 100% by taking an original sixteen-dimensional FuncPLP data set as an input; according to the method, PCA and tSNE are used for carrying out dimension reduction operation on a FuncPLP data set to generate two different two-dimensional data sets, the two-dimensional data sets serve as input of a TransfClsf classifier, four different kernel methods including a linear kernel, a Gaussian kernel, a Sigmoid kernel and a polynomial kernel are used, hyper-parameters C and gamma are set to be different values, and the highest accuracy of the classifier is 99.2188%; according to the method, the Train DLenGap problem is provided, and the FuncPLP data set and the TransfClsf classifier are constructed, so that programmers can quickly and accurately develop a large language model.

Description

Technical Field

[0001] This invention relates to the Transformer submodule for large language model development. During data loading, the train_dataloader len value of 2638 appears inconsistent with the train_dataset len ​​value of 168891, but is actually consistent. It particularly relates to implicit mechanisms, namely decorators and class inheritance in the Python programming language. Background Art

[0002] The TransfClsf classifier includes SVM algorithms with linear and nonlinear kernels; using the original 16-dimensional FuncPLP dataset as input to the TransfClsf classifier, the accuracy was 100%; using two different methods to generate two-dimensional data by applying PCA and tSNE; using linear kernels, Gaussian kernels, Sigmoid kernels, and polynomial kernels, and setting the hyperparameters C and γ to different values, using two two-dimensional datasets as input to the TransfClsf classifier, the highest accuracy achieved was 99.2188%.

[0003] Pure Function and Functional Programming are a set of complementary concepts; the result of Pure Function depends only on its input and has no side effects; Python language also supports Functional Programming; Python functions have the following four characteristics: 1. First-Class objects: functions can be passed and used as parameters, in particular, functions are represented by name only without parentheses; 2. Inner functions: defined inside other functions; the definition of inner functions makes them order-independent; inner functions are limited to local variables of the parent function and are only valid in the local scope; 3. Function as return value: functions can be returned from other functions, for example, inner functions can return certain values; 4. Anonymous functions using lambda: anonymous functions are created using lambda expressions, which are suitable for simple, one-time functions; Python decorators are a feature of the programming language that allow programmers to dynamically modify the behavior of functions or classes.

[0004] The Transformer is a deep learning model that combines feedforward layers and self-attention layers. A typical Transformer model can be constructed by stacking feedforward layers and head attention modules. The input consists of T * D tokens, where T represents the total number of tokens and D represents the size of the token embedding. Typical application scenarios of the standard Transformer model include computer vision and natural language processing, namely Vision Transformers and Language Transformers. Existing research results have shown that the Vision Transformer (ViT) and its variants can perform a variety of computer vision tasks with significant success. The original ViT model originally split an image into multiple segments and fed them into the Transformer encoder. However, the ViT model's main focus is on in-sample annotation, which can reduce the performance of the Transformer model. Summary of the Invention

[0005] The present invention constructs the FuncPLP dataset and the TransfClsf classifier, and analyzes the FuncPLP dataset to solve the TrainDLenGap problem that occurs during the debugging process of large language model codes, and assists programmers in developing large language models more quickly and accurately.

[0006] In order to achieve the above objectives, the present invention provides a method for constructing a FuncPLP dataset, which comprises the following steps: Design the dataset features and divide the features into three groups: PyDec, ClsInh, and implicit mechanism summary.

[0007] The first group of features has a total of 7; PyDec stands for Python Decorator, indicating that this group of features is related to Python decorators; Programmable Language is related to the FunProgram and PureFunc fields; in particular, Programmable Language includes a group of features, such as FClsObj, InnerFunc, FuncARetVal, and AnonyFuncLam; they represent First-Class objects, inner functions, functions as return values, and anonymous functions using lambda, respectively.

[0008] The second group of features has a total of 7; ClsInh represents class inheritance, indicating that this group of features are all related to the class inheritance mechanism of object-oriented programming languages; for Python classes, InitField and NewField represent two different initialization processes; objects are created and initialized through two class initialization methods; LenTSampler, LenTDataloader, LenTDataset, and LenBSampler represent debugging statement variables related to the Transformer module in the large language model; LenTSampler represents the length of the train_sample variable; similarly, the other three fields calculate the length of the train_dataloader, train_dataset, and batch_sampler variables.

[0009] The third set of features is named Implicit Mechanism Summary; ImplicitMech represents the implicit mechanism and CorrectVAR represents the correct variable value.

[0010] The present invention constructs a TransfClsf classifier to solve the TrainDLenGap problem in the debugging process of large language model code, including the following steps: The first submodule of the TransfClsf classifier takes the original 16-dimensional FuncPLP dataset as input and achieves an accuracy of 100%. By reducing the dimensionality of the original FuncPLP dataset using PCA and tSNE, the second submodule of the TransfClsf classifier takes two-dimensional data as input and achieves a maximum accuracy of 99.2188%. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The present invention is further described below with reference to the accompanying drawings and examples.

[0012] Figure 1 The decorator mechanism for the Python programming language.

[0013] Figure 2 Schematic diagram of the TrainDLenGap problem.

[0014] Figure 3 are the features and labels of the FuncPLP dataset.

[0015] Figure 4 For the TransfClsf architecture. Specific implementation methods

[0016] Figure 1 The decorator mechanism for the Python programming language.

[0017] Pure functions and functional programming are complementary concepts. A pure function is a function whose output value depends entirely on its input value, without any observable side effects. In functional programming, a program primarily consists of evaluating pure functions. Computations are performed through nested or combined function calls, without changing state or modifying data. Many programming languages ​​support functional programming to some degree.

[0018] Python also supports Functional Programming. At the same time, Function in Python may have side effects instead of just converting input to output.

[0019] First-class objects are one of the concepts in functional programming, which means that functions are passed and used as parameters in a similar way to other objects (such as int, float, str, list, etc.); specifically, there are no parentheses after the function name, which simply refers to the function represented by the name.

[0020] Inner functions are functions defined inside other functions; the order in which inner functions are defined does not matter; calling an inner function produces a result, just like calling any other function, but only when the inner function is actually executed; furthermore, inner functions are not defined until the parent function is called; they are scoped as local variables within the parent function and are only available within its local scope.

[0021] Functions that return values ​​allow code to return functions from other functions; for example, inner functions can return some value.

[0022] Anonymous functions using lambda are also one of the functions of Python functions; calling and passing functions is a feature of functional programming, which requires the design of a large number of functions; the def keyword can be used to define functions in a standard way; when the program is running, if the function name is not given, lambda can be used to define an anonymous function. Similar to the function defined by the def keyword, the lambda expression is a callable function.

[0023] Figure 2 Schematic diagram of the TrainDLenGap problem.

[0024] The Transformer package defines the Trainer class, which also defines the _inner_training_loop member function. Within the _inner_training_loop member function, another member function of the Trainer class, get_train_dataloader, is called and its return value is assigned to the train_dataloader variable. The len value of the train_dataloader variable is 2638. Before the get_train_dataloader member function returns, the len value of the train_dataset variable is 168891, which is inconsistent with the len value of 168891. The last statement of the get_train_dataloader member function is a return statement, which calls the DataLoader class initialization function. However, the final statement of the initialization function does not call a return statement. Moreover, at the end of the initialization function, the len value of the train_dataset variable is 168891. Therefore, according to the explicit call statement, the len value of the train_dataloader variable, 2638, is inconsistent with the len value of the train_dataset variable, 168891, which is inconsistent with the result of the explicit call.

[0025] Figure 2 The implicit mechanism consists of two modules: Python decorator: a mechanism used to modify the behavior of functions or classes in the Python programming language; class inheritance: a mechanism in object-oriented programming where a subclass inherits the properties and methods of the parent class.

[0026] Figure 3 are the features and labels of the FuncPLP dataset; the first group represents the Python decorator module: PureFunc input can determine its output without producing any further results; Python belongs to the category of functional programming (FuncProgram), which means that functional programming is a mechanism for implementing Python functions; functions in Python have the function of calculating input as output results, and may also produce other results; in particular, FuncProgram contains many functions, and the present invention lists four of them: FClsObj, InnerFunc, FuncARetVal, AnonyFuncLam; they respectively represent First-Class objects, inner functions, functions as return values, and anonymous functions using lambda.

[0027] The second group represents the class inheritance mechanism: it includes seven features; the first feature is class inheritance, which means that all these features are associated with the class inheritance mechanism of object-oriented programming languages; for Python classes, the initialization field and the New field represent two different initialization processes; two class initialization methods are used to create and initialize objects; the debugging statements related to the Transformer module in the large language model are represented by the training sample length, training data loader length, training dataset length, and Batch sample set length respectively; the training sample length represents the length of the calculated training sample variable; in a similar way, the lengths of the other three different variables, training loader, training dataset, and Batch sample set are calculated by the representations of the other three fields.

[0028] The third group represents GapMatch: the implicit mechanism represents the two mechanisms of Python decorator and class inheritance. The correct variable value is represented by explaining the difference, calculating the value of the correct variable, and masking the values ​​of irrelevant variables.

[0029] The value of each of the sixteen features listed above is 0 or 1.

[0030] Figure 4 It is the TransfClsf architecture; it includes the first and second submodules of TransfClsf; the first submodule takes the original sixteen-dimensional FuncPLP dataset as input; uses two dimensionality reduction methods, PCA and tSNE, with the original sixteen-dimensional FuncPLP dataset as input to generate a two-dimensional dataset, which is then used as the input of the second submodule of TransfClsf, and uses four different kernel methods: linear kernel, Gaussian kernel, Sigmoid kernel, and polynomial kernel to train and test the accuracy of the classifier.

[0031] The TransfClsf algorithm was implemented using the Python programming language and the sklearn software package. Experimental results show that the accuracy of TransfClsf is 100% when the original 16-dimensional FuncPLP dataset is used as input. The maximum accuracy of the TransfClsf classifier is 99.2188% when a two-dimensional dataset is generated using the PCA and tSNE methods and used as input.

Claims

1. A method and system for learning the implicit mechanism of TrainDLenGap to assist in the development of large language models, characterized by: The following steps are involved: During the debugging of the large language model code, the TrainDLenGap issue was identified and named. The Transformer submodule is used to build the neural network model and load data. When loading data, the output size of the loaded data at different locations was displayed inconsistently, but was actually consistent. Understand the causes of the TrainDLenGap problem; Construct the FuncPLP dataset. Based on the causes and specific variables of the TrainDLenGap problem encountered during the debugging of the Transformer submodule, the FuncPLP dataset is designed. It includes 16 features and a label to represent the TrainDLenGap problem. Use PCA and tSNE to generate two-dimensional and three-dimensional versions of the FuncPLP dataset. After dimensionality reduction, the data distribution is visualized to compare the performance of the classifiers. The original 16-dimensional FuncPLP dataset was used as the input of the TransfClsf classifier, and the training accuracy of TransfClsf was 100%. Using linear kernels, Gaussian kernels, polynomial kernels, and Sigmoid kernels, the hyperparameters C and γ were set to different values. By applying PCA and tSNE methods to generate two-dimensional data as the input of the TransfClsf classifier, the highest accuracy achieved was 99.2188%.

2. The method and system for identifying the TrainDLenGap problem by learning an implicit mechanism of TrainDLenGap to assist in the development of a large language model according to claim 1, characterized in that: The following steps are involved: During program debugging, the processed 6mA dataset of size 211115 was used. The trainer class defined the _inner_training_loop function in the Transformer package. The get_train_dataloader member function of the Trainer class was called in the _inner_training_loop function, and its return value was assigned to the train_dataloader variable. The len value of the train_dataloader variable was 2638. Before the get_train_dataloader function returned, the len value of train_dataset was 168891. The values ​​2638 and 168891 were inconsistent. In Python, there are two different methods for class initialization: init and new, represented by init and new respectively. These methods are used to create and initialize objects; new is a static method belonging to the class itself, which is used to create and return a new instance of the class. new takes the class as its first argument and can accept any other arguments as needed; the init method is an instance method used to initialize a newly created object. It takes the object as its first argument and can also accept other arguments. The TrainDLenGap problem is related to decorators and class inheritance in the Python language. The two related reasons are represented by the implicit calling mechanism of member functions. The present invention names the problem TrainDLenGap.

3. The FuncPLP dataset construction method according to claim 1, wherein the method and system for learning the implicit mechanism of TrainDLenGap to assist in the development of large language models is characterized in that: The following steps are involved: Design sixteen features and one label, and divide the sixteen features into three groups; The first group includes FuncProgram, PureFunc, FClsObj, InnerFunc, FuncARetVal, and AnonyFuncLam. FunProgram is associated with PureFunc. After executing the function with PureFunc as input, there is no other result other than the result of running the current function statement. ClsObj, InnerFunc, FuncARetVal, and AnonyFuncLam are the four features of FuncProgram, representing first-class objects, inner functions, functions with return values, and anonymous functions with lambdas, respectively. The second group of features includes InitField, NewField, LenTSampler, LenTDataloader, LenTDataset, LenBSampler, all of which are associated with the class inheritance mechanism of object-oriented programming languages, where InitField and NewField represent two different initialization processes, LenTSampler, LenTDataloader, LenTDataset, and LenBSampler represent debugging statements related to the Transformer module in the large language model, and LenTSampler represents the length of the train sample variable; in a similar way, the other three fields calculate the len values ​​of three different variables (such as train_dataloader, train_dataset, batch_sampler); The third set of features is ImplicitMech and CorrectVAR, where ImplicitMech represents ImplicitMechanism and CorrectVAR represents Correct Variable. The solution to the TrainDLenGap problem is to determine the cause of the implicit mechanism and the value of the correct variable. Each of the sixteen features listed above has a value of 0 or 1. The label of the FuncPLP dataset is GapMatch. When GapMatch is 1, it means that the feature combination can accurately explain the TrainDLenGap problem. That is, the len value of the train_dataloader variable, 2638, and the len value of train_dataset, 168891, appear inconsistent, but are actually consistent. When GapMatch is 0, it means that the current feature combination cannot explain the TrainDLenGap problem.

4. The method and system for visualizing the FuncPLP dataset according to claim 1, wherein the method and system for learning the implicit mechanism of TrainDLenGap to assist in the development of large language models is characterized in that: The following steps are involved: The FuncPLP dataset constructed in the present invention is a 16-dimensional dataset; PCA and tSNE methods are selected to reduce the dimensionality of the FuncPLP data to two and three dimensions to demonstrate the characteristics of the FuncPLP dataset; after using the two dimensionality reduction methods, the distribution of the two-dimensional data is visualized using scatter plots. Compared with PCA, tSNE has a better separation effect; at the same time, a two-dimensional density map and a three-dimensional scatter plot are used to demonstrate the distribution of the FuncPLP dataset.

5. The method for constructing a TransfClsf classifier according to the method and system for learning the implicit mechanism of TrainDLenGap to assist in the development of a large language model according to claim 1, characterized in that: The following steps are involved: Using the support vector machine algorithm in the sklearn package as the basic module of TransfClsf, a TransfClsf classifier model consisting of two submodules was constructed: one classifier submodule took the original 16-dimensional dataset FuncPLP as input, while the other classifier submodule reduced the FuncPLP data to two dimensions by applying the PCA and tSNE methods. The two-dimensional data (i.e., the original 16-dimensional data was reduced to two dimensions) was used as input, and the linear kernel, Gaussian kernel, Sigmoid kernel, and Poly kernel were used to train and test the accuracy of the classifier. When the TransfClsf algorithm was implemented using Python and the Sklearn package, experimental results showed that the accuracy of the original 16-dimensional FuncPLP dataset as input of TransfClsf was 100%; when the PCA and tSNE methods were used to reduce the data to two dimensions respectively, the maximum accuracy of the TransfClsf classifier was 99.2188%.

Images