Large language model, model training method, device, medium, equipment and product

Abstract

The present invention relates to the field of artificial intelligence technology, and discloses a large language model, a model training method, an apparatus, a medium, a device and a product. The model training method comprises inputting a plurality of text data into a large language model after training convergence to obtain a linearly independent vector matrix; performing singular value decomposition on the linearly independent vector matrix to obtain a singular value matrix; determining the approximate maximum dimension of the linearly independent vector matrix based on the number of singular values ​​greater than a preset cutoff value; optimizing the dimension of the large language model based on the approximate maximum dimension of the linearly independent vector matrix to obtain the actual output vector dimension of the large language model; determining the number of neural network nodes based on the actual output vector dimension of the large language model, and performing model training on the large language model with the determined number of neural network nodes to obtain the large language model after dimension optimization. The present invention realizes the effective optimization of the vector dimension of the large language model, and improves the training and reasoning efficiency of the large language model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a large language model, a model training method, an apparatus, a medium, equipment and a product. Background Art

[0002] Neural networks are the most mainstream and widely used machine learning technology in artificial intelligence. The number of nodes in a neural network corresponds to the dimension of the vector in this layer. Only when the dimension is large enough can the performance of the model be guaranteed. However, when the dimension is too large, the model calculation efficiency will be low (in addition, there will be problems such as difficulty in training and overfitting). Therefore, it is particularly important to design a suitable dimension for the model.

[0003] With the continuous breakthroughs in generative large language models, the law of scaling has gradually been recognized, and the number of model parameters has become larger and larger. However, the vector dimension of large language models is difficult to determine and difficult to systematically optimize, which hinders the training and reasoning efficiency of the model. Summary of the invention

[0004] In view of this, the present invention provides a large language model, a model training method, an apparatus, a medium, a device and a product to solve the problem that the vector dimension of a large language model is difficult to determine and difficult to systematically optimize, which hinders the training and reasoning efficiency of the model.

[0005] In a first aspect, the present invention provides a large language model training method, the method comprising:

[0006] Acquire multiple text data, input the multiple text data into the large language model after training convergence, and obtain a linearly independent vector matrix;

[0007] Perform singular value decomposition on the linearly independent vector matrix to obtain a singular value matrix;

[0008] Obtaining the number of singular values ​​greater than a preset cutoff value in the singular value matrix, and determining an approximate maximum dimension of the linearly independent vector matrix based on the number of singular values ​​greater than the preset cutoff value;

[0009] Optimize the dimension of the large language model based on the approximate maximum dimension of the linearly independent vector matrix to obtain the actual output vector dimension of the large language model;

[0010] The number of neural network nodes is determined based on the actual output vector dimension of the large language model, and the large language model with the determined number of neural network nodes is trained to obtain a large language model with optimized dimension.

[0011] The present embodiment provides a large language model training method, which implements importance judgment of vector dimensions in the linearly independent vector matrix by performing singular value decomposition on the linearly independent vector matrix, determines the redundancy of the model output vector dimension, and then obtains the number of singular values ​​greater than a preset truncation value in the singular value matrix, determines the approximate maximum dimension of the linearly independent vector matrix based on the number of singular values ​​greater than the preset truncation value, and optimizes the dimension of the large language model using the approximate maximum dimension based on the linearly independent vector matrix, thereby achieving the acquisition of the output vector dimension of the large language model and the effective optimization of the vector dimension of the large language model, thereby improving the training and reasoning efficiency of the large language model and providing more theoretical basis for the size design of the large language model.

[0012] In an optional implementation, obtaining the number of singular values ​​greater than a preset cutoff value in a singular value matrix, and determining an approximate maximum dimension of a linearly independent vector matrix based on the number of singular values ​​greater than the preset cutoff value, includes:

[0013] Compare the diagonal elements in the singular value matrix with a preset cutoff value;

[0014] Obtaining the number of diagonal elements greater than a preset cutoff value, and using the number of diagonal elements greater than the preset cutoff value as the approximate maximum dimension of the linearly independent vector matrix;

[0015] Alternatively, if all diagonal elements in the singular value matrix are greater than a preset cutoff value, the model dimension of the large language model is expanded to determine the approximate maximum dimension of the linearly independent vector matrix.

[0016] The present embodiment provides a large language model training method, which compares the diagonal elements in the singular value matrix with a preset truncation value, and then implements singular value truncation based on the comparison result using the truncation value, determines the approximate maximum dimension of the linearly independent vector matrix by the number of diagonal elements, and uses a method for approximately finding the reasonable dimension of the large language model to achieve rapid determination of the vector dimension of the large language model, laying a foundation for dimensional optimization of the large language model.

[0017] In an optional implementation, if the diagonal elements in the singular value matrix are all greater than a preset cutoff value, the model dimension of the large language model is expanded to determine the approximate maximum dimension of the linearly independent vector matrix, including:

[0018] If the diagonal elements in the singular value matrix are all greater than the preset cutoff value, the additional dimension of the model is calculated based on the diagonal elements and the preset cutoff value;

[0019] Determine the approximate maximum dimension of the matrix of linearly independent vectors based on the total dimension of the singular value matrix and the additional dimensions added by the model.

[0020] This embodiment provides a large language model training method. When the diagonal elements in the singular value matrix are all greater than a preset cutoff value, the model dimension of the large language model is expanded so that the total dimension of the large language model conforms to the approximate maximum dimension of the linearly independent vector matrix, thereby optimizing the model dimension to a certain extent.

[0021] In an optional implementation, the approximate maximum dimension of the linearly independent vector matrix is ​​determined based on the total dimension of the singular value matrix and the newly added dimension of the model. The calculation formula of the approximate maximum dimension of the linearly independent vector matrix is ​​as follows:

[0022]

[0023]

[0024] Among them, r represents the remainder, represents the diagonal elements in the singular value matrix, Indicates the preset cutoff value, represents the approximate maximum dimension of a matrix of linearly independent vectors, represents the total dimension of the singular value matrix, Indicates that a new dimension is added to the model.

[0025] In an optional implementation, the dimension of the large language model is optimized based on the approximate maximum dimension of the linearly independent vector matrix to obtain the actual output vector dimension of the large language model, including:

[0026] Obtaining a computing hardware memory capacity, and comparing the computing hardware memory capacity with a memory of a current large language model; wherein the current large language model satisfies an approximate maximum dimension of a linearly independent vector matrix;

[0027] If the memory of the current large language model is larger than the memory capacity of the computing hardware, the actual output vector dimension of the large language model is determined based on the upper limit of the memory capacity of the computing hardware;

[0028] Alternatively, if the memory of the current large language model is smaller than the memory capacity of the computing hardware, the approximate maximum dimension of the linearly independent vector matrix is ​​used as the actual output vector dimension of the large language model.

[0029] This embodiment provides a large language model training method, which selects the upper limit of the computing hardware memory capacity or the approximate maximum dimension of the linearly independent vector matrix as the actual output vector dimension of the large language model by comparing the computing hardware memory capacity with the memory of the current large language model. The current output vector dimension of the large language model and the limitations of the computing hardware device are taken into consideration, so that the actual output vector dimension of the large language model is more in line with actual application requirements, thereby achieving effective optimization of the vector dimension of the large language model.

[0030] In an optional implementation, singular value decomposition is performed on the linearly independent vector matrix to obtain a singular value matrix, including:

[0031] Transpose the linearly independent vector matrix to obtain a transposed matrix;

[0032] Perform eigendecomposition on the product of the linearly independent vector matrix and the transposed matrix to obtain eigenvalues ​​and eigenvectors;

[0033] Determine a first orthogonal matrix and a second orthogonal matrix based on the eigenvalues ​​and the eigenvectors;

[0034] A singular value matrix is ​​constructed based on the linearly independent vector matrix, the first orthogonal matrix, and the second orthogonal matrix.

[0035] The present embodiment provides a large language model training method, which realizes the importance judgment of the vector dimensions of the linearly independent vector matrix by performing singular value decomposition on the linearly independent vector matrix, determines the redundancy of the dimension from a mathematical level, and lays a foundation for the subsequent dimensional optimization of the large language model.

[0036] In an optional implementation, before obtaining a plurality of text data and inputting the plurality of text data into a large language model after training convergence to obtain a linearly independent vector matrix, the method further includes:

[0037] Randomly select multiple sample data from a pre-set text database, and input the multiple sample data into the initialized large language model to obtain an output vector matrix;

[0038] Decompose the output vector matrix and use the decomposed matrix to construct the loss function;

[0039] The large language model is iteratively trained using the loss function until the large language model reaches the convergence condition, and the iteration is stopped to obtain the large language model after the training converges.

[0040] The present embodiment provides a large language model training method, which decomposes the output vector matrix corresponding to the initialized large language model, constructs a loss function using the decomposed matrix, and then iteratively trains the large language model using the loss function. When training the model, the loss function constructed by the decomposed matrix is ​​introduced, and the model is trained under this loss function, which improves the rank of the output vector matrix, realizes the initial training of the large language model, and lays a foundation for the subsequent optimization of the large language model dimension.

[0041] In an optional implementation, the output vector matrix is ​​decomposed, and the decomposed matrix is ​​used to construct a loss function, including:

[0042] Based on the output vector matrix, a diagonal matrix is ​​obtained using the singular value decomposition algorithm;

[0043] Construct a loss function based on a diagonal matrix and a preset cutoff value.

[0044] In an optional implementation, a loss function is constructed based on a diagonal matrix and a preset cutoff value, and the expression of the loss function is as follows:

[0045]

[0046] in, represents the loss function.

[0047] In an optional implementation, the output vector matrix is ​​decomposed, and the decomposed matrix is ​​used to construct a loss function, including:

[0048] Perform Schmidt orthogonalization on the output vector matrix to obtain the matrix eigenvalues;

[0049] Construct a loss function based on matrix eigenvalues ​​and preset cutoff values.

[0050] In a second aspect, the present invention provides a large language model, which is obtained according to the large language model training method of the first aspect or any corresponding embodiment thereof.

[0051] In a third aspect, the present invention provides a large language model training device, the device comprising:

[0052] An acquisition module is used to acquire multiple text data, input the multiple text data into the large language model after training convergence, and obtain a linearly independent vector matrix;

[0053] A singular value decomposition module is used to perform singular value decomposition on a linearly independent vector matrix to obtain a singular value matrix;

[0054] A determination module, used for obtaining the number of singular values ​​greater than a preset cutoff value in the singular value matrix, and determining the approximate maximum dimension of the linearly independent vector matrix based on the number of singular values ​​greater than the preset cutoff value;

[0055] A dimension optimization module is used to optimize the dimension of the large language model based on the approximate maximum dimension of the linearly independent vector matrix to obtain the actual output vector dimension of the large language model;

[0056] The model training module is used to determine the number of neural network nodes based on the actual output vector dimension of the large language model, and to perform model training on the large language model with the determined number of neural network nodes to obtain a large language model with optimized dimension.

[0057] In a fourth aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the large language model training method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0058] In a fifth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the large language model training method of the first aspect or any corresponding embodiment thereof.

[0059] In a sixth aspect, the present invention provides a computer program product, comprising computer instructions, wherein the computer instructions are used to enable a computer to execute the large language model training method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0061] Figure 1 is a schematic diagram of the structure of a neural network according to an embodiment of the present invention;

[0062] Figure 2 is a schematic diagram of a working process of a language model according to an embodiment of the present invention;

[0063] Figure 3 is a flowchart of a large language model training method according to an embodiment of the present invention;

[0064] Figure 4 is a schematic diagram of a neural network structure of a large language model according to an embodiment of the present invention;

[0065] Figure 5 is a flow chart of another large language model training method according to an embodiment of the present invention;

[0066] Figure 6 is a flowchart of another large language model training method according to an embodiment of the present invention;

[0067] Figure 7 is a flowchart of another large language model training method according to an embodiment of the present invention;

[0068] Figure 8is a flow chart of a method for optimizing the output vector dimension of a large language model according to an embodiment of the present invention;

[0069] Fig. 9 is a structural block diagram of a large language model training device according to an embodiment of the present invention;

[0070] Fig.10 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0072] like Figure 1 As shown in the figure, each circle in the Neural Networks (NN) corresponds to a neuron (or node), and each line corresponds to a weight parameter. Data is input into the neural network in the form of vectors or matrices. Generally speaking, each vector element corresponds to a neuron. After the vector is input into the neural network, the calculation is completed through multiple weighted lines, and then it undergoes a nonlinear transformation and is passed to the next layer. The vector of the input layer is generally called the input vector, the middle layer is called the hidden layer, which also corresponds to a hidden layer vector, and the vector of the output layer is called the output vector. Neural networks generally have multiple layers, and the output of the previous layer is used as the input of the next layer and is passed to the last layer.

[0073] The more nodes a neural network has, the deeper the number of layers, the stronger the expressive power, and the corresponding model performance. Therefore, artificial intelligence models all use larger neural networks, which require extremely high computing power. The number of nodes in a neural network corresponds to the dimension of the vector in that layer. Only when the dimension is large enough can the performance of the model be guaranteed. However, when the dimension is too large, the model calculation efficiency will be low (in addition, there will be problems such as difficulty in training and overfitting). Therefore, it is particularly important to design a suitable dimension for the model.

[0074] Figure 1What is displayed is a fully connected neural network, which is the most primitive neural network structure. At the same time, a variety of network models with special structures have been proposed, such as CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), Transformer (a deep learning architecture for natural language processing tasks), etc. Some of the above models have very complex architectures and require special training techniques.

[0075] With the continuous breakthroughs of generative large language models such as the GPT series, the law of scaling has gradually been recognized, and the number of model parameters has become larger and larger. Large language models refer to deep learning models trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. Figure 2 As shown in the figure, the language model is used to predict the next word, and the predicted word is output according to a certain probability based on the score. The specific steps are as follows: the input text is processed through a series of processes and passed to the neural network, and then the language model obtains a vector , then Perform matrix operation W to get a new vector ,vector The space corresponding to the vocabulary is the same size as the space corresponding to the vocabulary, which is the sum of all the words that can be output by the model. Performing SoftMax transformation (a mathematical function that is usually used to convert a set of arbitrary real numbers into real numbers representing probability distribution) can give the probability of the output word.

[0076] like Figure 2 As shown in the figure, with "what to eat today" as the input text, the model will finally get the probability table of each word in the vocabulary, among which the probability of "what" is the highest, which is 78%; "Jin" is second, which is 9%, and so on. For autoregressive models such as GPT (Generative Pre-Trained, a deep learning model for text generation based on the Internet and available data for training), "what" will be added to the previous sentence to get "what to eat today", and then it will be input into the model again, and the process will be repeated to get the next predicted word until it terminates.

[0077] However, neural networks are still a black box in general. There is a lack of a solid theoretical basis for how large a language model should be designed, and the "trial and error" method is generally still adopted, which makes it difficult to determine the dimension of the vector. In addition, the dimension of the vector in the model is a feature that is highly correlated with the model size, and there is a lack of systematic optimization methods.

[0078] In order to solve the above technical problems, the embodiments of the present invention provide a large language model training method. It should be noted that the execution subject of the large language model training method provided by the embodiments of the present invention can be a device for optimizing the dimension of a large language model. The device for optimizing the dimension of a large language model can be implemented as part or all of an electronic device through software, hardware, or a combination of software and hardware. The electronic device can be a server or a terminal. The server in the embodiments of the present application can be a single server or a server cluster composed of multiple servers. The terminal in the embodiments of the present application can be a smart phone, a personal computer, a tablet computer, a wearable device, an intelligent robot, and other intelligent hardware devices. In the following method embodiments, the execution subject is an electronic device as an example for explanation.

[0079] According to an embodiment of the present invention, an embodiment of a large language model training method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0080] In this embodiment, a large language model training method is provided, which can be used for the above-mentioned electronic device. Figure 3 is a flow chart of a large language model training method according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0081] Step S301, obtaining a plurality of text data, inputting the plurality of text data into a large language model after training convergence, and obtaining a linearly independent vector matrix.

[0082] Specifically, a large language model (LLM) refers to a deep learning model trained using a large amount of text data, which can generate natural language text or understand the meaning of language text; a large language model can handle a variety of natural language tasks, such as text classification, question and answer, dialogue, etc., and is an important path to artificial intelligence.

[0083] Furthermore, if Figure 4As shown in the figure, the neural network structure of the large language model currently generally adopts the Transformer architecture (a deep learning architecture used for natural language processing tasks), which is essentially a more complex connection method, in which the Input Embedding (input layer) converts the input text (such as "what to eat today") into a vector, which is then input into the neural network for processing; the structure of the neural network includes position encoding, "Transformer Block" part (that is, the basic unit of the deep learning model), normalization layer, linear layer and activation function Softmax, and then outputs probability distribution.

[0084] Furthermore, the core part of the neural network structure of the large language model is the "Transformer Block" part, which consists of two parts: Multi-Head Attention (multi-head attention layer) and Feed Forward (feedforward neural network layer).

[0085] Furthermore, for most language processing scenarios, the size of the matrix composed of linearly independent vectors is generally limited, so it is not necessary to process all the corpus, only a small amount of corpus needs to be processed. Therefore, in general, the text data is not less than 1.5 , Represents the output vector dimension of the large language model.

[0086] Furthermore, the output vector (ie, linearly independent vector) corresponding to each text data is used as a row vector to construct a linearly independent vector matrix.

[0087] Step S302, performing singular value decomposition on the linearly independent vector matrix to obtain a singular value matrix.

[0088] Step S303, obtaining the number of singular values ​​in the singular value matrix that are greater than a preset cutoff value, and determining the approximate maximum dimension of the linearly independent vector matrix based on the number of singular values ​​that are greater than the preset cutoff value.

[0089] Specifically, the singular values ​​in the singular value matrix represent the degree of linear independence of the vectors, that is, the smaller the singular value, the weaker the importance of the corresponding vector, that is, the dimension can be removed or compressed. Therefore, it is necessary to compare the truncated value with the singular value, and then determine the degree of dimensional redundancy of the model based on the comparison result, and then perform singular value truncation to determine the approximate maximum dimension of the linearly independent vector matrix.

[0090] Furthermore, Schmidt orthogonalization may be adopted and the similarity comparison of the vectors may be performed by setting a cutoff value or an approximate maximum dimension of a linearly independent vector matrix.

[0091] Furthermore, the QR algorithm, Jacobi algorithm, etc. can also be used to obtain the matrix eigenvalues, and the approximate maximum dimension of the linearly independent vector matrix can be obtained by setting the truncation value; the QR algorithm is an algorithm for solving the eigenvalues ​​and eigenvectors of a matrix by recursively decomposing the matrix into an orthogonal matrix Q and an upper triangular matrix R, and then calculating the eigenvalues ​​and eigenvectors by iteration; the Jacobi algorithm is an iterative method for solving linear equations and calculating the eigenvalues ​​and eigenvectors of real symmetric matrices. The basic idea of ​​the Jacobi algorithm is to gradually transform the original matrix into a diagonal matrix through a series of orthogonal transformations, so as to obtain the solution to the system of equations or the eigenvalues ​​and eigenvectors of the matrix.

[0092] Step S304 , optimizing the dimension of the large language model based on the approximate maximum dimension of the linearly independent vector matrix to obtain the actual output vector dimension of the large language model.

[0093] Specifically, referring to the approximate maximum dimension of the linearly independent vector matrix, according to the constraints such as computing hardware devices, the actual output vector dimension of the large language model (that is, the dimension of the output vector of the large language model) is determined.

[0094] Step S305, determining the number of neural network nodes based on the actual output vector dimension of the large language model, and performing model training on the large language model with the determined number of neural network nodes to obtain a large language model with optimized dimension.

[0095] The present embodiment provides a large language model training method, which implements importance judgment of vector dimensions in the linearly independent vector matrix by performing singular value decomposition on the linearly independent vector matrix, determines the redundancy of the model output vector dimension, and then obtains the number of singular values ​​greater than a preset truncation value in the singular value matrix, determines the approximate maximum dimension of the linearly independent vector matrix based on the number of singular values ​​greater than the preset truncation value, and optimizes the dimension of the large language model using the approximate maximum dimension based on the linearly independent vector matrix, thereby achieving the acquisition of the output vector dimension of the large language model and the effective optimization of the vector dimension of the large language model, thereby improving the training and reasoning efficiency of the large language model and providing more theoretical basis for the size design of the large language model.

[0096] In this embodiment, a large language model training method is provided, which can be used for the above-mentioned electronic device. Figure 5 is a flow chart of a large language model training method according to an embodiment of the present invention. Figure 5 As shown, the process includes the following steps:

[0097] Step S501, obtain multiple text data, input the multiple text data into the large language model after training convergence, and obtain a linearly independent vector matrix. For details, please refer to Figure 3Step S301 of the illustrated embodiment will not be described in detail here.

[0098] Step S502, performing singular value decomposition on the linearly independent vector matrix to obtain a singular value matrix. Specifically, the above step S502 includes:

[0099] Step S5021, transpose the linearly independent vector matrix to obtain a transposed matrix.

[0100] Specifically, assume that the linearly independent vector matrix is A , linearly independent vector matrix A The corresponding transposed matrix is A T .

[0101] Step S5022, perform eigendecomposition on the product of the linearly independent vector matrix and the transposed matrix to obtain eigenvalues ​​and eigenvectors.

[0102] Specifically, and Solve for eigenvalues ​​and eigenvectors; For example, Substitute into the following formula to solve:

[0103] (1)

[0104] in, represents the identity matrix, Represents the eigenvalue.

[0105] Furthermore, according to the above formula (1), we can calculate , and then Substitution Solve the formula to find each The corresponding eigenvector :

[0106] (2)

[0107] Step S5023: determine a first orthogonal matrix and a second orthogonal matrix based on the eigenvalues ​​and the eigenvectors.

[0108] Specifically, the eigenvalues ​​are sorted in descending order, and the first l characteristic values, according to the previous l The eigenvectors corresponding to the eigenvalues ​​construct the first orthogonal matrix and the second orthogonal matrix .

[0109] Step S5024: construct a singular value matrix based on the linearly independent vector matrix, the first orthogonal matrix and the second orthogonal matrix.

[0110] Specifically, the linearly independent vector matrix can be decomposed into the form of multiplication of the first orthogonal matrix, the diagonal matrix and the second orthogonal matrix. The formula for singular value decomposition is as follows:

[0111] (3)

[0112] Among them, the linearly independent vector matrix A is a The matrix of the first orthogonal matrix is a The matrix of is a The matrix of represents the singular value matrix, which is a The diagonal matrix of .

[0113] Furthermore, the first orthogonal matrix obtained in step S5023 is and the second orthogonal matrix Substituting into the above formula (3), we get the singular value matrix, the singular value matrix It can be expressed as:

[0114] (4)

[0115] in, , , , , Represents the diagonal elements of the singular value matrix.

[0116] Furthermore, the eigenvalues ​​calculated in step S5022 may be squared to obtain singular values, which may characterize the degree of linear correlation of the vectors. The singular values ​​may then be arranged in descending order to obtain diagonal elements of a singular value matrix, thereby constructing a singular value matrix.

[0117] Step S503 , obtaining the number of singular values ​​in the singular value matrix that are greater than a preset cutoff value, and determining the approximate maximum dimension of the linearly independent vector matrix based on the number of singular values ​​that are greater than the preset cutoff value.

[0118] Specifically, the above step S503 includes:

[0119] Step S5031, comparing the diagonal elements in the singular value matrix with the preset cutoff value.

[0120] Step S5032, obtaining the number of diagonal elements greater than a preset cutoff value, and taking the number of diagonal elements greater than the preset cutoff value as the approximate maximum dimension of the linearly independent vector matrix.

[0121] Specifically, obtain the value that satisfies the preset cutoff value. The smallest diagonal element of , according to the minimum diagonal element of Determine the number of diagonal elements (i.e., the number of singular values), and then As an approximate maximum dimension of a matrix of linearly independent vectors .

[0122] Furthermore, if the approximate maximum dimension of the linearly independent vector matrix is If the dimension of the large language model is smaller than that of the large language model, the large language model is directly compressed to dimension.

[0123] Step S5033, or, if all diagonal elements in the singular value matrix are greater than the preset cutoff value, the model dimension of the large language model is expanded to determine the approximate maximum dimension of the linearly independent vector matrix.

[0124] Specifically, if the diagonal elements in the singular value matrix are all greater than the preset cutoff value, it means that the dimension of the large language model is small, and the model dimension of the large language model should be expanded.

[0125] In some optional implementations, the above step S5033 includes:

[0126] Step a1: If the diagonal elements in the singular value matrix are all greater than the preset cutoff value, the additional dimension of the model is calculated based on the diagonal elements and the preset cutoff value.

[0127] Specifically, set the model's new dimension to , calculate the remainder of 2 using the following calculation formula :

[0128] (5)

[0129] in, represents the diagonal elements in the singular value matrix, , Indicates the preset cutoff value.

[0130] Step a2, determining the approximate maximum dimension of the linearly independent vector matrix based on the total dimension of the singular value matrix and the newly added dimension of the model.

[0131] Specifically, the calculation formula for the approximate maximum dimension of the linearly independent vector matrix is ​​as follows:

[0132] (6)

[0133] in, represents the approximate maximum dimension of a matrix of linearly independent vectors, represents the total dimension of the singular value matrix (that is, the rank of the singular value matrix), Indicates that a new dimension is added to the model.

[0134] This embodiment provides a large language model training method. When the diagonal elements in the singular value matrix are all greater than a preset cutoff value, the model dimension of the large language model is expanded so that the total dimension of the large language model conforms to the approximate maximum dimension of the linearly independent vector matrix, thereby optimizing the model dimension to a certain extent.

[0135] Step S504: Optimize the dimension of the large language model based on the approximate maximum dimension of the linearly independent vector matrix to obtain the actual output vector dimension of the large language model. Figure 3 Step S304 of the illustrated embodiment will not be described in detail here.

[0136] Step S505: determine the number of neural network nodes based on the actual output vector dimension of the large language model, and perform model training on the large language model with the determined number of neural network nodes to obtain a large language model with optimized dimension. Figure 3 Step S305 of the illustrated embodiment will not be described in detail here.

[0137] The present embodiment provides a large language model training method, which compares the diagonal elements in the singular value matrix with a preset truncation value, and then implements singular value truncation based on the comparison result using the truncation value, determines the approximate maximum dimension of the linearly independent vector matrix by the number of diagonal elements, and uses a method for approximately finding the reasonable dimension of the large language model to achieve rapid determination of the vector dimension of the large language model, laying a foundation for dimensional optimization of the large language model.

[0138] In this embodiment, a large language model training method is provided, which can be used for the above-mentioned electronic device. Figure 6 is a flow chart of a large language model training method according to an embodiment of the present invention. Figure 6 As shown, the process includes the following steps:

[0139] Step S601, obtain multiple text data, input the multiple text data into the large language model after training convergence, and obtain a linearly independent vector matrix. For details, please refer to Figure 5 Step S501 of the illustrated embodiment will not be described in detail here.

[0140] Step S602, perform singular value decomposition on the linearly independent vector matrix to obtain a singular value matrix. Figure 5 Step S502 of the illustrated embodiment will not be described in detail here.

[0141] Step S603, obtaining the number of singular values ​​in the singular value matrix that are greater than a preset cutoff value, and determining the approximate maximum dimension of the linearly independent vector matrix based on the number of singular values ​​greater than the preset cutoff value. Figure 5Step S503 of the illustrated embodiment will not be described in detail here.

[0142] Step S604 , optimizing the dimension of the large language model based on the approximate maximum dimension of the linearly independent vector matrix to obtain the actual output vector dimension of the large language model.

[0143] Specifically, the above step S604 includes:

[0144] Step S6041, obtaining the memory capacity of the computing hardware, and comparing the memory capacity of the computing hardware with the memory of the current large language model; wherein the current large language model satisfies the approximate maximum dimension of the linearly independent vector matrix.

[0145] Step S6042: If the memory of the current large language model is larger than the memory capacity of the computing hardware, the actual output vector dimension of the large language model is determined based on the upper limit of the memory capacity of the computing hardware.

[0146] Step S6043, or, if the memory of the current large language model is smaller than the memory capacity of the computing hardware, the approximate maximum dimension of the linearly independent vector matrix is ​​used as the actual output vector dimension of the large language model.

[0147] Specifically, if the memory of the current large language model is smaller than the memory capacity of the computing hardware, As the actual output vector dimension of the large language model.

[0148] Step S605: Determine the number of neural network nodes based on the actual output vector dimension of the large language model, and perform model training on the large language model with the determined number of neural network nodes to obtain a large language model with optimized dimension. Figure 5 Step S505 of the illustrated embodiment will not be described in detail here.

[0149] This embodiment provides a large language model training method, which selects the upper limit of the computing hardware memory capacity or the approximate maximum dimension of the linearly independent vector matrix as the actual output vector dimension of the large language model by comparing the computing hardware memory capacity with the memory of the current large language model. The current output vector dimension of the large language model and the limitations of the computing hardware device are taken into consideration, so that the actual output vector dimension of the large language model is more in line with actual application requirements, thereby achieving effective optimization of the vector dimension of the large language model.

[0150] In this embodiment, a large language model training method is provided, which can be used for the above-mentioned electronic device. Figure 7 is a flow chart of a large language model training method according to an embodiment of the present invention. Figure 7 As shown, the process includes the following steps:

[0151] Step S701 , randomly selecting a plurality of sample data from a preset text database, and inputting the plurality of sample data into an initialized large language model to obtain an output vector matrix.

[0152] Specifically, the large language model is initialized, the input vector dimension is P, the output vector dimension is N, and each layer of the large language model has basically the same structure and the same number of parameters, so the output vector dimension of each layer is the same; if there is a special large language model with different dimensions of each layer, the output vector dimension can be obtained layer by layer, and then processed according to steps S702 to S707.

[0153] Furthermore, randomly select from a pre-set text database Sample data, considering the limitation of computing resources, the number of sample data Generally should not be less than 1.5 indivual.

[0154] Further, The sample data is input into the large language model, and after being processed by the large language model, the output vector is obtained. ,in ,Will Output vector The output vector matrix is ​​formed; wherein, it is assumed that the output vector matrix is ​​composed of a linearly independent vector group for subsequent calculations.

[0155] Step S702, decompose the output vector matrix, and use the decomposed matrix to construct a loss function.

[0156] Specifically, based on the output vector matrix, a singular value decomposition algorithm is used to obtain a diagonal matrix; a loss function is constructed based on the diagonal matrix and a preset truncation value; the steps of the singular value decomposition algorithm are the same as the above steps S5021-S5024.

[0157] Furthermore, the expression of the loss function is as follows:

[0158] (7)

[0159] in, represents the loss function, The function can be in the form of , it can also be in other forms.

[0160] Furthermore, the loss function can be expressed as:

[0161] (8)

[0162] in, represents the diagonal element.

[0163] Furthermore, the output vector matrix is ​​Schmidt orthogonalized to obtain the matrix eigenvalues; and a loss function is constructed based on the matrix eigenvalues ​​and preset cutoff values.

[0164] Furthermore, a QR algorithm, a Jacobi algorithm, or the like may be used to obtain matrix eigenvalues, and a loss function may be constructed by setting a preset cutoff value.

[0165] Step S703, iteratively train the large language model using the loss function until the large language model reaches a convergence condition, stop iteration, and obtain a large language model after training convergence.

[0166] Specifically, the large language model is iteratively trained under the loss function, the model parameters of the large language model are updated, and the loss function value is updated under the new parameters until the convergence condition is reached, and the iteration is stopped to obtain the large language model after the training converges; wherein the convergence condition includes reaching a specified number of training rounds, or the loss function reaches a specified threshold wait.

[0167] Step S704, obtain multiple text data, input the multiple text data into the large language model after training convergence, and obtain a linearly independent vector matrix. For details, please refer to Figure 6 Step S601 of the illustrated embodiment will not be described in detail here.

[0168] Step S705, perform singular value decomposition on the linearly independent vector matrix to obtain a singular value matrix. Figure 6 Step S602 of the illustrated embodiment will not be described in detail here.

[0169] Step S706, obtaining the number of singular values ​​in the singular value matrix that are greater than a preset cutoff value, and determining the approximate maximum dimension of the linearly independent vector matrix based on the number of singular values ​​greater than the preset cutoff value. Figure 6 Step S603 of the illustrated embodiment will not be described in detail here.

[0170] Step S707: Optimize the dimension of the large language model based on the approximate maximum dimension of the linearly independent vector matrix to obtain the actual output vector dimension of the large language model. Figure 6 Step S604 of the illustrated embodiment will not be described in detail here.

[0171] Step S708: Determine the number of neural network nodes based on the actual output vector dimension of the large language model, and perform model training on the large language model with the determined number of neural network nodes to obtain a large language model with optimized dimension. Figure 6 Step S605 of the illustrated embodiment will not be described in detail here.

[0172] The present embodiment provides a large language model training method, which decomposes the output vector matrix corresponding to the initialized large language model, constructs a loss function using the decomposed matrix, and then iteratively trains the large language model using the loss function. When training the model, the loss function constructed by the decomposed matrix is ​​introduced, and the model is trained under this loss function, which improves the rank of the output vector matrix, realizes the initial training of the large language model, and lays a foundation for the subsequent optimization of the large language model dimension.

[0173] The following is a specific example to illustrate the specific steps of a large language model training method.

[0174] Embodiment 1:

[0175] like Figure 8 As shown, the steps of the large language model training method include:

[0176] S1: Initialize the large language model, the input vector dimension is , the output vector dimension is ;

[0177] S2: Randomly filter from a text database Sample data, the number of sample data Generally should not be less than 1.5 indivual;

[0178] S3: The sample data is input into the large language model, and after being processed by the large language model, the output vector is obtained. ,in ;

[0179] S4: Output vector Form an output vector matrix and perform singular value decomposition on the output vector matrix to obtain a singular value matrix ;

[0180] S5: Determine the cutoff value , based on the singular value matrix Create a loss function;

[0181] S6: Perform training under this loss function, update the large language model parameters, and update the loss function value under the new parameters;

[0182] S7: Repeat the above until the convergence condition is reached;

[0183] S8: After the large language model converges, determine the minimum diagonal element that satisfies the cutoff value Corresponding , and then determine the approximate maximum dimension of the linearly independent vector matrix that can be obtained by the large language model .

[0184] When all If both are greater than the cutoff value c, it may indicate that the large language model dimension is too small and the model dimension should be increased accordingly. The method can be: Take the remainder r of 2, and then set the new dimension of the model to , then the approximate maximum dimension of the linearly independent vector matrix is The calculation formula is as follows:

[0185]

[0186] S9: Reference , according to the constraints of computing hardware equipment, the actual output vector dimension of the large language model is determined. The constraints of computing hardware equipment specifically refer to: When the memory requirement of a large language model is significantly greater than the hardware memory capacity, the actual output vector dimension is determined by the hardware upper limit; when When the memory requirement of a large model is less than the hardware memory capacity, as the actual output vector dimension.

[0187] S10: After determining the actual output vector dimension of the large language model, conventional model training is performed to obtain a model (i.e., the large language model after dimension optimization).

[0188] In the above-mentioned embodiment 1, a loss function based on the rank of the matrix composed of the model output vector is introduced when training the model, and the model is trained under this loss function to improve the rank of the model output vector matrix; after the model converges, a singular value matrix is ​​obtained by singular value decomposition, and the degree of dimensional redundancy of the model is determined by recording the number of singular values ​​greater than a preset truncation value and performing singular value truncation, thereby optimizing the model dimension, and then entering the normal training mode to train the model, thereby solving the technical problem that the dimension of a large language model vector is difficult to determine and difficult to systematically optimize, thereby improving the training and reasoning efficiency of the model.

[0189] In this embodiment, a large language model is also provided, which can be obtained according to the above large language model training method.

[0190] Specifically, large language models can be applied to electronic devices in the fields of natural language processing, content creation, dialogue systems, knowledge base construction, data analysis, personalized recommendations, speech recognition and synthesis, social media analysis, etc. in the field of artificial intelligence.

[0191] In this embodiment, a large language model training device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0192] This embodiment provides a large language model training device, such as Fig. 9 As shown, including:

[0193] An acquisition module 901 is used to acquire multiple text data, input the multiple text data into a large language model after training convergence, and obtain a linearly independent vector matrix;

[0194] A singular value decomposition module 902 is used to perform singular value decomposition on a linearly independent vector matrix to obtain a singular value matrix;

[0195] A determination module 903 is used to obtain the number of singular values ​​greater than a preset cutoff value in the singular value matrix, and determine the approximate maximum dimension of the linearly independent vector matrix based on the number of singular values ​​greater than the preset cutoff value;

[0196] A dimension optimization module 904 is used to optimize the dimension of the large language model based on the approximate maximum dimension of the linearly independent vector matrix to obtain the actual output vector dimension of the large language model;

[0197] The model training module 905 is used to determine the number of neural network nodes based on the actual output vector dimension of the large language model, and perform model training on the large language model with the determined number of neural network nodes to obtain a large language model with optimized dimension.

[0198] In some optional implementations, the determining module 903 includes:

[0199] A first comparison unit, used for comparing the diagonal elements in the singular value matrix with a preset cutoff value;

[0200] An acquisition unit, used for acquiring the number of diagonal elements greater than a preset cutoff value, and taking the number of diagonal elements greater than the preset cutoff value as the approximate maximum dimension of the linearly independent vector matrix;

[0201] The model dimension expansion unit is used to expand the model dimension of the large language model if the diagonal elements in the singular value matrix are all greater than the preset truncation value, and determine the approximate maximum dimension of the linearly independent vector matrix.

[0202] In some optional embodiments, the model dimension augmentation unit includes:

[0203] A calculation subunit, used for calculating the additional dimension of the model based on the diagonal elements and the preset truncation value if all diagonal elements in the singular value matrix are greater than the preset truncation value;

[0204] A determination subunit is provided for determining an approximate maximum dimension of a linearly independent vector matrix based on a total dimension of the singular value matrix and a newly added dimension of the model.

[0205] In some optional implementations, the calculation formula for determining the approximate maximum dimension of the linearly independent vector matrix in the subunit is as follows:

[0206]

[0207]

[0208] Among them, r represents the remainder, represents the diagonal elements in the singular value matrix, Indicates the preset cutoff value, represents the approximate maximum dimension of a matrix of linearly independent vectors, represents the total dimension of the singular value matrix, Indicates that a new dimension is added to the model.

[0209] In some optional implementations, the dimension optimization module 904 includes:

[0210] A second comparison unit is used to obtain the memory capacity of the computing hardware and compare the memory capacity of the computing hardware with the memory of the current large language model; wherein the current large language model satisfies the approximate maximum dimension of the linearly independent vector matrix;

[0211] A first determining unit, configured to determine an actual output vector dimension of the large language model based on an upper limit of the memory capacity of the computing hardware if the memory of the current large language model is larger than the memory capacity of the computing hardware;

[0212] The second determining unit is used to use the approximate maximum dimension of the linearly independent vector matrix as the actual output vector dimension of the large language model if the memory of the current large language model is smaller than the memory capacity of the computing hardware.

[0213] In some optional implementations, the singular value decomposition module 902 includes:

[0214] A transposition unit, used for performing a transposition process on a linearly independent vector matrix to obtain a transposed matrix;

[0215] A decomposition unit, used for performing eigendecomposition on the product of the linearly independent vector matrix and the transposed matrix to obtain eigenvalues ​​and eigenvectors;

[0216] A third determining unit, configured to determine a first orthogonal matrix and a second orthogonal matrix based on the eigenvalues ​​and the eigenvectors;

[0217] The first construction unit is used to construct a singular value matrix based on the linearly independent vector matrix, the first orthogonal matrix and the second orthogonal matrix.

[0218] In some optional embodiments, the method further comprises:

[0219] A screening module is used to randomly screen multiple sample data from a preset text database, and input the multiple sample data into the initialized large language model to obtain an output vector matrix;

[0220] The decomposition module is used to decompose the output vector matrix and construct the loss function using the decomposed matrix;

[0221] The iterative training module is used to iteratively train the large language model using the loss function until the large language model reaches the convergence condition, stop the iteration, and obtain the large language model after the training converges.

[0222] In some optional embodiments, the decomposition module includes:

[0223] A calculation unit, used for obtaining a diagonal matrix by using a singular value decomposition algorithm based on the output vector matrix;

[0224] The second construction unit is used to construct a loss function based on the diagonal matrix and the preset cutoff value.

[0225] In some optional implementations, the expression of the loss function in the second building block is as follows:

[0226]

[0227] in, Represents the loss function.

[0228] In some optional embodiments, the decomposition module further includes:

[0229] A processing unit, used for performing Schmidt orthogonalization processing on the output vector matrix to obtain matrix eigenvalues;

[0230] The third construction unit is used to construct a loss function based on the matrix eigenvalues ​​and the preset cutoff value.

[0231] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0232] A large language model training device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0233] The embodiment of the present invention also provides a computer device having the above Fig. 9 A large language model training device is shown.

[0234] See also Fig.10 , Fig.10 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig.10 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.10 A processor 10 is taken as an example.

[0235] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0236] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0237] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0238] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0239] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Fig.10 The example of connecting through bus is taken in the following.

[0240] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device may be a touch screen.

[0241] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0242] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0243] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A large language model training method, It is characterized in that The method comprises: Acquire multiple text data, input the multiple text data into the large language model after training convergence, and obtain a linearly independent vector matrix; Performing singular value decomposition on the linearly independent vector matrix to obtain a singular value matrix; Obtaining the number of singular values ​​in the singular value matrix that are greater than a preset cutoff value, and determining an approximate maximum dimension of a linearly independent vector matrix based on the number of singular values ​​that are greater than the preset cutoff value; Optimizing the dimension of the large language model based on the approximate maximum dimension of the linearly independent vector matrix to obtain the actual output vector dimension of the large language model; Determining the number of neural network nodes based on the actual output vector dimension of the large language model, and performing model training on the large language model with the determined number of neural network nodes to obtain a large language model with optimized dimension; The obtaining of the number of singular values ​​in the singular value matrix that are greater than a preset cutoff value, and determining the approximate maximum dimension of the linearly independent vector matrix based on the number of singular values ​​that are greater than the preset cutoff value, comprises: Comparing the diagonal elements in the singular value matrix with the preset cutoff value; Obtaining the number of diagonal elements greater than the preset truncation value, and using the number of diagonal elements greater than the preset truncation value as the approximate maximum dimension of the linearly independent vector matrix; Alternatively, if all diagonal elements in the singular value matrix are greater than the preset cutoff value, the model dimension of the large language model is expanded to determine the approximate maximum dimension of the linearly independent vector matrix.

2. The method according to claim 1, It is characterized in that If all diagonal elements in the singular value matrix are greater than the preset cutoff value, the large language model is expanded in model dimension to determine the approximate maximum dimension of the linearly independent vector matrix, including: If all diagonal elements in the singular value matrix are greater than the preset cutoff value, then additional dimensions of the model are calculated based on the diagonal elements and the preset cutoff value; The approximate maximum dimension of the linearly independent vector matrix is ​​determined based on the total dimension of the singular value matrix and the newly added dimension of the model.

3. The method according to claim 2, It is characterized in that The approximate maximum dimension of the linearly independent vector matrix is ​​determined based on the total dimension of the singular value matrix and the newly added dimension of the model. The calculation formula of the approximate maximum dimension of the linearly independent vector matrix is ​​as follows: in, represents the remainder, represents the diagonal elements in the singular value matrix, Indicates the preset cutoff value, represents the approximate maximum dimension of a matrix of linearly independent vectors, represents the total dimension of the singular value matrix, Indicates that a new dimension is added to the model.

4. The method according to claim 1, It is characterized in that The step of optimizing the dimension of the large language model based on the approximate maximum dimension of the linearly independent vector matrix to obtain the actual output vector dimension of the large language model includes: Acquire computing hardware memory capacity, and compare the computing hardware memory capacity with the memory of a current large language model; wherein the current large language model satisfies the approximate maximum dimension of the linearly independent vector matrix; If the memory of the current large language model is larger than the memory capacity of the computing hardware, determining the actual output vector dimension of the large language model based on the upper limit of the memory capacity of the computing hardware; Alternatively, if the memory of the current large language model is smaller than the memory capacity of the computing hardware, the approximate maximum dimension of the linearly independent vector matrix is ​​used as the actual output vector dimension of the large language model.

5. The method according to claim 1, It is characterized in that The step of performing singular value decomposition on the linearly independent vector matrix to obtain a singular value matrix includes: Transpose the linearly independent vector matrix to obtain a transposed matrix; Performing eigendecomposition on the product of the linearly independent vector matrix and the transposed matrix to obtain eigenvalues ​​and eigenvectors; Determine a first orthogonal matrix and a second orthogonal matrix based on the eigenvalues ​​and the eigenvectors; The singular value matrix is ​​constructed based on the linearly independent vector matrix, the first orthogonal matrix and the second orthogonal matrix.

6. The method according to claim 1, It is characterized in that Before acquiring a plurality of text data and inputting the plurality of text data into a large language model after training convergence to obtain a linearly independent vector matrix, the method further includes: Randomly select a plurality of sample data from a preset text database, and input the plurality of sample data into the initialized large language model to obtain an output vector matrix; Decomposing the output vector matrix and constructing a loss function using the decomposed matrix; The large language model is iteratively trained using the loss function until the large language model reaches a convergence condition, and iteration is stopped to obtain the large language model after the training converges.

7. The method according to claim 6, It is characterized in that Decomposing the output vector matrix and constructing a loss function using the decomposed matrix includes: Based on the output vector matrix, a diagonal matrix is ​​obtained using a singular value decomposition algorithm; The loss function is constructed based on the diagonal matrix and the preset cutoff value.

8. The method according to claim 7, It is characterized in that The loss function is constructed based on the diagonal matrix and the preset cutoff value, and the expression of the loss function is as follows: in, Represents the loss function.

9. The method according to claim 6, It is characterized in that Decomposing the output vector matrix and constructing a loss function using the decomposed matrix includes: Performing Schmidt orthogonalization processing on the output vector matrix to obtain matrix eigenvalues; The loss function is constructed based on the matrix eigenvalues ​​and the preset cutoff value.

10. A large language model, It is characterized in that The large language model is obtained according to the large language model training method according to any one of claims 1 to 9.

11. A large language model training device, It is characterized in that The device comprises: An acquisition module, used for acquiring a plurality of text data, inputting the plurality of text data into a large language model after training convergence, and obtaining a linearly independent vector matrix; A singular value decomposition module, used for performing singular value decomposition on the linearly independent vector matrix to obtain a singular value matrix; A determination module, configured to obtain the number of singular values ​​in the singular value matrix that are greater than a preset cutoff value, and determine an approximate maximum dimension of a linearly independent vector matrix based on the number of singular values ​​greater than the preset cutoff value; A dimension optimization module, used for performing dimension optimization on the large language model based on the approximate maximum dimension of the linearly independent vector matrix to obtain an actual output vector dimension of the large language model; A model training module, used to determine the number of neural network nodes based on the actual output vector dimension of the large language model, and perform model training on the large language model with the determined number of neural network nodes to obtain a large language model with optimized dimension; The modules are determined to include: A first comparison unit, used for comparing the diagonal elements in the singular value matrix with a preset cutoff value; An acquisition unit, used for acquiring the number of diagonal elements greater than a preset cutoff value, and taking the number of diagonal elements greater than the preset cutoff value as the approximate maximum dimension of the linearly independent vector matrix; The model dimension expansion unit is used to expand the model dimension of the large language model if the diagonal elements in the singular value matrix are all greater than the preset truncation value, and determine the approximate maximum dimension of the linearly independent vector matrix.

12. A computer device, It is characterized in that include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the large language model training method according to any one of claims 1 to 9 by executing the computer instructions.

13. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the large language model training method according to any one of claims 1 to 9.

14. A computer program product, It is characterized in that The method comprises computer instructions for causing a computer to execute the large language model training method according to any one of claims 1 to 9.

Images