Optimization method, device and electronic equipment for large model

Abstract

The disclosure provides an optimization method and device of a large model and electronic equipment, relates to the technical field of artificial intelligence, in particular to the technical field of large models and deep learning, and comprises the following steps: extracting model parameters from a first large model of a multi-head self-attention (MHA) structure to obtain a model parameter set, performing singular value decomposition (SVD) on the first type of model parameters to obtain an approximate matrix pair of the first type of model parameters, and generating a second large model of a multi-head implicit self-attention (MLA) structure based on the approximate matrix pair of the first type of model parameters and the second type of model parameters. The disclosure improves the efficiency of generating a large model and the performance of the large model by performing singular value decomposition on the first type of model parameters to obtain an approximate matrix pair of the first type of model parameters and generating a second large model of an MLA structure based on the approximate matrix pair of the first type of model parameters and the second type of model parameters, thereby reducing the cost of generating a large model.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to the fields of large models and deep learning, and particularly to a method, apparatus, and electronic device for optimizing large models. Background Technology

[0002] Large models have been widely used in deep learning. However, training large models often requires starting from scratch, a process that is not only tedious and complex but also demands a large amount of text data and computational resources. Due to the high resource requirements and costs, training costs are high. Existing large models are usually multi-head self-attention structures. However, multi-head implicit self-attention structures can effectively reduce the caching requirements for key-value pairs by performing low-rank processing on key-value pairs, thereby optimizing the storage efficiency and performance of large models. Therefore, how to generate multi-head implicit self-attention structures based on large models with multi-head self-attention structures to optimize large models, improve the efficiency and performance of large model generation, and reduce the cost of large model generation has become an urgent problem to be solved. Summary of the Invention

[0003] This disclosure presents a method, apparatus, electronic device, storage medium, and computer program product for optimizing large models.

[0004] According to a first aspect of this disclosure, a method for optimizing a large model is proposed, comprising: extracting model parameters from a first large model of a multi-head self-attention MHA structure to obtain a set of model parameters, wherein the set of model parameters includes a first type of model parameters and a second type of model parameters; performing singular value decomposition (SVD) on the first type of model parameters to obtain approximate matrix pairs of the first type of model parameters; and generating a second large model of a multi-head implicit self-attention MLA structure based on the approximate matrix pairs of the first type of model parameters and the second type of model parameters.

[0005] According to a second aspect of this disclosure, an optimization apparatus for a large model is proposed, comprising: an extraction module for extracting model parameters from a first large model of a multi-head self-attention (MHA) structure to obtain a set of model parameters, wherein the set of model parameters includes a first type of model parameters and a second type of model parameters; an acquisition module for performing singular value decomposition (SVD) on the first type of model parameters to obtain approximate matrix pairs of the first type of model parameters; and a generation module for generating a second large model of a multi-head implicit self-attention (MLA) structure based on the approximate matrix pairs of the first type of model parameters and the second type of model parameters.

[0006] According to a third aspect of this disclosure, an electronic device is proposed, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the optimization method of the large model proposed in the first aspect above.

[0007] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is proposed, wherein the computer instructions are used to cause the computer to execute the optimization method of the large model proposed in the first aspect above.

[0008] According to a fifth aspect of this disclosure, a computer program product is proposed, comprising a computer program that, when executed by a processor, implements the optimization method for the large model proposed in the first aspect above.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0011] Figure 1 This is a flowchart illustrating a method for optimizing a large model according to an embodiment of this disclosure;

[0012] Figure 2 This is a flowchart illustrating a method for optimizing a large model according to an embodiment of this disclosure;

[0013] Figure 3 This is a flowchart illustrating a method for optimizing a large model according to an embodiment of this disclosure;

[0014] Figure 4 This is a schematic diagram of the structure of an optimization device for a large model according to an embodiment of the present disclosure;

[0015] Figure 5 This is a schematic block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0016] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0017] Artificial Intelligence (AI) is a new technical science that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence.

[0018] Large models refer to machine learning models with a large number of parameters and high complexity. They require a lot of computing resources and storage space for training and storage, and often require distributed computing and special hardware acceleration technologies. Large models have stronger generalization and expressive capabilities.

[0019] Deep learning (DL) is a new research direction in the field of machine learning (ML). It was introduced into machine learning to bring it closer to its original goal—artificial intelligence. Deep learning learns the inherent patterns and hierarchical representations of sample data. The information gained during this learning process greatly helps in interpreting data such as text, images, and sound. Its ultimate goal is to enable machines to have analytical and learning capabilities like humans, and to recognize data such as text, images, and sound.

[0020] Figure 1 This is a flowchart illustrating a large-scale model optimization method according to an embodiment of this disclosure. Figure 1 As shown, the method includes:

[0021] S101, extract model parameters from the first large model of the multi-head self-attention MHA structure to obtain a model parameter set, which includes first-class model parameters and second-class model parameters.

[0022] It should be noted that the execution entity of the large model optimization method in this embodiment can be a hardware device with the capability to optimize large data models and / or the necessary software to drive the hardware device. Optionally, the execution entity may include a workstation, server, computer, user terminal, and other intelligent devices. The user terminal includes, but is not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, and in-vehicle terminals.

[0023] Among them, the largest model in the Multi-Head Attention (MHA) structure is a large model that has been trained (converged).

[0024] It should be noted that this disclosure can train a first-class model based on sample data. This first-class model can process different types of data according to different application scenarios to obtain processing results.

[0025] For example, the first major model can be used to process different types of data, such as text data, image data, and audio / video data, to obtain processing results.

[0026] It should be noted that this disclosure does not limit the specific method for extracting model parameters from the first large model of the multi-head self-attention MHA structure, and the method can be selected according to the actual situation.

[0027] Optionally, model parameters can be extracted from the first large model of the multi-head self-attention (MHA) structure based on the PyTorch framework to obtain a model parameter set, wherein the model parameter set includes first-class model parameters and second-class model parameters.

[0028] For example, the first type of model parameters may include the weight matrix and bias vector of the query parameter Q, key parameter K, and value parameter V in the MHA structure.

[0029] For example, the second type of model parameters may include the number of heads in the MHA structure, the embedding dimension, the model parameters associated with the feedforward network layer in the first large model, and the model parameters associated with the embedding layer in the first large model.

[0030] In this embodiment of the disclosure, after obtaining the first type of model parameters and the second type of model parameters, the first type of model parameters and the second type of model parameters can be cached for subsequent use.

[0031] S102, Perform singular value decomposition (SVD) on the parameters of the first type of model to obtain approximate matrix pairs of the parameters of the first type of model.

[0032] Singular Value Decomposition (SVD) is an effective matrix factorization technique that can decompose any matrix into the product of three matrices: the left singular vector matrix, the singular value matrix, and the right singular vector matrix.

[0033] In this embodiment of the disclosure, the weight matrices of the query parameter Q, key parameter K, and value parameter V in the MHA structure can be obtained from the cache, and singular value decomposition (SVD) can be performed on the weight matrix Wq of the query parameter Q, the weight matrix Wk of the key parameter K, and the weight matrix Wv of the value parameter V in sequence to obtain approximate matrix pairs of the first type of model parameters.

[0034] Optionally, the weight matrix Wq of parameter Q, the weight matrix Wk of key parameter K, and the weight matrix Wv of value parameter V can be queried for SVD processing to obtain the decomposition results of the first type of model parameters. The decomposition results include the left singular vector matrix, the singular value matrix, and the right singular vector matrix. Based on the left singular vector matrix, the singular value matrix, and the right singular vector matrix, the approximate matrix pairs of the first type of model parameters are obtained.

[0035] S103 generates the second largest model of the multi-head implicit self-attention MLA structure based on the approximate matrix pairs of the first type of model parameters and the second type of model parameters.

[0036] In this embodiment of the disclosure, a pre-trained initial second-largest model can be obtained. Based on the approximate matrix pairs of the parameters of the first type of model, the parameters of the MLA structure in the initial second-largest model are configured. Based on the parameters of the second type of model, the parameters of the remaining network layers in the initial second-largest model other than the MLA structure are configured to obtain a second-largest model with a multi-head latent attention (MLA) structure.

[0037] It should be noted that the multi-head implicit self-attention mechanism is an attention mechanism for natural language processing and sequence processing tasks. The core idea is to perform low-rank processing on key-value pairs, thereby reducing the caching of corresponding key-value pairs. During inference, matrix absorption avoids recomputation of the implicit cache, thus reducing the amount of computation while minimizing the amount of key-value pair caching. By performing low-rank processing, key-value pairs can be mapped to a lower-dimensional space, which significantly reduces the amount of data that needs to be cached, lowers the model's storage requirements, ensures model performance while avoiding redundant calculations, effectively reduces computational load, and achieves faster inference speed while maintaining high accuracy.

[0038] It should be noted that after generating the second largest model, training samples can be obtained, and the second largest model can be fine-tuned based on the training samples to obtain the target large model after training.

[0039] Optionally, the training samples can be input into the second large model, which will then train the training samples to obtain the output of the second large model. Based on the output of the second large model, the loss value of the second large model can be determined. Based on the loss value, the model parameters of the second large model can be adjusted, and the next training sample can be used to continue training the large model with adjusted model parameters until the model training termination condition is met, thus obtaining the target large model that has been trained.

[0040] The large model optimization method proposed in this disclosure extracts model parameters from the first large model of the multi-head self-attention (MHA) structure to obtain a model parameter set, which includes first-class model parameters and second-class model parameters. Singular value decomposition (SVD) is performed on the first-class model parameters to obtain approximate matrix pairs of the first-class model parameters. These approximate matrix pairs include a dimensionality-reduced matrix and an increased-dimensional matrix of the first-class model parameters. Based on the approximate matrix pairs of the first-class model parameters and the second-class model parameters, a second large model of the multi-head implicit self-attention (MLA) structure is generated. Therefore, this disclosure improves the efficiency and performance of large model generation and reduces the cost of large model generation by extracting model parameters from the first large model of the MHA structure, performing singular value decomposition on the first-class model parameters to obtain approximate matrix pairs of the first-class model parameters, and generating a second large model of the MLA structure based on the approximate matrix pairs of the first-class model parameters and the second-class model parameters.

[0041] Figure 2 This is a flowchart illustrating the optimization method for a large model according to the second embodiment of this disclosure.

[0042] like Figure 2 As shown, in Figure 2 Based on the illustrated embodiments, the optimization method for the large model in this disclosure may specifically include the following steps:

[0043] S201, extract model parameters from the first large model of the multi-head self-attention MHA structure to obtain a model parameter set, which includes first-class model parameters and second-class model parameters.

[0044] For details regarding step S201, please refer to the above embodiments, which will not be repeated here.

[0045] The step S102 in the above embodiment, "perform singular value decomposition (SVD) on the first type of model parameters to obtain approximate matrix pairs of the first type of model parameters", may specifically include the following steps S202 and S204.

[0046] S202, Perform SVD processing on the first type of model parameters to obtain the decomposition results of the first type of model parameters, wherein the decomposition results include the left singular vector matrix, the singular value matrix and the right singular vector matrix.

[0047] In this embodiment of the disclosure, the first type of model parameters includes the weight matrix Wq of the query parameter Q, the weight matrix Wk of the key parameter K, and the weight matrix Wv of the value parameter V.

[0048] It should be noted that the weight matrix Wq of the query parameter Q can be processed by SVD, which decomposes Wq into the product of three matrices, i.e.: Wq=U∑V TWhere Wq is the weight matrix of the query parameter Q, U is the left singular vector matrix, and V is the weight matrix of the query parameter Q. T Let Wk be the transpose of the right singular vector matrix and ∑ be the singular value matrix. The process of performing SVD on the weight matrix Wk of the key parameter K and the weight matrix Wv of the value parameter V is the same as the process of performing SVD on the weight matrix Wq of the query parameter Q, and will not be repeated here.

[0049] S203. Based on the left singular vector matrix, singular value matrix, and right singular vector matrix, an approximate matrix pair of the first type of model parameters is obtained, wherein the approximate matrix pair includes the dimension reduction matrix and the dimension increase matrix of the first type of model parameters.

[0050] like Figure 3 As shown, step S203 in the above embodiment, "obtaining an approximate matrix pair of the first type of model parameters based on the left singular vector matrix, the singular value matrix, and the right singular vector matrix", may specifically include the following steps S301 and S204.

[0051] S301, determine the first K largest singular values ​​from the singular value matrix, where K is a natural number greater than or equal to 1.

[0052] In this embodiment of the disclosure, after obtaining the singular value matrix, the singular values ​​in the singular value matrix can be sorted from largest to smallest to determine the top K largest singular values, where K is a natural number greater than or equal to 1.

[0053] It should be noted that this disclosure does not impose any restrictions on the value of K, and it can be set according to the actual situation.

[0054] S302, based on the first K largest singular values, the left singular vector matrix, and the right singular vector matrix, obtain the approximate matrix pairs of the first type of model parameters.

[0055] In this embodiment of the disclosure, after obtaining the first K largest singular values, a first diagonal matrix can be constructed based on the first K largest singular values, and the first diagonal matrix can be transformed to obtain a second diagonal matrix. Based on the second diagonal matrix, the left singular vector matrix, and the right singular vector matrix, an approximate matrix pair of the first type of model parameters is obtained.

[0056] Optionally, the square root of each diagonal element in the first diagonal matrix can be obtained, and the second diagonal matrix can be obtained based on the square root of each diagonal element.

[0057] In this embodiment of the disclosure, matrix multiplication can be performed on the left singular vector matrix and the second diagonal matrix to obtain a first approximate matrix of the first type of model parameters, wherein the first approximate matrix is ​​a dimension reduction matrix. Then, matrix multiplication can be performed on the second diagonal matrix and the right singular vector matrix to obtain a second approximate matrix of the first type of model parameters, wherein the second approximate matrix is ​​a dimension increase matrix.

[0058] The step S103 in the above embodiment, "generating the second major model of the multi-head implicit self-attention MLA structure based on the approximate matrix pair of the first type of model parameters and the second type of model parameters", may specifically include the following steps S204 and S205.

[0059] S204, Obtain the initial second-largest pre-trained model.

[0060] Among them, the initial second largest model in the pre-training includes the MLA structure.

[0061] S205, based on approximate matrix pairs, configures the parameters of the MLA structure in the initial second-largest model, and based on the parameters of the second type of model, configures the parameters of the remaining network layers in the initial second-largest model other than the MLA structure, so as to obtain the second-largest model.

[0062] The MLA structure includes a first network layer for dimensionality reduction and a second network layer for dimensionality increase. The number of input dimensions of the first network layer is the same as the number of output dimensions of the second network layer, and the number of output dimensions of the first network layer is the same as the number of input dimensions of the second network layer.

[0063] In this embodiment of the disclosure, the bias vector of the first type of model parameters can be obtained from the cache, and the approximate matrix pair can be biased based on the bias vector to obtain the target approximate matrix pair. Based on the target approximate matrix pair, the parameters of the MLA structure in the second large model can be configured.

[0064] In this embodiment of the disclosure, for the remaining network layer i in the second large model, the target network layer j of the first large model corresponding to the remaining network layer is determined, where i and j are natural numbers greater than or equal to 1. The second type of model parameters associated with the target network layer j are determined, and the parameters of the remaining network layer i are configured based on the second type of model parameters associated with the target network layer j.

[0065] In this embodiment of the disclosure, the first major model further includes a feedforward neural network (FNN) layer and an embedding layer MHL structure, and the second major model reuses the structure of the FFN layer and the embedding layer in the first major model.

[0066] For example, if the remaining network layers in the second largest model are FNN layers, the target network layer of the first largest model corresponding to the remaining FNN layers is determined to be the FFN layer in the first largest model. The second type of model parameters of the FFN layer in the first largest model are determined from the cache, and the parameters of the remaining network layers are configured as FNN layers based on the second type of model parameters.

[0067] For example, if the remaining network layer in the second largest model is the embedding layer, the target network layer of the first largest model corresponding to the embedding layer of the remaining network layer is determined to be the embedding layer in the first largest model. The second type of model parameters of the embedding layer in the first largest model are determined from the cache. Based on the second type of model parameters, the parameters of the remaining network layer as the embedding layer are configured to realize the migration of the second type of model parameters associated with the target network layer in the first largest model.

[0068] It should be noted that after generating the second large model, training samples can be obtained, and the second large model can be fine-tuned based on the training samples to obtain the target large model after training. The target large model after training can process different types of data according to different application scenarios to obtain the processing results output by the target large model.

[0069] For example, the target large model can be used to process different types of data such as text data, image data, and audio and video data to obtain the processing results output by the target large model.

[0070] In summary, the large model optimization method proposed in this disclosure extracts model parameters from the first large model of the multi-head self-attention (MHA) structure to obtain a model parameter set, which includes first-class and second-class model parameters. The first-class model parameters are then subjected to SVD processing to obtain decomposition results, including a left singular vector matrix, a singular value matrix, and a right singular vector matrix. Based on these matrices, approximate matrix pairs of the first-class model parameters are obtained, leading to the acquisition of a pre-trained initial second large model. Based on these approximate matrix pairs, the parameters of the MLA structure in the initial second large model are configured. Finally, based on the second-class model parameters, the parameters of the remaining network layers in the initial second large model, excluding the MLA structure, are configured to obtain the desired model. In the second major model, this disclosure uses Singular Value Decomposition (SVD) to perform on the first type of model parameters extracted from the first major model of the MHA structure to obtain approximate matrix pairs of the first type of model parameters. Based on the approximate matrix pairs of the first type of model parameters and the second type of model parameters, a second major model of the MLA structure is generated. The second major model of the MLA structure can reduce the dependence on large-scale data and computing resources, reduce the cost in the training process of large models, help alleviate the pressure on enterprises in terms of resource investment, shorten the development cycle of large models, and improve the deployment speed of large models. In the inference stage of the second major model of the MLA structure, the second major model of the MLA structure cleverly avoids the recomputation of implicit caching, which can effectively reduce the amount of computation and improve the inference speed of large models while ensuring the performance of large models.

[0071] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0072] According to embodiments of this disclosure, this disclosure also provides a large model optimization apparatus for implementing the above-described large model optimization method.

[0073] Figure 4 This is a block diagram of an optimization apparatus for a large model according to an embodiment of the present disclosure.

[0074] like Figure 4 As shown, the optimization device 400 for the large model includes: an extraction module 401, an acquisition module 402, and a generation module 403.

[0075] Extraction module 401 is used to extract model parameters from the first large model of the multi-head self-attention MHA structure to obtain a set of model parameters, wherein the set of model parameters includes a first type of model parameters and a second type of model parameters.

[0076] The acquisition module 402 is used to perform singular value decomposition (SVD) on the parameters of the first type of model to obtain approximate matrix pairs of the parameters of the first type of model.

[0077] The generation module 403 is used to generate a second large model of the multi-head implicit self-attention MLA structure based on the approximate matrix pair of the first type of model parameters and the second type of model parameters.

[0078] In one embodiment of this disclosure, the acquisition module 402 is configured to: perform SVD processing on the first type of model parameters to obtain the decomposition result of the first type of model parameters, wherein the decomposition result includes a left singular vector matrix, a singular value matrix, and a right singular vector matrix; and obtain an approximate matrix pair of the first type of model parameters based on the left singular vector matrix, the singular value matrix, and the right singular vector matrix.

[0079] In one embodiment of this disclosure, the acquisition module 402 is configured to: determine the top K largest singular values ​​from the singular value matrix, where K is a natural number greater than or equal to 1; and obtain an approximate matrix pair of the first type of model parameters based on the top K largest singular values, the left singular vector matrix, and the right singular vector matrix.

[0080] In one embodiment of this disclosure, the acquisition module 402 is configured to: obtain a second diagonal matrix by transforming the first diagonal matrix based on the first K largest singular values; and obtain an approximate matrix pair of the first type of model parameters based on the second diagonal matrix, the left singular vector matrix, and the right singular vector matrix, wherein the approximate matrix pair includes a dimension-reduced matrix and a dimension-increasing matrix of the first type of model parameters.

[0081] In one embodiment of this disclosure, the acquisition module 402 is configured to: perform matrix multiplication on the left singular vector matrix and the second diagonal matrix to obtain a first approximate matrix of the first type of model parameters, wherein the first approximate matrix is ​​the dimension reduction matrix; and perform matrix multiplication on the second diagonal matrix and the right singular vector matrix to obtain a second approximate matrix of the first type of model parameters, wherein the second approximate matrix is ​​the dimension increase matrix.

[0082] In one embodiment of this disclosure, the acquisition module 402 is configured to: acquire the square root of each diagonal element in the first diagonal matrix, and obtain the second diagonal matrix based on the square root of each diagonal element.

[0083] In one embodiment of this disclosure, the first type of model parameters includes the weight matrix of the query parameter Q, key parameter K, and value parameter V in the MHA structure.

[0084] In one embodiment of this disclosure, the MLA structure includes a first network layer for dimensionality reduction and a second network layer for dimensionality increase, wherein the number of input dimensions of the first network layer is the same as the number of output dimensions of the second network layer; and the number of output dimensions of the first network layer is the same as the number of input dimensions of the second network layer.

[0085] In one embodiment of this disclosure, the generation module 403 is configured to: obtain a pre-trained initial second-largest model; configure the parameters of the MLA structure in the initial second-largest model based on the approximation matrix pair; and configure the parameters of the remaining network layers in the initial second-largest model other than the MLA structure based on the second type of model parameters, so as to obtain the second-largest model.

[0086] In one embodiment of this disclosure, the generation module 403 is configured to: obtain the bias vector of the first type of parameter model, and perform bias processing on the approximation matrix pair based on the bias vector to obtain a target approximation matrix pair; and configure the parameters of the MLA structure in the second large model based on the target approximation matrix pair.

[0087] In one embodiment of this disclosure, the generation module 403 is configured to: determine, for the remaining network layer i in the second large model, the target network layer j corresponding to the first large model, wherein i and j are natural numbers greater than or equal to 1; determine the second type of model parameters associated with the target network layer j, and configure the parameters of the remaining network layer i based on the second type of model parameters associated with the target network layer j.

[0088] In one embodiment of this disclosure, the first large model further includes a feedforward network (FNN) layer and an embedding layer, and the second large model reuses the structure of the FNN layer and the embedding layer in the first large model.

[0089] The optimization device for large models proposed in this disclosure extracts model parameters from a first large model of a multi-head self-attention (MHA) structure to obtain a set of model parameters, which includes first-class model parameters and second-class model parameters. Singular value decomposition (SVD) is performed on the first-class model parameters to obtain approximate matrix pairs of the first-class model parameters. Based on the approximate matrix pairs of the first-class model parameters and the second-class model parameters, a second large model of a multi-head implicit self-attention (MLA) structure is generated. Therefore, this disclosure improves the efficiency and performance of generating large models and reduces the cost of generating large models by extracting model parameters from the first large model of the MHA structure, performing singular value decomposition on the first-class model parameters to obtain approximate matrix pairs of the first-class model parameters, and generating a second large model of the MLA structure based on the approximate matrix pairs of the first-class model parameters and the second-class model parameters.

[0090] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0091] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0092] like Figure 5As shown, device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 502 or a computer program loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0093] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0094] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as large model optimization methods. For example, in some embodiments, the large model optimization method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the large model optimization method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform large model optimization methods by any other suitable means (e.g., by means of firmware).

[0095] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0096] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0097] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0098] To initiate interaction with a user account, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user account; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user account can submit input to the computer. Other types of devices can also be used to initiate interaction with the user account; for example, feedback submitted to the user account can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user account can be received in any form (including voice input, speech input, or tactile input).

[0099] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user account computer with a graphical user interface or web browser through which a user account can interact with the implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0100] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0101] According to embodiments of this disclosure, this disclosure also provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, it implements the steps of the optimization method for the large model described in the above embodiments of this disclosure.

[0102] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0103] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An optimization method for a large model, wherein, The method includes: Model parameters are extracted from the first large model of the multi-head self-attention (MHA) structure to obtain a model parameter set, which includes a first type of model parameters and a second type of model parameters. The first large model is used to process different types of data, including text data, image data, and audio / video data. The first type of model parameters includes the weight matrix of the query parameter Q, key parameter K, and value parameter V in the MHA structure. The second type of model parameters includes the number of heads, embedding dimension, model parameters associated with the feedforward network layer in the first large model, and model parameters associated with the embedding layer in the first large model. Singular value decomposition (SVD) is performed on the first type of model parameters to obtain approximate matrix pairs of the first type of model parameters, wherein the approximate matrix pairs include the dimension-reduced matrix and the dimension-increasing matrix of the first type of model parameters; Based on the approximate matrix pairs of the first type of model parameters and the second type of model parameters, a second major model of a multi-head implicit self-attention MLA structure is generated. The multi-head implicit self-attention mechanism is an attention mechanism for natural language processing and sequence processing tasks. The MLA structure includes a first network layer for dimensionality reduction and a second network layer for dimensionality increase. The number of input dimensions of the first network layer is the same as the number of output dimensions of the second network layer; the number of output dimensions of the first network layer is the same as the number of input dimensions of the second network layer.

2. The method according to claim 1, wherein, The step of performing singular value decomposition (SVD) on the parameters of the first type of model to obtain approximate matrix pairs of the parameters of the first type of model includes: The first type of model parameters are subjected to SVD processing to obtain the decomposition results of the first type of model parameters, wherein the decomposition results include a left singular vector matrix, a singular value matrix and a right singular vector matrix; Based on the left singular vector matrix, singular value matrix, and right singular vector matrix, an approximate matrix pair of parameters for the first type of model is obtained.

3. The method according to claim 2, wherein, The step of obtaining approximate matrix pairs of the first type of model parameters based on the left singular vector matrix, singular value matrix, and right singular vector matrix includes: Determine the top K largest singular values ​​from the singular value matrix, where K is a natural number greater than or equal to 1; Based on the first K largest singular values, the left singular vector matrix, and the right singular vector matrix, an approximate matrix pair of parameters for the first type of model is obtained.

4. The method according to claim 3, wherein, The step of obtaining approximate matrix pairs of the first type of model parameters based on the first K largest singular values, the left singular vector matrix, and the right singular vector matrix includes: The first diagonal matrix is ​​formed by the first K largest singular values, and the second diagonal matrix is ​​obtained by transforming the first diagonal matrix. Based on the second diagonal matrix, the left singular vector matrix, and the right singular vector matrix, an approximate matrix pair of parameters for the first type of model is obtained.

5. The method according to claim 4, wherein, The step of obtaining the approximate matrix pair of the first type of model parameters based on the second diagonal matrix, the left singular vector matrix, and the right singular vector matrix includes: Perform matrix multiplication on the left singular vector matrix and the second diagonal matrix to obtain a first approximate matrix of the parameters of the first type of model, wherein the first approximate matrix is ​​the dimension reduction matrix; Matrix multiplication is performed on the second diagonal matrix and the right singular vector matrix to obtain a second approximate matrix of the parameters of the first type of model, wherein the second approximate matrix is ​​the increased-dimensional matrix.

6. The method according to claim 4, wherein, The process of transforming the first diagonal matrix to obtain the second diagonal matrix includes: Obtain the square root of each diagonal element in the first diagonal matrix, and obtain the second diagonal matrix based on the square root of each diagonal element.

7. The method according to any one of claims 1-6, wherein, The second major model for generating a multi-head implicit self-attention MLA structure based on the approximate matrix pair of the first type of model parameters and the second type of model parameters includes: Obtain the initial second-largest pre-trained model; Based on the approximate matrix pair, the parameters of the MLA structure in the initial second large model are configured, and based on the parameters of the second type of model, the parameters of the remaining network layers in the initial second large model other than the MLA structure are configured to obtain the second large model.

8. The method according to claim 7, wherein, The parameter configuration of the MLA structure in the second large model based on the approximate matrix pair includes: Obtain the bias vector of the parameters of the first type of model, and perform bias processing on the approximate matrix pair based on the bias vector to obtain the target approximate matrix pair; Based on the target approximation matrix pair, the parameters of the MLA structure in the second large model are configured.

9. The method according to claim 7, wherein, The step of configuring parameters for the remaining network layers in the second large model, excluding the MLA structure, based on the second type of model parameters includes: For the remaining network layers in the second major model i The remaining network layers are determined to correspond to the target network layer of the first large model. j ,in, i , j Natural numbers greater than or equal to 1; Determine the target network layer j The associated second type of model parameters, and based on the target network layer j The associated second type of model parameters for the remaining network layers i Configure the parameters.

10. The method according to claim 9, wherein, The first major model also includes a feedforward network (FNN) layer and an embedding layer, while the second major model reuses the structure of the FFN layer and the embedding layer in the first major model.

11. An optimization device for a large model, wherein, The device includes: An extraction module is used to extract model parameters from the first large model of the multi-head self-attention (MHA) structure to obtain a model parameter set. The model parameter set includes a first type of model parameters and a second type of model parameters. The first large model is used to process different types of data, including text data, image data, and audio / video data. The first type of model parameters includes the weight matrix of the query parameter Q, key parameter K, and value parameter V in the MHA structure. The second type of model parameters includes the number of heads, embedding dimension, model parameters associated with the feedforward network layer in the first large model, and model parameters associated with the embedding layer in the first large model. The acquisition module is used to perform singular value decomposition (SVD) on the first type of model parameters to obtain approximate matrix pairs of the first type of model parameters, wherein the approximate matrix pairs include the dimension-reduced matrix and the dimension-increasing matrix of the first type of model parameters; The generation module is used to generate a second major model of a multi-head implicit self-attention MLA structure based on the approximate matrix pairs of the first type of model parameters and the second type of model parameters. The multi-head implicit self-attention mechanism is an attention mechanism for natural language processing and sequence processing tasks. The MLA structure includes a first network layer that undergoes dimensionality reduction and a second network layer that undergoes dimensionality increase. The number of input dimensions of the first network layer is the same as the number of output dimensions of the second network layer; the number of output dimensions of the first network layer is the same as the number of input dimensions of the second network layer.

12. The apparatus according to claim 11, wherein, The acquisition module is used for: The first type of model parameters are subjected to SVD processing to obtain the decomposition results of the first type of model parameters, wherein the decomposition results include a left singular vector matrix, a singular value matrix and a right singular vector matrix; Based on the left singular vector matrix, singular value matrix, and right singular vector matrix, an approximate matrix pair of parameters for the first type of model is obtained.

13. The apparatus according to claim 12, wherein, The acquisition module is used for: Determine the top K largest singular values ​​from the singular value matrix, where K is a natural number greater than or equal to 1; Based on the first K largest singular values, the left singular vector matrix, and the right singular vector matrix, an approximate matrix pair of parameters for the first type of model is obtained.

14. The apparatus according to claim 13, wherein, The acquisition module is used for: The first diagonal matrix is ​​formed by the first K largest singular values, and the second diagonal matrix is ​​obtained by transforming the first diagonal matrix. Based on the second diagonal matrix, the left singular vector matrix, and the right singular vector matrix, an approximate matrix pair of parameters for the first type of model is obtained.

15. The apparatus according to claim 14, wherein, The acquisition module is used for: Perform matrix multiplication on the left singular vector matrix and the second diagonal matrix to obtain a first approximate matrix of the parameters of the first type of model, wherein the first approximate matrix is ​​the dimension reduction matrix; Matrix multiplication is performed on the second diagonal matrix and the right singular vector matrix to obtain a second approximate matrix of the parameters of the first type of model, wherein the second approximate matrix is ​​the increased-dimensional matrix.

16. The apparatus according to claim 14, wherein, The acquisition module is used for: Obtain the square root of each diagonal element in the first diagonal matrix, and obtain the second diagonal matrix based on the square root of each diagonal element.

17. The apparatus according to any one of claims 11-16, wherein, The generation module is used for Obtain the initial second-largest pre-trained model; Based on the approximate matrix pair, the parameters of the MLA structure in the initial second large model are configured, and based on the parameters of the second type of model, the parameters of the remaining network layers in the initial second large model other than the MLA structure are configured to obtain the second large model.

18. The apparatus according to claim 17, wherein, The generation module is used for: Obtain the bias vector of the parameters of the first type of model, and perform bias processing on the approximate matrix pair based on the bias vector to obtain the target approximate matrix pair; Based on the target approximation matrix pair, the parameters of the MLA structure in the second large model are configured.

19. The apparatus according to claim 17, wherein, The generation module is used for: For the remaining network layers in the second major model i The remaining network layers are determined to correspond to the target network layer of the first large model. j ,in, i , j Natural numbers greater than or equal to 1; Determine the target network layer j The associated second type of model parameters, and based on the target network layer j The associated second type of model parameters for the remaining network layers i Configure the parameters.

20. The apparatus according to claim 19, wherein, The first major model also includes a feedforward network (FNN) layer and an embedding layer, while the second major model reuses the structure of the FFN layer and the embedding layer in the first major model.

21. An electronic device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the method as described in any one of claims 1-10.

22. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-10.

23. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-10.

Images