Large language model expansion method, electronic equipment, storage medium and program product
By deleting and cross -fusion of architecture layers under the architecture layer under the large language model and the constraints of hardware resources, the problem of discontinuity of the semantic discontinuity of the large language model after expansion is solved, and the full extension and performance improvement of the model are realized.
Patent Information
- Application Number
- CN202411411774.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-10
- Publication Date
- 2025-01-24
AI Technical Summary
When the prior art expands the large language model, it leads to a semantic gap at the model stitching, resulting in semantic incoherence of the expanded large language model.
Obtain the replicated model by obtaining the original model and copying it, determine the target architecture layer number and delete layer number of the expanded target model based on the amount of hardware resources, delete the corresponding architecture layer to obtain the original sub-model and the replicated sub-model, and Cross-fusion is performed according to the order of its architecture layers to obtain the expanded target model.
It realizes the full expansion of the large language model, reduces the loss of model parameters, alleviates the semantic incoherence problem of the expanded target model, and improves the performance of the target model.
Smart Images
Figure CN119358514A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a large language model expansion method, electronic device, storage medium and program product. Background Art
[0002] Large Language Model (LLM) is an important technology in the field of natural language processing. It is a deep learning model trained based on massive text data. It can generate high-quality text, understand complex semantic relationships, and perform a variety of language tasks such as translation, question-answering, and summarization.
[0003] As the complexity of processing tasks increases and the generalization ability is insufficient, the model needs to be expanded. In the prior art, when expanding a large language model, the large language model is directly copied, and the last layer of the model before the copy is stitched with the first layer of the model after the copy to form a new model. This expansion method forms a semantic gap at the model stitching point, and causes the semantics of the expanded large language model to be incoherent. Summary of the invention
[0004] The embodiments of the present application provide a large language model expansion method, an electronic device, a storage medium, and a program product to alleviate the problems of semantic gap and semantic incoherence.
[0005] In a first aspect, an embodiment of the present application provides a large language model expansion method, including:
[0006] Acquire an original model and copy the original model to obtain a copy model, wherein the copy model has the same number of architecture layers and the same order of arrangement of the architecture layers as the original model;
[0007] Obtaining the resource amount of hardware resources of the hardware used to run the large speech model, and determining the number of target architecture layers of the expanded target model according to the resource amount of the hardware resources, wherein the number of target architecture layers after the copy is an even number and does not exceed the sum of the number of architecture layers of the original model and the copy model;
[0008] Determine the number of layers to be deleted according to the number of layers of the original model and the number of layers of the target model, wherein the number of layers to be deleted is an integer greater than or equal to 0;
[0009] Deleting the number of deleted architectural layers from the original model and the copy model respectively to obtain an original sub-model and a copy sub-model;
[0010] According to the order of the architectural layers of the original sub-model and the copied sub-model, they are split layer by layer and cross-fused to obtain the expanded target model.
[0011] In a possible implementation manner, determining the number of target architecture layers of the expanded target model according to the resource amount of the hardware resources includes:
[0012] Determining whether the resource amount of the hardware resources is greater than a preset resource amount, wherein the preset resource amount is determined according to the resource amount occupied by the expanded target model;
[0013] If yes, determining the target number of architecture layers to be the sum of the number of architecture layers of the original model and the number of architecture layers of the copied model;
[0014] If not, the target number of architecture layers is determined according to the resource amount and the resource amount of each architecture layer.
[0015] In a possible implementation manner, the hardware resource is a video memory resource of a graphics processor, and before determining whether the resource amount of the hardware resource is greater than a preset resource amount, the method further includes:
[0016] Obtaining a video memory occupancy coefficient during large language model training, where the video memory occupancy coefficient is obtained by summing a first occupancy coefficient of a model parameter, a second occupancy coefficient of a gradient parameter of a propagation algorithm, and a third occupancy coefficient of an optimizer state parameter;
[0017] Determining the resource amount of the original model according to the size of the video memory occupied by the original model and the video memory occupancy coefficient;
[0018] The preset resource amount is determined based on the resource amount of the original model and the number of copied models.
[0019] In a possible implementation manner, determining the target number of architecture layers according to the resource amount and the resource amount of each architecture layer includes:
[0020] Determine the video memory size that can be occupied by the expanded target model according to the video memory size and video memory occupancy coefficient of the current video memory resources;
[0021] Calculating the video memory size of each of the architecture layers, wherein the architecture layer includes an attention layer and a feedforward neural network layer, the video memory size of the attention layer is determined according to the hidden layer parameters, the video memory size of the feedforward neural network layer is determined according to the hidden layer parameters and the intermediate layer parameters, and the video memory size of the architecture layer is determined according to the video memory size of the attention layer and the feedforward neural network layer;
[0022] Determine the number of candidate layers based on the video memory size that can be occupied by the expanded target model and the video memory size of each architecture layer;
[0023] Determine whether the number of candidate layers is an even number,
[0024] If so, the candidate number of layers is determined to be the target architecture number of layers;
[0025] If not, select an even number less than the candidate number of layers as the target architecture number of layers
[0026] In a possible implementation manner, determining the number of layers to be deleted according to the number of architecture layers of the original model and the number of target architecture layers includes:
[0027] The target number of architecture layers is evenly divided to obtain the sub-target number of architecture layers;
[0028] The number of deleted layers is obtained according to the number of architecture layers of the original model and the number of architecture layers of the sub-target.
[0029] In a possible implementation, if the number of deleted layers is 0, the layer-by-layer segmentation and cross-fusion according to the arrangement order of the architecture layers of the original sub-model and the copied sub-model to obtain the expanded target model includes:
[0030] According to the arrangement order of the architecture layers of the original sub-model and the replica sub-model, the original sub-model and the replica sub-model are respectively segmented to obtain M original model layers and M replica model layers, wherein M is an integer greater than or equal to 2;
[0031] For each copy model layer, the nth copy model layer is arranged after the corresponding nth original model layer to obtain the expanded target model through cross-fusion.
[0032] In a possible implementation manner, if the number of deleted layers is not 0, deleting the architectural layers of the number of deleted layers from the original model and the copy model respectively to obtain the original sub-model and the copy sub-model includes:
[0033] Deleting the architecture layer at the end of the original model according to the number of deleted layers to obtain the original sub-model;
[0034] The architecture layer of the front end of the replication model is deleted according to the number of deleted layers to obtain the replication sub-model.
[0035] In a possible implementation, the layer-by-layer segmentation and cross-fusion according to the order of arrangement of the architecture layers of the original sub-model and the copied sub-model to obtain the expanded target model includes:
[0036] According to the arrangement order of the architecture layers of the original sub-model and the replica sub-model, the original sub-model and the replica sub-model are respectively segmented to obtain P original model layers and P replica model layers, wherein P is an integer greater than or equal to 2;
[0037] The first Q original model layers are arranged in sequence, the nth copy model layer is arranged after the n+Qth original model layer, and the last Q copy model layers are arranged in sequence to cross-fuse to obtain the expanded target model, wherein the value of Q is equal to the number of deleted layers.
[0038] In a second aspect, an embodiment of the present application provides a large language model expansion device, including:
[0039] A model acquisition module, used for acquiring an original model and copying the original model to obtain a copy model, wherein the copy model has the same number of architecture layers and the same order of arrangement of the architecture layers as the original model;
[0040] A resource determination module, used to obtain the resource amount of hardware resources of the hardware used to run the large speech model, and determine the target architecture layer number of the expanded target model according to the resource amount of the hardware resources, wherein the target architecture layer number after the copy is an even number and does not exceed the sum of the architecture layers of the original model and the copy model;
[0041] A layer number determination module, used to determine the number of layers to be deleted according to the number of architecture layers of the original model and the number of target architecture layers;
[0042] A processing module, used for deleting the architectural layers of the number of deleted layers from the original model and the copy model respectively, so as to obtain an original sub-model and a copy sub-model;
[0043] The fusion module is used to split and cross-fuse the original sub-model and the copied sub-model layer by layer according to the arrangement order of the architectural layers to obtain the expanded target model.
[0044] In a third aspect, an embodiment of the present application provides a large language model expansion device, including: a memory, a processor;
[0045] The memory stores computer-executable instructions;
[0046] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.
[0047] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.
[0048] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.
[0049] The large language model expansion method, electronic device, storage medium and program product provided in the embodiments of the present application realize the expansion of the original model by fusing the original model and the copied model obtained by copying the original model. According to the resource amount of the hardware resources when running the large language model, the target architecture layer number and the number of deleted layers of the expanded target model can be determined. After the original model and the copied model delete the corresponding architecture layers according to the number of deleted layers to obtain the original sub-model and the copied sub-model, the architecture layer arrangement order of the original sub-model and the copied sub-model is divided by layer and cross-fused to obtain the expanded target model. When expanding the original model, the number of deleted layers is determined by hardware resources, so that the large language model can be fully expanded, and the adjacent architecture layers of the original sub-model and the copied sub-model are fused by cross-fusion to alleviate the semantic incoherence problem of the expanded target model and improve the performance of the target model. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0051] Figure 1 Schematic diagram of the process of expanding the large language model provided in this application Figure 1 ;
[0052] Figure 1a A schematic diagram of the structure of the large language model expansion method provided in this application;
[0053] Figure 2 Schematic diagram of the process of expanding the large language model provided in this application Figure 2 ;
[0054] Figure 3 Schematic diagram of the process of expanding the large language model provided in this application Figure 3 ;
[0055] Figure 4 A schematic diagram of the structure of the large language model expansion device provided in this application;
[0056] Figure 5 A schematic diagram of the structure of the large language model expansion device provided in this application.
[0057] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0058] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0059] In the existing technology, when expanding a large language model, the Mixture of Experts (MoE) technology is usually used. The core of MoE is to decompose the large language model into multiple "experts", each of which focuses on processing different parts of the input data or different types of tasks. A gating network determines which expert should be activated under a given input. MoE allows large language models to be easily expanded to hundreds or thousands of experts, greatly increasing the capacity of large language models while maintaining good computational efficiency.
[0060] SOLAR is a large language model developed by Upstage AI, which introduces a model expansion method called Depth Up-Scaling (DUS). By copying the model, the last layer of the model before the copy and the first layer of the model after the copy are directly stitched together to form a new model. This method covers deep expansion and continuous pre-training. Unlike other large language model expansion methods using MoE technology, DUS can achieve efficiency improvements without complex modifications to the training and inference frameworks.
[0061] Although MoE can efficiently scale large language models, it requires major changes to the training and inference frameworks, which hinders wide applicability. At the same time, its implementation is complex, especially in the handling of dynamic routing and load imbalance, which challenges the compatibility and efficiency of hardware such as TPUs and GPUs, and requires expensive pre-training to achieve. SOLAR does not require complex modifications to the training and inference frameworks, and only requires continuous pre-training to improve performance. Its model expansion method directly stitches the last layer of the model before copying and the first layer of the model after copying to form a new model. This method will form a semantic gap at the model stitching point. Even if the method of sacrificing some intermediate layer parameters is adopted, there is still a problem of semantic incoherence, and the active sacrifice of model layers will cause performance loss, which requires performance recovery during a large amount of continuous pre-training.
[0062] Therefore, the large language model expansion method, electronic device, storage medium and program product provided by the present application, on the technical route of SOLAR, expands the original model after copying the original model to obtain the copied model, thereby avoiding the poor applicability of MOE caused by the major change of the training framework. At the same time, when the original model is expanded, the target architecture layer number of the expanded target model is determined by hardware resources, so as to determine the number of deleted layers of the original model and the copied model according to the target architecture layer number. When the number of deleted layers is 0, the architecture layers of the original model and the copied model are cross-fused in sequence to achieve full expansion of the large language model and reduce the semantic incoherence problem caused by parameter loss of the large language model. When the number of deleted layers is not 0, the architecture layers of the original sub-model and the copied sub-model deleted by the number of deleted layers are cross-fused in sequence. The sequential cross-fusion of the architecture layers alleviates the semantic incoherence and performance loss problems caused by the direct splicing method of SOLAR and reduces the model parameter loss.
[0063] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0064] Figure 1 Schematic diagram of the process of expanding the large language model provided in this application Figure 1 ,in, Figure 1a A schematic diagram of the structure of the large language model expansion method provided in this application; Figure 1 As shown, the method includes:
[0065] S101, obtaining an original model and copying the original model to obtain a copy model.
[0066] The original model is a large language model built using the Transformer architecture layer. The original model includes several architecture layers, which are designed as input layer, processing layer, output layer, etc. according to functional requirements. The input layer is responsible for receiving user input, the processing layer is used for the core mechanism processing of the model, and the output layer is used to output the model's prediction or response.
[0067] The architecture layer provides the necessary support and guarantee for the design, implementation, and application of large language models. The implementation quality of the architecture layer of large language models directly affects the stability and reliability of the entire model. A good architecture layer can ensure that the model remains efficient and accurate when processing large amounts of data.
[0068] The architectural layers in the large language model are arranged in order according to their functions to ensure the interaction and influence between the architectural layers.
[0069] When the original model is copied to obtain a copied model, the copied model has the same number of architecture layers and the same order of arrangement of the architecture layers as the original model.
[0070] For example, if the original model has 32 architectural layers, the duplicate model also has 32 architectural layers, and the arrangement order of the 32 architectural layers in the duplicate model is the same as the arrangement order of the architectural layers in the original model.
[0071] S102. Obtain the resource amount of hardware resources of the hardware used to run the large speech model, and determine the number of target architecture layers of the expanded target model according to the resource amount of the hardware resources.
[0072] The amount of hardware resources directly affects the training and operation efficiency of the large language model that can be run. When the hardware resources are too large relative to the large language model, it will lead to a waste of hardware resources; when the amount of hardware resources is less than the operation and training requirements of the large language model, the large language model will not be able to run normally.
[0073] Among them, the number of target architecture layers after copying is an even number and does not exceed the sum of the number of architecture layers of the original model and the copied model.
[0074] When the original model is expanded, the target architecture layer number of the expanded target model can be determined by obtaining the resource amount of the hardware resources, so as to achieve full expansion of the target model with the support of the hardware resources and reduce the architecture layer loss of the original model and the copied model.
[0075] When the sum of the resources required for the architecture layers of the original model and the copied model is less than or equal to the amount of hardware resources, the number of target architecture layers after replication is the sum of the architecture layers of the original model and the copied model; when the sum of the resources required for the architecture layers of the original model and the copied model is greater than the amount of hardware resources, the architecture layers of the original model and the copied model need to be pruned and then summed, and the number of architecture layers retained by the original model and the copied model after the architecture layers are pruned is the same.
[0076] S103. Determine the number of layers to be deleted according to the number of architecture layers of the original model and the number of architecture layers of the target model.
[0077] After the target number of architecture layers is determined according to the amount of hardware resources, the target number of architecture layers is averaged to obtain the number of architecture layers retained in the original model after expansion, thereby obtaining the number of deleted layers according to the number of architecture layers of the original model and the number of architecture layers retained after expansion. The number of deleted layers is an integer greater than or equal to 0.
[0078] That is, when the sum of the resources required for the architecture layers of the original model and the copied model is less than or equal to the amount of hardware resources, the number of layers to be deleted is 0; when the sum of the resources required for the architecture layers of the original model and the copied model is greater than the amount of hardware resources, the number of layers to be deleted is the number of architecture layers of the original model minus half of the number of target architecture layers.
[0079] For example, if the resource size of hardware resource A is 200G, the number of parameters of the large language model that can be run by hardware resource A is 10B (B is 10 9 , i.e., one billion);
[0080] If the parameter amount of the original model C is 3B, the parameter amount of the expanded target model is 3B*2=6B. If the expanded target model is smaller than the resource amount of the hardware resource A, the number of target architecture layers of the expanded target model is the sum of the architecture layers of the original model and the copied model, and the number of deleted layers is 0;
[0081] If the parameter size of the original model D is 6B, the expanded target model should be 6B*2=12B. 12B is larger than the 10B large language model that can be run by hardware resource A. In this case, the architecture layers of the original model and the copied model need to be deleted; if the parameter size of each architecture layer is 0.2B, the target architecture layers of the large language model that can be run by hardware resource A are 10 / 0.2=50 layers, and the original model D and the copied model of the original model D are both 6 / 0.2=30 layers. After dividing the 50 layers equally, there are 25 layers. In this case, 5 layers should be deleted from the original model B and the copied model of the original model D respectively.
[0082] S104. Delete the architecture layers of the deleted number from the original model and the copied model respectively to obtain the original sub-model and the copied sub-model.
[0083] By deleting the architecture layers of the original model and the copied model according to the number of deleted layers, the original sub-model and the copied sub-model to be fused can be obtained.
[0084] S105, dividing and cross-merging the original sub-model and the copied sub-model layer by layer according to the order of arrangement of the architecture layers to obtain an expanded target model.
[0085] Since the original sub-model and the copied sub-model include several architecture layers, and the architecture layers are arranged in sequence, cross-fusion is performed according to the order of the architecture layers in the original sub-model and the copied sub-model, that is, the architecture layer numbers are arranged in order from small to large, among which the layers with the same sequence number are arranged adjacently, so that the expanded target model can be obtained.
[0086] For example, the architecture layers of the original model E are {1, 2, ..., 10};
[0087] The architecture layers of the replication model E_copy are {1, 2, ..., 10};
[0088] If the number of deleted layers is 0, and the original model architecture layer with the same architecture layer sequence number is placed before the architecture layer of the copied model when setting cross fusion, the architecture layer of the expanded target model is {1, 1, 2, 2..., 10, 10};
[0089] If the number of deleted layers is 2, and the original model deletes the last 2 layers of the structure, and the copied model deletes the first 2 layers of the structure, then:
[0090] The architecture layers of the original sub-model F are {1, 2, ..., 8};
[0091] The architecture layers of the replication model F_copy are {3, 4, ..., 10};
[0092] When setting cross-fusion, the original sub-model architecture layer with the same architecture layer sequence number is placed before the architecture layer of the copied sub-model, then the architecture layer of the expanded target model is {1, 2, 3, 3..., 8, 8, 9, 10}.
[0093] The large language model expansion method provided in the embodiment of the present application determines the target architecture layer number and the number of deleted layers of the expanded target model by the resource amount of the hardware resources, and then deletes the architecture layers of the original model and the copied model to obtain the original sub-model and the copied sub-model. The original sub-model and the copied sub-model are divided and cross-fused according to the architecture layer arrangement order to obtain the expanded target model. When expanding the original model, the number of deleted layers is determined by the hardware resources. When the number of deleted layers is 0, the original sub-model and the copied sub-model are the original model and the copied model. The original model and the copied sub-model are There is no loss in the architecture layers and parameters of the copied model. The architecture layers of the original model and the copied model are divided and cross-fused in sequence, which can achieve full expansion of the large language model without sacrificing the number of layers of the large language model architecture layer, reducing the parameter loss of the original model and the copied model, and making the semantic coherence of the expanded target model; when the number of deleted layers is not 0, the adjacent architecture layers of the original sub-model and the copied sub-model are fused by cross-fusion. The fusion and splicing of adjacent architecture layers can promote cross-level meaning alignment, alleviate the semantic incoherence problem of the expanded target model, and improve the performance of the target model.
[0094] Figure 2 Schematic diagram of the process of expanding the large language model provided in this application Figure 2 In this embodiment, Figure 1 Based on the embodiment, a specific implementation method for obtaining the number of target architecture layers is described in detail. The method includes:
[0095] S201, obtaining an original model and copying the original model to obtain a copy model.
[0096] When expanding a large language model, the original model is copied to obtain a copy model, and the copy model has the same number of architecture layers and the same order of arrangement of the architecture layers as the original model.
[0097] S202: Obtain the resource amount of hardware resources used to run the large speech model.
[0098] When a large language model is expanded, the number of parameters of the expanded large language model increases accordingly. These parameters need to be stored in the video memory so that they can be updated and optimized during the training process. The size of the video memory directly determines the scale to which the model can be expanded. Therefore, when a large language is expanded, the hardware resources are the video memory resources of the graphics processor.
[0099] S203: Obtain the video memory occupancy coefficient during large language model training.
[0100] When a large language model is trained using the AdamW optimizer, FP32 (32-bit floating point numbers), and BF16 (16-bit floating point mixed precision), the memory usage is affected by the model parameters, gradients, and optimizer status.
[0101] The video memory occupancy coefficient is determined according to the first occupancy coefficient of the model parameter, the second occupancy coefficient of the gradient parameter of the propagation algorithm, and the third occupancy coefficient of the optimizer state parameter, that is, the video memory occupancy coefficient is the sum of the first occupancy coefficient, the second occupancy coefficient and the third occupancy coefficient.
[0102] Among them, when the model parameters of the large language model are determined, under FP32 precision, each parameter usually occupies 4 bytes (32 bits), and under BF16 precision, each parameter usually occupies 2 bytes (16 bits). Since the large language model is trained with mixed precision, that is, 2 bytes and 4 bytes in forward propagation and back propagation respectively, each parameter of the model requires 6 bytes when training the large language model. That is, if the number of parameters of the large language model is P_n, the video memory occupied by the model parameters is 6*P_n bytes, and the first occupancy coefficient is 6.
[0103] When the gradient of the large language model is determined, the video memory occupied is determined according to the number and precision of the parameters. During mixed precision training, that is, 2 bytes and 4 bytes are respectively used in forward propagation and backward propagation. Therefore, the gradient requires 6 bytes for each parameter when training the large language model. That is, if the number of parameters of the large language model is P_n, the video memory occupied by the gradient is 6*P_n bytes, and the second occupancy coefficient is 6.
[0104] When the optimizer state parameters of the large language model are determined, the AdamW optimizer includes first-order moment estimation and second-order moment estimation. Under FP32 precision, each of these state information occupies the same space as the parameters (i.e. 4 bytes). Therefore, the state information of the AdamW optimizer will increase the memory usage to several times the model parameters and gradients. Usually, the multiple is usually 2 times. If the number of parameters of the large language model is P_n, the memory occupied by the optimizer state parameters is 8*P_n bytes, and the second occupancy coefficient is 8.
[0105] When the first occupancy coefficient is 6, the second occupancy coefficient is 6, and the third occupancy coefficient is 8, the video memory occupancy coefficient is the sum of the first occupancy coefficient, the second occupancy coefficient, and the third occupancy coefficient, that is, the video memory occupancy coefficient is 20.
[0106] S204: Determine the resource amount of the original model according to the size of the video memory occupied by the original model and the video memory occupancy coefficient.
[0107] The resource amount of the original model, that is, the resource amount required for the original model during training, can be determined based on the product of the video memory size occupied by the original model and the video memory occupancy coefficient.
[0108] For example, if the video memory size occupied by the original model is 7B, that is, the parameter size of the original model is 7B, when the video memory occupancy coefficient is 20, the resource size of the original model should be 20*7B=140B.
[0109] S205: Determine a preset resource amount according to the resource amount of the original model and the number of duplicate models.
[0110] The preset resource amount is determined according to the resource amount occupied by the expanded target model.
[0111] The amount of resources occupied by the expanded target model is obtained by adding the amount of resources of the original model and the amount of resources of the copied model. Since the amount of resources of the copied model is the same as that of the original model, the preset amount of resources can be determined based on the amount of resources of the original model and the number of copied models.
[0112] For example, if the resource volume of the original model is 140B and there is 1 copy model, the expanded target model should be the original model plus 1 copy model, and the preset resource volume is 140B*2=280B.
[0113] S206: Determine whether the resource amount of the hardware resources is greater than a preset resource amount.
[0114] The resource volume of the hardware resources is limited by its configuration or design and has a clear limit, for example, the resource volume of the hardware resources is 200G.
[0115] By judging whether the amount of hardware resources is greater than the preset amount of resources, it is possible to judge whether the original model can be fully expanded after being copied.
[0116] If yes, then execute step S207:
[0117] S207. Determine the target architecture layer number as the sum of the architecture layer numbers of the original model and the copied model.
[0118] For example, if the original model has 24 layers, and the corresponding copied model also has 24 layers, then the target architecture layer is 48 layers.
[0119] If not, then execute the following steps S208-S213:
[0120] S208. Determine the size of the video memory that can be occupied by the expanded target model according to the video memory size and the video memory occupancy coefficient of the current video memory resources.
[0121] Referring to the above example, if the current video memory resource has a video memory size of 200G and the video memory occupancy factor is 20, then the video memory size that can be occupied by the expanded target model should be 200G / 20=10B.
[0122] Among them, 1 parameter is 1 byte, 1G is 1024 3 Bytes, usually, 1G can be regarded as 10 3 Bytes, and 1B is also 10 3 Parameters can be understood as 1G and 1B parameters.
[0123] S209: Calculate the video memory size of each architecture layer.
[0124] Among them, the architecture layer includes an attention layer and a feedforward neural network layer. The memory size of the attention layer is determined according to the hidden layer parameters, the memory size of the feedforward neural network layer is determined according to the hidden layer parameters and the intermediate layer parameters, and the memory size of the architecture layer is determined according to the memory size of the attention layer and the feedforward neural network layer.
[0125] Among them, hidden_size represents the hidden layer parameters, intermediate_size represents the FFN intermediate layer parameters, and the video memory size of the attention layer is: 4*hidden_size 2 , the video memory size of the feedforward neural network layer is: 3*hidden_size*intermediate_size, then the video memory size of the architecture layer is: 4*hidden_size 2 +3*hidden_size*intermediate_size.
[0126] Taking the large language model Llama2 as an example, in Llama2, hidden_size is 4096 and intermediate_size is 11008. Therefore, the parameter p_n of each architecture layer is: 4*4096*4096+3*4096*11008=202375168, that is, the video memory size of each architecture layer is about 0.202B.
[0127] S210 , determining the number of candidate layers based on the video memory size that can be occupied by the expanded target model and the video memory size of each architecture layer.
[0128] After determining the memory size that can be occupied by the expanded target model based on the memory size of the current memory resources, the number of candidate layers can be determined according to the memory size of each architecture layer, where the number of candidate layers = the memory size that can be occupied by the expanded target model / the memory size of each architecture layer.
[0129] Referring to the above example, if the memory size that can be occupied by the expanded target model is 10B, and the memory size of each architecture layer is 0.202B, then the number of candidate layers = 10 / 0.202 = 49.5.
[0130] S211. Determine whether the number of candidate layers is an even number.
[0131] Since the expanded target model consists of the original model and the copied model, the number of target architecture layers of the expanded target model should be an even number.
[0132] If yes, then execute step S212:
[0133] S212: Determine the candidate number of layers as the target architecture number of layers.
[0134] If not, execute step S213:
[0135] S213. Select an even number that is smaller than the candidate number of layers as the target architecture number of layers.
[0136] Referring to the above example, if the candidate number of layers is 49.5, the even number 48 is taken down as the target architecture number of layers.
[0137] The large language model expansion method provided in the embodiment of the present application obtains the resource amount of the original resource and the resource amount of the copied resource according to the video memory size occupied by the original model and the video memory occupancy coefficient, obtains the preset resource amount by summing the resource amount of the original model and the resource amount of the copied model, and compares the preset resource amount with the resource amount of the hardware resources to determine whether the original model can be fully expanded after being copied. When the resource amount of the hardware resources is greater than the preset resource amount, the target number of architecture layers is the sum of the number of architecture layers of the original model and the copied model; when the resource amount of the hardware resources is not greater than the preset resource amount, the number of candidate layers can be determined based on the video memory size that can be occupied by the expanded target model and the video memory size of each architecture layer, and the target number of architecture layers is obtained after determining whether the number of candidate layers is an even number.
[0138] Figure 3 Schematic diagram of the process of expanding the large language model provided in this application Figure 3 In this embodiment, Figure 1 and / or Figure 2 Based on the embodiment, a method of cross-fusion based on the number of target architecture layers is described in detail. The method includes:
[0139] S301, evenly divide the number of target architecture layers to obtain the number of sub-target architecture layers.
[0140] Since the expanded target model consists of the original model and the copied model, and the target architecture layer number is the total architecture layer number of the expanded target model, the relationship between the architecture layers of the original model or the copied model can be determined through the sub-target architecture layer number.
[0141] For example, if the number of layers of the target architecture is 48, then after equal division, the number of layers of the sub-target architecture should be 48 / 2=24.
[0142] S302. Obtain the number of layers to be deleted according to the number of architectural layers of the original model and the number of architectural layers of the sub-target.
[0143] The number of deleted layers can be obtained by subtracting the number of sub-target architecture layers from the number of architecture layers of the original model.
[0144] For example, if the number of layers in the original model is 32 and the number of layers in the sub-target architecture is 24, then the number of layers to be deleted is 32-24=8. If the number of layers in the original model is 24 and the number of layers in the sub-target architecture is 24, then the number of layers to be deleted is 0.
[0145] S303 , respectively deleting the architecture layers of the deleted number of layers from the original model and the copy model to obtain the original sub-model and the copy sub-model.
[0146] When the number of deleted layers is 0, the original sub-model is the original model, and the copied sub-model is the copied model.
[0147] When the number of deleted layers is not 0, the original sub-model is part of the original model, and the copied sub-model is part of the copied model.
[0148] S304: If the number of deleted layers is 0, the original sub-model and the copied sub-model are divided according to the arrangement order of their architecture layers to obtain M original model layers and M copied model layers.
[0149] Wherein, M is an integer greater than or equal to 2.
[0150] S305 , for each copy model layer, arranging the nth copy model layer after the corresponding nth original model layer, and obtaining an expanded target model by cross-fusion.
[0151] For example, if the original sub-model Y has 24 architecture layers after segmentation: {1, 2, ..., 23, 24};
[0152] The architecture layers of the replication sub-model Y_copy are: {1, 2, ..., 23, 24};
[0153] The architecture layers of the target model after cross-fusion expansion are: {1, 1, 2, 2, ..., 23, 23, 24, 24}.
[0154] S306: If the number of deleted layers is not 0, delete the architecture layer at the end of the original model according to the number of deleted layers to obtain the original sub-model.
[0155] S307. Delete the architecture layer of the front end of the replication model according to the number of deleted layers to obtain the replication sub-model.
[0156] S308. According to the arrangement order of the architecture layers of the original sub-model and the copied sub-model, the original sub-model and the copied sub-model are divided respectively to obtain P original model layers and P copied model layers.
[0157] Wherein, P is an integer greater than or equal to 2.
[0158] S309, arranging the first Q original model layers in order, arranging the nth copy model layer to after the n+Qth original model layer, and arranging the last Q copy model layers in order to obtain the expanded target model by cross-fusion.
[0159] The value of Q is equal to the number of deleted layers.
[0160] For example, if the original model L has 32 layers of architecture after segmentation: {1, 2, ..., 31, 32}, and the number of deleted layers is 8, then the architecture of the original sub-model L' is: {1, 2, ..., 23, 24};
[0161] After the replication model L_copy is split, it has 32 layers of architecture: {1, 2, ..., 31, 32}, and the architecture layers of the replication sub-model L'_copy are: {9, 10, ..., 31, 32};
[0162] The architecture layers of the target model after cross-fusion expansion are: {1, 2, 3, 4, 5, 6, 7, 8, 9, 9, 10, 10, …, 21, 21, 22, 22, 23, 23, 24, 24, 25, 26, 27, 28, 29, 30, 31, 32}.
[0163] The large language model expansion method provided in the embodiment of the present application obtains the sub-target architecture layer number after evenly dividing the target architecture layer number, obtains the number of deleted layers by subtracting the sub-target architecture layer number from the architecture layer number of the original model, deletes the architecture layers of the original model and the copy model according to the number of deleted layers to obtain the original sub-model and the copy sub-model, and when the number of deleted layers is 0, the original sub-model and the copy sub-model are the original model and the copy model, and after the original model and the copy model are divided respectively, the M original model layers and the M copy model layers are arranged in sequence, and for the original model layers and the copy model layers with the same sequence number, the copy model layers are arranged after the corresponding original model layers to obtain the expanded target model; and when When the number of deleted layers is not 0, the corresponding architecture layers of the original model and the copied model are deleted according to the preset deletion rules and the number of deleted layers. The deleted original sub-model and the copied sub-model have an overlapping part. The first Q original model layers are arranged in sequence, the nth copied model layer is arranged after the n+Qth original model layer, and the last Q copied model layers are arranged in sequence. Under the resource limit of hardware resources, the maximum expansion of the original model is achieved, and the loss of model parameters is reduced. The architecture layers of the original sub-model and the copied sub-model are fused in sequence and crosswise, so that the cross-layer gradient between adjacent architecture layers of the architecture layer of the fused target model is reduced, and the semantic gap and semantic incoherence of the expanded target model are alleviated.
[0164] Figure 4 A schematic diagram of the structure of the large language model expansion device provided in this application, such as Figure 4 As shown, the large language model expansion device 40 provided in this embodiment includes:
[0165] A model acquisition module 401 is used to acquire an original model and copy the original model to obtain a copy model, wherein the copy model has the same number of architecture layers and the same order of arrangement of the architecture layers as the original model;
[0166] A resource determination module 402, used to obtain the resource amount of hardware resources of the hardware used to run the large speech model, and determine the target architecture layer number of the expanded target model according to the resource amount of the hardware resources, wherein the target architecture layer number after the copy is an even number and does not exceed the sum of the architecture layers of the original model and the copy model;
[0167] The layer number determination module 403 is used to determine the number of layers to be deleted according to the number of architecture layers of the original model and the number of target architecture layers;
[0168] Processing module 404, for deleting the architectural layers of the deleted number of layers from the original model and the copy model respectively, to obtain the original sub-model and the copy sub-model;
[0169] The fusion module 405 is used to divide and cross-fuse the original sub-model and the copied sub-model according to the order of the architectural layer arrangement to obtain the expanded target model.
[0170] The large language model expansion device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effects are similar, and are not described in detail in this embodiment.
[0171] Figure 5 This is a schematic diagram of the structure of the large language model expansion device provided in this application. Figure 5 As shown, the electronic device 50 provided in this embodiment includes: at least one processor 501 and a memory 502. Optionally, the device 50 also includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected via a bus 504.
[0172] In a specific implementation process, at least one processor 501 executes the computer-executable instructions stored in the memory 502, so that at least one processor 501 executes the above method.
[0173] The specific implementation process of the processor 501 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.
[0174] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor.
[0175] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.
[0176] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.
[0177] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.
[0178] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.
[0179] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.
[0180] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.
[0181] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.
[0182] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0183] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0184] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0185] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.
[0186] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.
Claims
1. A large language model expansion method, characterized in that: include: Acquire an original model and copy the original model to obtain a copy model, wherein the copy model has the same number of architecture layers and the same order of arrangement of the architecture layers as the original model; Obtaining the resource amount of hardware resources of the hardware used to run the large speech model, and determining the number of target architecture layers of the expanded target model according to the resource amount of the hardware resources, wherein the number of target architecture layers after the copy is an even number and does not exceed the sum of the number of architecture layers of the original model and the copy model; Determine the number of layers to be deleted according to the number of layers of the original model and the number of layers of the target model, wherein the number of layers to be deleted is an integer greater than or equal to 0; Deleting the number of deleted architectural layers from the original model and the copy model respectively to obtain an original sub-model and a copy sub-model; According to the order of the architectural layers of the original sub-model and the copied sub-model, they are split layer by layer and cross-fused to obtain the expanded target model.
2. The method according to claim 1, characterized in that Determining the number of target architecture layers of the expanded target model according to the resource amount of the hardware resources includes: Determining whether the resource amount of the hardware resources is greater than a preset resource amount, wherein the preset resource amount is determined according to the resource amount occupied by the expanded target model; If yes, determining the target number of architecture layers to be the sum of the number of architecture layers of the original model and the number of architecture layers of the copied model; If not, the target number of architecture layers is determined according to the resource amount and the resource amount of each architecture layer.
3. The method according to claim 2, characterized in that The hardware resource is a video memory resource of a graphics processor. Before determining whether the resource amount of the hardware resource is greater than a preset resource amount, the method further includes: Obtaining a video memory occupancy coefficient during large language model training, where the video memory occupancy coefficient is obtained by summing a first occupancy coefficient of a model parameter, a second occupancy coefficient of a gradient parameter of a propagation algorithm, and a third occupancy coefficient of an optimizer state parameter; Determining the resource amount of the original model according to the size of the video memory occupied by the original model and the video memory occupancy coefficient; The preset resource amount is determined based on the resource amount of the original model and the number of copied models.
4. The method according to claim 3, characterized in that The step of determining the target number of architecture layers according to the amount of resources and the amount of resources of each architecture layer includes: Determine the video memory size that can be occupied by the expanded target model according to the video memory size and video memory occupancy coefficient of the current video memory resources; Calculating the video memory size of each of the architecture layers, wherein the architecture layer includes an attention layer and a feedforward neural network layer, the video memory size of the attention layer is determined according to the hidden layer parameters, the video memory size of the feedforward neural network layer is determined according to the hidden layer parameters and the intermediate layer parameters, and the video memory size of the architecture layer is determined according to the video memory size of the attention layer and the feedforward neural network layer; Determine the number of candidate layers based on the video memory size that can be occupied by the expanded target model and the video memory size of each architecture layer; Determine whether the number of candidate layers is an even number, If so, the candidate number of layers is determined to be the target architecture number of layers; If not, an even number smaller than the candidate number of layers is selected as the target architecture number of layers.
5. The method according to claim 1, characterized in that The step of determining the number of layers to be deleted according to the number of architecture layers of the original model and the number of architecture layers of the target model includes: The target number of architecture layers is evenly divided to obtain the sub-target number of architecture layers; The number of deleted layers is obtained according to the number of architecture layers of the original model and the number of architecture layers of the sub-target.
6. The method according to claim 5, characterized in that If the number of deleted layers is 0, the layer-by-layer segmentation and cross-fusion according to the arrangement order of the architecture layers of the original sub-model and the copied sub-model to obtain the expanded target model includes: According to the arrangement order of the architecture layers of the original sub-model and the replica sub-model, the original sub-model and the replica sub-model are respectively segmented to obtain M original model layers and M replica model layers, wherein M is an integer greater than or equal to 2; For each copy model layer, the nth copy model layer is arranged after the corresponding nth original model layer to obtain the expanded target model through cross-fusion.
7. The method according to claim 5, characterized in that If the number of deleted layers is not 0, deleting the architectural layers of the deleted number from the original model and the copy model respectively to obtain the original sub-model and the copy sub-model includes: Deleting the architecture layer at the end of the original model according to the number of deleted layers to obtain the original sub-model; The architecture layer of the front end of the replication model is deleted according to the number of deleted layers to obtain the replication sub-model.
8. The method according to claim 7, characterized in that The method of dividing the original sub-model and the copied sub-model by layers and performing cross-fusion according to the order of the architectural layers to obtain the expanded target model includes: According to the arrangement order of the architecture layers of the original sub-model and the replica sub-model, the original sub-model and the replica sub-model are respectively segmented to obtain P original model layers and P replica model layers, wherein P is an integer greater than or equal to 2; The first Q original model layers are arranged in sequence, the nth copy model layer is arranged after the n+Qth original model layer, and the last Q copy model layers are arranged in sequence to cross-fuse to obtain the expanded target model, wherein the value of Q is equal to the number of deleted layers.
9. A large language model expansion device, characterized in that: include: A model acquisition module, used for acquiring an original model and copying the original model to obtain a copy model, wherein the copy model has the same number of architecture layers and the same order of arrangement of the architecture layers as the original model; A resource determination module, used to obtain the resource amount of hardware resources of the hardware used to run the large speech model, and determine the target architecture layer number of the expanded target model according to the resource amount of the hardware resources, wherein the target architecture layer number after the copy is an even number and does not exceed the sum of the architecture layers of the original model and the copy model; A layer number determination module, used to determine the number of layers to be deleted according to the number of architecture layers of the original model and the number of target architecture layers; A processing module, used for deleting the architectural layers of the number of deleted layers from the original model and the copy model respectively, so as to obtain an original sub-model and a copy sub-model; The fusion module is used to split and cross-fuse the original sub-model and the copied sub-model layer by layer according to the arrangement order of the architectural layers to obtain the expanded target model.
10. A large language model expansion device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 8.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 8 when executed by a processor.
12. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 8 when being executed by a processor.