Large language model training method, device, electronic device and medium

Abstract

The present invention provides a training method, device, electronic device and medium for a large language model. The training method for the large language model includes: initializing the large language model to be trained according to the target sequence length M and the maximum length N supported by the hardware of the current device used for training; reading a text sequence as a training sample; dynamically preprocessing the text sequence according to the target sequence length M and the maximum length N to obtain data blocks; wherein the length of the data block matches the maximum length N, and the text sequence is padded when the sequence length of the text sequence is less than the length threshold; training the large language model according to the data blocks to perform natural language processing based on the trained large language model. The present invention can realize effective ultra-long text training under limited hardware resources, making large-scale language model training more economical and efficient.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a training method, device, electronic equipment and medium for a large language model. Background Art

[0002] In recent years, with the rapid development of deep learning technology, large language models (for example, Transformer models) have achieved remarkable results in fields such as natural language processing. Taking the Transformer model as an example, it consists of a multi-head self-attention layer and a fully connected layer. The self-attention layer is a model layer that can handle inputs of different lengths. However, the computational complexity of the self-attention mechanism grows quadratically with the increase in the length of the input sequence, which limits its application on long sequences. Summary of the invention

[0003] In view of this, the embodiments of the present disclosure provide a method, device, electronic device and medium for training a large language model, which at least partially solve the problems existing in the prior art.

[0004] In a first aspect, the present disclosure provides a method for training a large language model, which includes:

[0005] Initialize the large language model to be trained according to the target sequence length M and the maximum length N supported by the hardware of the current device used for training, where M>N and M and N are both positive integers;

[0006] Read the text sequence as training sample;

[0007] Dynamically preprocessing the text sequence according to the target sequence length M and the maximum length N to obtain data blocks; wherein the length of the data block matches the maximum length N, and the text sequence is padded when the sequence length of the text sequence is less than a length threshold;

[0008] The large language model is trained according to the data blocks to perform natural language processing based on the trained large language model.

[0009] According to a specific implementation of an embodiment of the present disclosure, the large language model to be trained is initialized according to the target sequence length M and the maximum length N supported by the hardware of the current device used for training, including:

[0010] When the large language model is initialized, the cardinality and factor parameters of the positional encoding generation function are adjusted according to the target sequence length M and the maximum length N supported by the hardware of the current device to set the positional encoding of the large language model.

[0011] According to a specific implementation of the embodiment of the present disclosure, the formulas for setting the base and factor parameters are as follows:

[0012] factor=M / (N*4)

[0013] base = 10000.0 * factor * 32

[0014] Where base represents the cardinality and factor represents the factor.

[0015] The expression of the position encoding generation function is:

[0016] If i is even

[0017] If i is an odd number

[0018] Where pos is the position index, i is the dimension index, and d is the dimension of the large language model.

[0019] According to a specific implementation of the embodiment of the present disclosure, the text sequence is dynamically preprocessed according to the target sequence length M and the maximum length N to obtain data blocks, including:

[0020] Determine whether the ratio of the sequence length L of the text sequence to the target sequence length M is greater than a preset threshold;

[0021] If yes, split the text sequence into multiple data blocks of length N;

[0022] If not, pad the length of the text sequence to half of the target sequence length M to obtain a padded sequence, and then split the padded sequence into multiple data blocks of length N;

[0023] Among them, the position marks between data blocks are not reset during training.

[0024] According to a specific implementation of the embodiment of the present disclosure, the text sequence is a word unit sequence generated after the input text data is processed by a word segmenter; the threshold is 1 / 8.

[0025] According to a specific implementation method of the embodiment of the present disclosure, when splitting into data blocks, if the length of the last data block is less than N, the data block is padded so that its length reaches N.

[0026] According to a specific implementation manner of an embodiment of the present disclosure, in a filling sequence, the filled data are evenly distributed between two adjacent data of the text sequence or randomly distributed.

[0027] In a second aspect, the present disclosure provides a large language model training device, which includes:

[0028] an initialization unit, used to initialize the large language model to be trained according to a target sequence length M and a maximum length N supported by hardware of a current device for training, wherein M>N and both M and N are positive integers;

[0029] A reading unit, used for reading a text sequence as a training sample;

[0030] A preprocessing unit, used for dynamically preprocessing the text sequence according to the target sequence length M and the maximum length N to obtain data blocks; wherein the length of the data block matches the maximum length N, and the text sequence is padded when the sequence length of the text sequence is less than a length threshold;

[0031] A training unit is used to train the large language model according to the data blocks, so as to perform natural language processing based on the trained large language model.

[0032] In a third aspect, an embodiment of the present disclosure provides an electronic device, the electronic device comprising:

[0033] at least one processor; and,

[0034] a memory communicatively connected to the at least one processor; wherein,

[0035] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned large language model training method.

[0036] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the above-mentioned large language model training method.

[0037] In summary, compared with the prior art, this embodiment has the following advantages:

[0038] 1. By dynamically generating position codes, limited video memory resources are effectively utilized, and the length of the sequence processed by the model is expanded;

[0039] 2. The data preprocessing method enables the model to process text blocks of appropriate size in each training step and adapt to inputs of different lengths while maintaining the ability to capture long-distance dependencies.

[0040] In summary, this embodiment ensures that effective long text training can be achieved under limited hardware resources, making large-scale language model training more economical and efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific implementation methods and drawings. Here, the exemplary implementation methods of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0042] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0043] Figure 1 A schematic diagram of a flow chart of a method for training a large language model provided in the first embodiment of the present invention;

[0044] Figure 2 A schematic diagram for uniformly filling a text sequence;

[0045] Figure 3 A schematic diagram for dividing a text sequence;

[0046] Figure 4 A schematic diagram of the structure of a large language model training device provided in the second embodiment of the present invention.

[0047] Figure 5 An exemplary structural diagram of a device capable of implementing the method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0048] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0049] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0050] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0051] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0052] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0053] See also Figure 1 The first embodiment of the present invention provides a method for training a large language model, which can be executed by an electronic device, and in particular, by one or more processors in the electronic device, to implement the following steps:

[0054] S101, initializing a large language model to be trained according to a target sequence length M and a maximum length N supported by hardware of a current device used for training, wherein M>N and both M and N are positive integers.

[0055] In this embodiment, the electronic device may be a device with computing capabilities such as a laptop computer, a desktop computer, a workstation, a server, a field programmable gate array (FPGA), etc.

[0056] When the large language model is initialized, the position code is first generated. This embodiment uses a method based on rotational position encoding (RoPE, Rotary Position Embedding) to generate the position code.

[0057] The reason for using RoPE is that RoPE is a mathematically unified encoding method for absolute and relative positions. RoPE can be used to represent tokens at different positions. Under this premise, after splitting a long sequence, tokens at different positions can still be marked. For example, a long sequence of tokens with a length of 4096 can be split into four token subsequences of 0-1024, 1025-2048, 2048-3072, and 3076-4096 by RoPE. The existing position encoding method is to split it into 4 subsequences of 0-1024. Therefore, this embodiment maintains the ability to capture long-distance dependencies.

[0058] In this embodiment, the generation of RoPE position coding is affected by two main parameters, namely, factor and base, the former determines the sampling step size, and the latter determines the frequency of position coding.

[0059] For the target sequence length M (such as 256k tokens) and the maximum length N supported by the training hardware, the parameter setting formula is as follows:

[0060] factor=M / (N*4)

[0061] base = 10000.0 * factor * 32

[0062] Here, the target sequence length M may be the maximum input token length supported by the large language model.

[0063] After that, according to the above parameters factor and base, the positional encoding generation function can be formalized as:

[0064] If i is even

[0065] If i is an odd number

[0066] Where pos is the position index, i is the dimension index, and d is the dimension of the model.

[0067] Here, the maximum length N supported by the training hardware is determined by the performance of the hardware, for example, by the CPU, GPU performance or the number of bits of the training hardware.

[0068] S102, reading a text sequence as a training sample.

[0069] In this embodiment, the text sequence is a token sequence generated by processing the input text data through a tokenizer.

[0070] Usually, the sequence length L of a text sequence is greater than N.

[0071] S103, dynamically preprocessing the text sequence according to the target sequence length M and the maximum length N to obtain data blocks; wherein the length of the data block matches the maximum length N, and the text sequence is padded when the sequence length of the text sequence is less than the length threshold to maintain the consistency of the position encoding.

[0072] S104: training the large language model according to the data blocks, so as to perform natural language processing based on the trained large language model.

[0073] In this embodiment, during training, before the text sequence is input into the model, the text sequence is dynamically preprocessed according to its sequence length, and the preprocessing follows a specific splitting or padding strategy.

[0074] Specifically, for a text sequence with a sequence length of L, if L is much smaller than the target sequence length M (for example, L / M is less than 1 / 8, that is, the length threshold can be set to M / 8, and the specific threshold can be set according to actual needs), it is first padded to at least half of the target sequence length M in order to expand the associated position information.

[0075] Take uniform filling as an example. Figure 2 As shown, the filled data (PAD in the figure) is evenly distributed between two adjacent data (T0, T1, T2, T3) of the text sequence, that is, the values ​​of the three filled data between T0 and T1 can be T0+(T1-T0) / 4, T0+(T1-T0) / 2 and T0+(T1-T0)*3 / 4 respectively. Of course, there can be other filling methods, such as random filling, that is, the values ​​of the filled data PAD are randomly distributed, and the present invention does not make specific limitations.

[0076] In this embodiment, the filled data may also be a special token, such as 0.

[0077] Then, the padded sequence generated after padding is split according to the maximum length N supported by the hardware so that training can be performed under limited resources.

[0078] If L is not much smaller than the target sequence length M, the sequence is split directly according to the maximum length N supported by the hardware.

[0079] like Figure 3 As shown, a text sequence with a sequence length of 13 can be split into 4 data blocks. The first three data blocks use the data of the original sequence (i.e., tokens), and the fourth data block is padded to the length N supported by the training hardware based on the data of the original sequence, such as by right padding.

[0080] In this embodiment, the split data blocks are input into the model for training. It should be noted here that the position marks of the data blocks in the model training do not need to be reset.

[0081] In summary, compared with the prior art, this embodiment has the following advantages:

[0082] 1. By dynamically generating position codes, limited video memory resources are effectively utilized, and the length of the sequence processed by the model is expanded;

[0083] 2. The data preprocessing method enables the model to process text blocks of appropriate size in each training step and adapt to inputs of different lengths while maintaining the ability to capture long-distance dependencies.

[0084] In summary, this embodiment ensures that effective long text training can be achieved under limited hardware resources, making large-scale language model training more economical and efficient.

[0085] The second embodiment of the present invention provides a training device for a large language model (such as Figure 4 ), which includes:

[0086] An initialization unit 210 is used to initialize the large language model to be trained according to a target sequence length M and a maximum length N supported by hardware of a current device for training, wherein M>N and M and N are both positive integers;

[0087] A reading unit 220, used for reading a text sequence as a training sample;

[0088] A preprocessing unit 230 is used to dynamically preprocess the text sequence according to the target sequence length M and the maximum length N to obtain data blocks; wherein the length of the data block matches the maximum length N, and the text sequence is padded when the sequence length of the text sequence is less than a length threshold;

[0089] The training unit 240 is used to train the large language model according to the data blocks, so as to perform natural language processing based on the trained large language model.

[0090] A third embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0091] at least one processor; and,

[0092] a memory communicatively connected to the at least one processor; wherein,

[0093] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the large language model training method of any of the aforementioned embodiments.

[0094] The fourth embodiment of the present invention further provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the large language model training method described in any of the aforementioned embodiments.

[0095] The fifth embodiment of the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the large language model training method of any of the aforementioned embodiments.

[0096] The sixth embodiment of the present invention further provides a computer program, which includes program instructions. When the program instructions are executed by a computer, the computer executes the large language model training method of any of the aforementioned embodiments.

[0097] Figure 5 A schematic diagram of a method or device 1000 that can implement an embodiment of the present invention is shown, which may include more or fewer devices than shown in the figure in some embodiments. In some embodiments, it can be implemented using a single or multiple devices. In some embodiments, it can be implemented using cloud or distributed devices.

[0098] like Figure 5 As shown, the device 1000 includes a processor 1001, which can perform various appropriate operations and processes according to the program and / or data stored in the read-only memory (ROM) 1002 or the program and / or data loaded from the storage part 1008 to the random access memory (RAM) 1003. Processor 1001 can be a multi-core processor or can include multiple processors. In some embodiments, processor 1001 can include a general main processor and one or more special coprocessors, such as a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. In RAM 1003, various programs and data required for the operation of device 1000 are also stored. Processor 1001, ROM 1002 and RAM 1003 are connected to each other via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0099] The processor and the memory are used together to execute the program stored in the memory. When the program is executed by the computer, the methods, steps or functions described in the above embodiments can be implemented.

[0100] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, a touch screen, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed. Figure 5 Only some components are schematically shown in the figure, which does not mean that the device 1000 only includes Figure 5 Components shown.

[0101] The systems, devices, modules or units described in the above embodiments may be implemented by a computer or its associated components. The computer may be, for example, a mobile terminal, a smart phone, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a personal digital assistant, a media player, a navigation device, a game console, a tablet computer, a wearable device, a smart TV, an Internet of Things system, a smart home, an industrial computer, a server or a combination thereof.

[0102] Although not shown, in an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program / instruction is stored, and when the computer program / instruction is executed by a processor, the text adversarial sample generation method described in Example 1 is implemented.

[0103] Storage media in embodiments of the present invention include permanent and non-permanent, removable and non-removable items that can be used to store information by any method or technology. Examples of storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0104] Although not shown, an embodiment of the present invention further provides a computer program product, including: a computer program / instruction, which, when executed by a processor, implements the method for generating text adversarial samples described in Example 1.

[0105] The methods, programs, systems, devices, etc. of the embodiments of the present invention may be executed or implemented in a single or multiple networked computers, or may be practiced in a distributed computing environment. In the embodiments of this specification, in these distributed computing environments, tasks may be performed by remote processing devices connected via a communication network.

[0106] Those skilled in the art should understand that the embodiments of the present specification may be provided as methods, systems or computer program products. Therefore, those skilled in the art may imagine that the functional modules / units or controllers and related method steps described in the above embodiments may be implemented in software, hardware or a combination of software / hardware.

[0107] Unless explicitly stated, the actions or steps of the methods, programs, and embodiments of the present invention do not have to be performed in a specific order and can still achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.

[0108] In this article, multiple embodiments of the present invention are described, but for the sake of brevity, the description of each embodiment is not exhaustive, and the same or similar features or parts between the embodiments may be omitted. In this article, "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" are meant to be applicable to at least one embodiment or example according to the present invention, but not all embodiments. The above terms do not necessarily mean to refer to the same embodiment or example. In the absence of contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples.

[0109] The exemplary systems and methods of the present invention have been specifically shown and described with reference to the above embodiments, which are merely examples of the best modes for implementing the present systems and methods. It will be appreciated by those skilled in the art that various changes may be made to the embodiments of the systems and methods described herein when implementing the present systems and / or methods without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A method for training a large language model, characterized in that: include: Initialize the large language model to be trained according to the target sequence length M and the maximum length N supported by the hardware of the current device used for training, where M>N and M and N are both positive integers; Read the text sequence as training sample; Dynamically preprocessing the text sequence according to the target sequence length M and the maximum length N to obtain data blocks; wherein the length of the data block matches the maximum length N, and the text sequence is padded when the sequence length of the text sequence is less than a length threshold; The large language model is trained according to the data blocks to perform natural language processing based on the trained large language model, wherein When the large language model to be trained is initialized, the positional coding is generated based on the rotational position coding RoPE method, and the cardinality and factor parameters of the positional coding generation function are adjusted according to the target sequence length M and the maximum length N supported by the hardware of the current device to set the positional coding of the large language model.

2. The method for training a large language model according to claim 1, characterized in that: The formulas for the base and factor parameters are as follows: factor=M / (N*4) base = 10000.0 * factor * 32 Where base represents the cardinality and factor represents the factor. The expression of the position encoding generation function is: If i is even If i is an odd number Where pos is the position index, i is the dimension index, and d is the dimension of the large language model.

3. The method for training a large language model according to any one of claims 1 to 2, characterized in that: The text sequence is dynamically preprocessed according to the target sequence length M and the maximum length N to obtain data blocks, including: Determine whether the ratio of the sequence length L of the text sequence to the target sequence length M is greater than a preset threshold; If yes, split the text sequence into multiple data blocks of length N; If not, pad the length of the text sequence to half of the target sequence length M to obtain a padded sequence, and then split the padded sequence into multiple data blocks of length N; Among them, the position marks between data blocks are not reset during training.

4. The method for training a large language model according to claim 3, characterized in that: The text sequence is a word unit sequence generated by processing the input text data through a word segmenter; The threshold is 1 / 8.

5. The method for training a large language model according to claim 3, characterized in that: If the length of the last data block is less than N, the data block is padded so that its length reaches N.

6. The method for training a large language model according to claim 3, characterized in that: In the padding sequence, the padded data are evenly distributed between two adjacent data in the text sequence or randomly distributed.

7. A large language model training device, characterized in that: include: An initialization unit, used to initialize the large language model to be trained according to a target sequence length M and a maximum length N supported by hardware of a current device for training, where M>N and both M and N are positive integers; A reading unit, used for reading a text sequence as a training sample; A preprocessing unit, used for dynamically preprocessing the text sequence according to the target sequence length M and the maximum length N to obtain data blocks; wherein the length of the data block matches the maximum length N, and the text sequence is padded when the sequence length of the text sequence is less than a length threshold; A training unit is used to train the large language model according to the data blocks, so as to perform natural language processing based on the trained large language model, wherein The initialization unit is also configured to generate position coding based on the rotational position coding RoPE method when initializing the large language model to be trained, and adjust the cardinality and factor parameters of the position coding generation function according to the target sequence length M and the maximum length N supported by the hardware of the current device to set the position coding of the large language model.

8. An electronic device, characterized in that: The electronic device includes: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the large language model training method described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, which are used to enable a computer to execute the large language model training method described in any one of claims 1 to 6.

Images