Transformer neural network model training method and transformer neural network model

Abstract

The present application provides a Transformer neural network model training method and a Transformer neural network model. The method of the present application comprises: training a Transformer neural network model on the basis of a first normalization module and a second normalization module to obtain a target Transformer neural network model, wherein the first normalization module has an output proportion of 100% at the start of training, and the second normalization module has an output proportion of 0% at the start of training; and during training, gradually decreasing the output proportion of the first normalization module, and gradually increasing the output proportion of the second normalization module, wherein the first normalization module has an output proportion of 0% upon completion of the training, and the second normalization module has an output proportion of 100% upon completion of the training. The method provided by the present application contributes to reducing the inference latency of Transformer neural network models.

Description

Transformer neural network model training method and transformer neural network model

[0001] The present application claims priority from the Chinese patent application No. 202410417273.X filed on April 8, 2024, and entitled "Transformer neural network model training method and transformer neural network model", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence, and in particular to a transformer neural network model training method and a transformer neural network model. BACKGROUND

[0003] The normalization method is a technology widely used in deep learning neural networks. According to whether the mean and variance are calculated in real time during inference, the normalization method can be divided into online and offline methods. Among them, the inference through the online method needs to calculate the mean and variance in real time, which has additional computational overhead. The normalization method using the online method for inference includes the layer normalization (LN) method and the like. The inference through the offline method can usually be integrated with the adjacent linear layer, and there is no additional computational overhead during inference.

[0004] The transformer neural network is a deep learning-based neural network, which performs excellently in the field of natural language processing (NLP) and can be usually applied to various visual tasks and natural language processing tasks such as classification, segmentation, and detection. However, in the related art, the transformer neural network model usually adopts the normalization method such as the LN method as the normalization layer, which has additional computational overhead, resulting in a long inference time delay of the transformer neural network model. SUMMARY

[0005] The present application provides a transformer neural network model training method and a transformer neural network model, which helps to reduce the inference time delay of the transformer neural network model.

[0006] In a first aspect, the present application provides a Transformer neural network model training method applied to a Transformer neural network model, the Transformer neural network model comprising a first normalization module and a second normalization module at the beginning of training, the first normalization module adopting online normalization, and the second normalization module adopting offline normalization, the method comprising: training the Transformer neural network model based on the first normalization module and the second normalization module to obtain a target Transformer neural network model; wherein the first normalization module outputs at a proportion of 100% at the beginning of training, the second normalization module outputs at a proportion of 0% at the beginning of training, during the training process, the output proportion of the first normalization module is gradually decreased, and the output proportion of the second normalization module is gradually increased, the first normalization module outputs at a proportion of 0% after the training ends, and the second normalization module outputs at a proportion of 100% after the training ends.

[0007] The Transformer neural network model of the present application adds offline normalization for training on the basis of online normalization, and during the training process, the offline normalization module plays an increasingly important role, and the online normalization module plays an increasingly weak role, so that the trained Transformer neural network model can use offline normalization method, thereby reducing the latency of the Transformer neural network model during inference.

[0008] In one possible implementation, the gradually decreasing the output proportion of the first normalization module and the gradually increasing the output proportion of the second normalization module comprises: gradually decreasing the output proportion of the first normalization module based on a step length, and gradually increasing the output proportion of the second normalization module based on the step length.

[0009] In one possible implementation, the step length is a linear step length, or the step length is a nonlinear step length.

[0010] In one possible implementation, if the step length is a linear step length, the step length is determined by the number of training times.

[0011] In one possible implementation, the second normalization module comprises a first branch and a second branch, the output of the second normalization module comprises the output of the first branch and the output of the second branch, the output of the first branch is obtained by normalizing the input in an offline manner, and the output of the second branch is obtained by learning the input based on a preset learning parameter.

[0012] In a possible implementation, the Transformer neural network model further includes an attention module and a multi-layer perception (MLP) module, and the method further includes: fusing the second normalization module in the target Transformer neural network model with a linear layer adjacent to the second normalization module in the attention module, and fusing the second normalization module in the target Transformer neural network model with a linear layer adjacent to the second normalization module in the MLP module.

[0013] In a possible implementation, the attention module is a multi-head self-attention (MHSA) module.

[0014] In a possible implementation, the attention module adopts linear attention, adopts a Relu function as a similarity function, further includes a depthwise separable convolution (DWC) module, and the output of the attention module further includes an output of the DWC module.

[0015] In a second aspect, the present application provides a Transformer neural network model training apparatus, including one or more function modules, which are configured to implement the Transformer neural network model training method in the first aspect.

[0016] In a third aspect, the present application provides an electronic device, including a processor and a memory, the memory being configured to store a program, and the processor being configured to run the program to implement the Transformer neural network model training method in the first aspect.

[0017] In a fourth aspect, the present application provides a readable storage medium, which stores a program, and when the program is run on an electronic device, the electronic device implements the Transformer neural network model training method in the first aspect.

[0018] In a fifth aspect, the present application provides a program, and when the program is run on a processor of an electronic device, the electronic device implements the Transformer neural network model training method in the first aspect.

[0019] In a possible design, the program in the fifth aspect can be stored, in whole or in part, on a storage medium packaged together with the processor, or can be stored, in part or in whole, on a storage medium not packaged together with the processor. BRIEF DESCRIPTION OF DRAWINGS

[0020] FIG. 1 is a structural schematic diagram of a Transformer neural network model provided by the prior art;

[0021] FIG. 2 is a training architecture diagram of a Transformer neural network model provided by an embodiment of the present application;

[0022] FIG. 3 is a structural schematic diagram of a second normalization module provided by an embodiment of the present application;

[0023] FIG. 4 is a flow schematic diagram of an embodiment of a Transformer neural network model training method provided by the present application;

[0024] FIG. 5 is a structural schematic diagram of an embodiment of a Transformer neural network model provided by the present application;

[0025] FIG. 6 is a structural schematic diagram of another embodiment of a Transformer neural network model provided by the present application;

[0026] FIG. 7 is a structural schematic diagram of a Transformer neural network model training device provided by an embodiment of the present application;

[0027] FIG. 8 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0028] In the embodiments of the present application, unless otherwise specified, the character " / " represents a relationship of one or the other between the associated objects before and after. For example, A / B can represent A or B. The "and / or" describes the relationship between the associated objects, which means that there can be three relationships. For example, A and / or B, which means that there are three cases of A alone, A and B together, and B alone.

[0029] It should be noted that the "first", "second", and the like in the embodiments of the present application are only used for distinguishing purposes of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features, nor can it be understood as indicating or implying the order.

[0030] In the embodiments of the present application, "at least one" means one or more, and "multiple" means two or more. In addition, "at least one of the following" or the like means any combination of these items, which can include any combination of single item or multiple items. For example, at least one of A, B or C, which means A, B, C, A and B, A and C, B and C, or A, B and C. Wherein, each of A, B, C can be an element or a set containing one or more elements.

[0031] In the embodiments of the present application, “exemplary”, “in some embodiments”, “in another embodiment” and the like are used to indicate an example, illustration or description. Any embodiment or design scheme described as “exemplary” in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the use of the word “exemplary” is intended to present the concept in a specific manner.

[0032] “of”, “corresponding” and “corresponding” in the embodiments of the present application can be mixed sometimes. It should be pointed out that the meanings expressed are consistent when the distinction is not emphasized. In the embodiments of the present application, communication and transmission can be mixed sometimes. It should be pointed out that the meanings expressed are consistent when the distinction is not emphasized. For example, transmission can include sending and / or receiving, and can be a noun or a verb.

[0033] The equal to involved in the embodiments of the present application can be used with greater than, which is applicable to the technical solutions adopted when greater than. It can also be used with less than, which is applicable to the technical solutions adopted when less than. It should be noted that when equal to is used with greater than, it cannot be used with less than. When equal to is used with less than, it cannot be used with greater than.

[0034] The normalization method is a technology widely used in deep learning neural networks. According to whether the mean and variance and other statistical values need to be calculated in real time during inference, the normalization method can be divided into online and offline two ways. Among them, the inference through the online way needs to calculate the mean and variance in real time, which has additional computational overhead. The normalization method using the online way for inference includes layer normalization (Layer Normalization, LN) and other ways; the inference through the offline way can usually be integrated with the adjacent linear layer, and there is no additional computational overhead during inference.

[0035] The Transformer neural network is a deep learning-based neural network, which performs excellently in the field of natural language processing (Natural Language Processing, NLP), and can usually be applied to classification, segmentation, detection and other visual tasks and natural language processing tasks. However, in related technologies, the Transformer neural network model usually adopts online normalization such as LN as the normalization layer, which will have additional computational overhead, resulting in a long delay of the Transformer neural network model during inference.

[0036] FIG. 1 is a structure diagram of a Transformer neural network model provided by the prior art.

[0037] Referring to FIG. 1, the Transformer neural network model in the prior art comprises a normalization module, an MHSA module and an MLP module. The normalization module adopts an online manner, and can be used for processing before the MHSA module processes and before the MLP module processes.

[0038] As can be seen from FIG. 1, the online normalization is used for reasoning before the MHSA module and the MLP module process, which can increase the reasoning time delay.

[0039] Based on the above problems, the embodiment of the present application proposes a Transformer neural network model training method, which is applied to the Transformer neural network model and helps to reduce the reasoning time delay of the Transformer neural network model.

[0040] FIG. 2 is a training architecture diagram of the Transformer neural network model provided by the embodiment of the present application.

[0041] Referring to FIG. 2, the training architecture of the Transformer neural network model can comprise a first normalization module, a second normalization module, an attention module and an MLP module. The first normalization module adopts online normalization, for example, the first normalization module can be an LN module, or the first normalization module can be a normalization module adopting other online manners, which is not specially limited in the embodiment of the present application; the second normalization module adopts offline normalization, for example, the second normalization module can be a BN module, or the second normalization module can be a normalization module adopting other offline manners, which is not specially limited in the embodiment of the present application.

[0042] The first normalization module and the second normalization module are two parallel branches, the output of the first normalization module and the output of the second normalization module are both used as the input of the MHSA module, and the output of the first normalization module and the output of the second normalization module are both used as the input of the MLP module. The first normalization module comprises a weight coefficient γ, and the second normalization module comprises a weight coefficient 1-γ; by adjusting the weight coefficient γ, the output of the first normalization module and the output of the second normalization module can be adjusted.

[0043] In some optional embodiments, the second normalization module can comprise a first branch and a second branch, and the output of the second normalization module comprises the output of the first branch and the output of the second branch.

[0044] In some optional embodiments, the attention module can not adopt linear attention, for example, the attention module can be an MHSA module.

[0045] In some optional embodiments, the attention module can employ linear attention, for example, the attention module can employ linear attention with ReLU as the similarity function.

[0046] FIG. 3 is a structural diagram of a second normalization module according to an embodiment of the present application.

[0047] Referring to FIG. 3, the output of the first branch in the second normalization module is the output obtained after the input is normalized in an offline manner, and the output of the second branch in the second normalization module is the output obtained after the input is learned based on a preset learning parameter η.

[0048] For example, taking the second normalization module as a BN module, the output of the second normalization module can be represented by the following expression: BN'(x) = BN(x) + ηx;

[0049] where x is the input.

[0050] It can be understood that, through learning based on the preset learning parameter η, the stability and accuracy of the model can be improved.

[0051] Next, the Transformer neural network model training method provided by an embodiment of the present application will be exemplarily described in combination with FIG. 4.

[0052] As shown in FIG. 4, the flowchart of one embodiment of the Transformer neural network model training method provided by the present application includes the following steps:

[0053] At step 401, at the beginning of training, the first normalization module outputs at a proportion of 100%, and the second normalization module outputs at a proportion of 0%.

[0054] Specifically, at the beginning of training, only the first normalization module can be used, for example, the weight coefficient γ of the first normalization module can be set to 1, so that the first normalization module outputs at a proportion of 100%; at the same time, the second normalization module can not be used, for example, when the weight coefficient γ of the first normalization module is 0, the weight coefficient 1-γ of the second normalization module is 0, so that the second normalization module outputs at a proportion of 0%.

[0055] At step 402, during training, the output proportion of the first normalization module is gradually decreased, and the output proportion of the second normalization module is gradually increased.

[0056] Specifically, after the training starts, that is, during the training, the output proportion of the first normalization module can be gradually decreased, for example, the value of the weight coefficient γ of the first normalization module can be gradually decreased, so that the first normalization module plays a weaker and weaker role in training.

[0057] Correspondingly, the output proportion of the second normalization module can be gradually increased, for example, as the value of the weight coefficient γ of the first normalization module is gradually decreased, the value of the weight coefficient 1-γ of the second normalization module is gradually increased, so that the second normalization module plays an increasingly important role in training.

[0058] The manner of gradually decreasing the output proportion of the first normalization module can include gradually decreasing the output proportion of the first normalization module based on a step size.

[0059] The manner of gradually increasing the output proportion of the second normalization module can include gradually increasing the output proportion of the second normalization module based on a step size.

[0060] For example, assuming that the step size is s, the weight coefficient γ of the first normalization module is 1 in the first training, and the output proportion of the first normalization module is 100%; in the second training, the weight coefficient γ of the first normalization module is reduced to 1-s, and the output proportion of the first normalization module is (1-s)×100%; in the third training, the weight coefficient γ of the first normalization module is reduced to 1-2×s, and the output proportion of the first normalization module is (1-2×s)×100%; and so on, so that the output proportion of the first normalization module can be gradually decreased based on the step size s.

[0061] Correspondingly, in the first training, the weight coefficient 1-γ of the second normalization module is 0, and the output proportion of the second normalization module is 0%; in the second training, the weight coefficient 1-γ of the second normalization module is increased to s, and the output proportion of the second normalization module is s×100%; in the third training, the weight coefficient 1-γ of the second normalization module is increased to 2×s, and the output proportion of the second normalization module is (2×s)×100%; and so on, so that the output proportion of the second normalization module can be gradually increased based on the step size s.

[0062] In some optional embodiments, the step size s can be linear, in which case the step size s can be determined by the number of training times. For example, assuming that the number of training times is N, since the value of the weight coefficient γ of the first normalization module is expected to gradually decrease from the initial value 1 to the target value 0, the step size s can be determined by the quotient of the span of the weight coefficient and the number of training times, wherein the span of the weight coefficient is the difference between the initial value and the target value of the weight coefficient γ, i.e., the span of the weight coefficient is 1-0=1. For example, the step size s can be calculated by the following calculation formula:

[0063] s=1 / N.

[0064] In some optional embodiments, the step size s can be greater than 1 / N, that is, when the number of training times does not reach N, the value of the weight coefficient γ of the first normalization module has been reduced to 0; in subsequent training, the first normalization module can not play a role, only the second normalization module plays a role, until the number of training times reaches N, and the training ends.

[0065] In some optional embodiments, the step size s can also be nonlinear. For example, the weight coefficient γ can be reduced in a sinusoidal or cosine manner, or the weight coefficient γ can also be reduced in other nonlinear manners, and the embodiments of the present application do not make special limitations on this.

[0066] After the training ends, the output proportion of the first normalization module is 0%, and the output proportion of the second normalization module is 100%.

[0067] Specifically, after the training ends, the weight coefficient γ of the first normalization module is reduced to 0, so that the output proportion of the first normalization module is 0%.

[0068] Correspondingly, the weight coefficient 1-γ of the second normalization module is increased to 1, so that the output proportion of the second normalization module is 100%.

[0069] It can be understood that when the output proportion of the first normalization module is 0%, it means that the first normalization module does not play a role in the training of the Transformer neural network model, that is, the trained Transformer neural network model only contains the second normalization module, so that the second normalization module can adopt offline normalization, thereby reducing the inference delay of the model.

[0070] FIG. 5 is a structural diagram of an embodiment of the Transformer neural network model provided by the present application.

[0071] Referring to FIG. 5, in the Transformer neural network model in the present application, only the second normalization module is included, that is, offline normalization is performed before the attention module and the MLP module are processed, thereby reducing the inference delay.

[0072] In some optional embodiments, in order to further reduce the inference delay, the second normalization module can also be fused into the attention module and the MLP module, respectively.

[0073] The manner in which the second normalization module is fused into the attention module can include fusing the second normalization module with the linear layer adjacent to the second normalization module in the attention module.

[0074] The manner in which the second normalization module is fused into the MLP module can include fusing the second normalization module with a linear layer in the MLP module adjacent to the second normalization module.

[0075] Through the above fusion, the structure of the Transformer neural network model as shown in FIG. 6 can be obtained.

[0076] Referring to FIG. 6, after the second normalization module is fused with the attention module and the MLP module respectively, the structure of the Transformer neural network model no longer includes a normalization layer, thereby reducing the processing of the normalization layer, and thus reducing the inference latency.

[0077] In some optional embodiments, the attention module can not employ linear attention, for example, the attention module can be an MHSA module.

[0078] In some optional embodiments, in order to reduce the computational complexity, the attention module can employ linear attention, for example, the attention module can employ linear attention with ReLU as the similarity function. And on the basis of the attention module, a depth wise convolution (DWC) module can be added to enhance the local feature extraction capability through the linear attention mechanism.

[0079] wherein the output O of the linear attention can be obtained through the following calculation formula:

[0080] wherein N is the length of the input sequence, Q is the index, K is the key, V is the value, and Oi is the i-th output.

[0081] It can be understood that the output O of the linear attention can be obtained by splicing the N outputs O1 to ON.

[0082] When the DWC module is added, the output O' of the entire attention module can be obtained through the following calculation formula:

[0083] O' = O + DWC(V);

[0084] wherein DWC(V) is the output of the DWC module.

[0085] FIG. 7 is a structural schematic diagram of an embodiment of a Transformer neural network model training apparatus provided by the present application. As shown in FIG. 7, the Transformer neural network model training apparatus 70 is applied to a Transformer neural network model, the Transformer neural network model comprises a first normalization module and a second normalization module at the beginning of training, the first normalization module adopts online normalization, and the second normalization module adopts offline normalization. The Transformer neural network model training apparatus 70 can comprise: a training module 71; wherein

[0086] The training module 71 is configured to train the Transformer neural network model based on the first normalization module and the second normalization module to obtain a target Transformer neural network model.

[0087] In the above embodiment, the first normalization module outputs at a proportion of 100% at the beginning of training, the second normalization module outputs at a proportion of 0% at the beginning of training, the output proportion of the first normalization module is gradually decreased and the output proportion of the second normalization module is gradually increased during training, the first normalization module outputs at a proportion of 0% after training, and the second normalization module outputs at a proportion of 100% after training.

[0088] In one possible implementation, the training module 71 is further configured to gradually decrease the output proportion of the first normalization module based on a step length and gradually increase the output proportion of the second normalization module based on the step length.

[0089] In one possible implementation, the step length is a linear step length, or the step length is a nonlinear step length.

[0090] In one possible implementation, if the step length is a linear step length, the step length is determined by the number of training times.

[0091] In one possible implementation, the second normalization module comprises a first branch and a second branch, the output of the second normalization module comprises the output of the first branch and the output of the second branch, the output of the first branch is obtained by normalizing an input in an offline manner, and the output of the second branch is obtained by learning the input based on a preset learning parameter.

[0092] In one possible implementation, the Transformer neural network model further comprises an attention module and a multi-layer perception (MLP) module, and the Transformer neural network model training apparatus 70 further comprises:

[0093] fusing the second normalization module in the target Transformer neural network model with a linear layer in the attention module adjacent to the second normalization module, and fusing the second normalization module in the target Transformer neural network model with a linear layer in the MLP module adjacent to the second normalization module.

[0094] In one possible implementation, the attention module is a multi-head attention (MHSA) module.

[0095] In one possible implementation, the attention module adopts linear attention, adopts a Relu function as a similarity function, further includes a depthwise separable convolution (DWC) module, and the output of the attention module further includes the output of the DWC module.

[0096] The Transformer neural network model training apparatus 70 shown in FIG. 7 can be used to execute the technical solutions of the method embodiments shown in the present application, and the implementation principles and technical effects thereof can be further referred to the related descriptions in the method embodiments.

[0097] It should be understood that the division of each module of the Transformer neural network model training apparatus 70 shown in FIG. 7 is only a logical functional division, and all or part of them can be integrated into one physical entity, or can be physically separated. Moreover, all of these modules can be implemented in the form of being called by a processing element through software; all of them can be implemented in the form of hardware; some of them can be implemented in the form of being called by a processing element through software, and some of them can be implemented in the form of hardware. For example, the detection module can be a separately set processing element, or can be implemented in a certain chip of an electronic device. The implementation of other modules is similar. Moreover, all or part of these modules can be integrated together, or can be independently implemented. In the implementation process, each step of the above method or each module can be completed by the integrated logic circuit of hardware or the instruction in the form of software in the processing element.

[0098] For example, the above modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), or one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs), etc. For another example, the modules can be integrated together to implement a System-On-a-Chip (SOC) form.

[0099] FIG. 8 shows a structural schematic diagram of an electronic device 800.

[0100] The electronic device 800 can include at least one processor and at least one memory communicatively connected with the processor, wherein the memory stores program instructions executable by the processor, and the processor invoking the program instructions can execute the method provided by the embodiments shown herein.

[0101] FIG. 8 shows a block diagram of an exemplary electronic device 800 suitable for implementing the embodiments herein. The electronic device 800 shown in FIG. 8 is merely an example and should not impose any limitation on the functions and usage scope of the embodiments herein.

[0102] As shown in FIG. 8, the components of the electronic device 800 can include, but are not limited to, one or more processors 810, a memory 820, a communication bus 840 connecting different system components (including the memory 820 and the processor 810), and a communication interface 830.

[0103] The communication bus 840 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration bus, a processor or local bus using any of a variety of bus architectures. For example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0104] Electronic device 800 typically includes a variety of computer system readable media. These media can be any available media that is accessible by the computer system, including both volatile and non-volatile media, removable and non-removable media.

[0105] Memory 820 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The device can further include other removable / non-removable, volatile / non-volatile computer system storage media. Although not shown, a disk drive can provide memory for a removable disk and the disk drive can include machine readable media for the removable disk. Similarly, an optical disk drive can provide memory for a removable optical disk. Disk drives and disk drives can be connected to bus 840 by one or more disk drive interfaces. Storage media 820 can include a number of program products for use with the device, such as an operating system, one or more application programs, other program modules, and program data. Each of these examples and / or some combination thereof can include an implementation of a networking environment.

[0106] Program / utility, having a set (at least one) of program modules, can be stored in memory 820, such as an operating system, one or more application programs, other program modules, and program data, and each or some combination thereof, can include an implementation of a networking environment. Generally, program modules are executed by processing unit 810 in memory 820 to implement processes and / or methods of embodiments described herein.

[0107] The electronic device 800 can also communicate with one or more external devices such as a keyboard or a pointing device, displays, etc.; other devices that enable a user to interact with the device; and / or any devices (e.g., network card, modem, etc.) that enable the device to communicate with one or more other devices. Such communication can occur via the communication interface 830. Also, the electronic device 800 can communicate with one or more networks such as a local area network (LAN), a wide area network (WAN), and / or the Internet through a network adapter (not shown in FIG. 8). It will be appreciated that the network connections shown in FIG. 8 are illustrative and other means of establishing a communications link between the electronic device 800 and a network can be used. It will be further appreciated that the electronic device 800 can be any device that is configured to function as a server, a client, or both, including, without limitation, a desktop computer, a laptop computer, a handheld computer, a netbook computer, a smart phone, a gaming device, a music player, or any other device that is configured to function as a server, a client, or both.

[0108] The processor 810 performs various function applications and data processing by running programs stored in the memory 820, such as implementing the method provided in the embodiments.

[0109] It can be understood that the interface connection relationship between the modules shown in the embodiments is only illustrative and does not constitute a structural limitation on the electronic device 800. In other embodiments, the electronic device 800 can also use different interface connection modes or a combination of multiple interface connection modes.

[0110] In the above embodiments, the processor can include, for example, a CPU, a DSP, a microcontroller, or a digital signal processor, and can also include a GPU, an embedded neural network processing unit (NPU), and an image signal processor (ISP). The processor can also include necessary hardware accelerators or logic processing hardware circuits, such as an ASIC, or one or more integrated circuits for controlling the execution of programs of the technical solutions of the present application. In addition, the processor can have the function of operating one or more software programs, and the software programs can be stored in a storage medium.

[0111] The embodiments of the present application also provide a readable storage medium, which stores programs, and when the programs are run on an electronic device, the electronic device performs the method provided in the embodiments of the present application.

[0112] An embodiment of the present application also provides a program product, which includes a program. When the program product is run on an electronic device, the electronic device executes the method provided by the embodiment shown in the present application.

[0113] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can represent: a, b, c, a and b, a and c, b and c or a and b and c, where a, b, c can be single or multiple.

[0114] Those skilled in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented using a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0115] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0116] In the several embodiments provided in this application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of this application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of this application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory; hereinafter referred to as: ROM), random access memory (Random Access Memory; hereinafter referred to as: RAM), magnetic disk or optical disk, and other media that can store program codes.

[0117] The above description is only the specific implementation of the present application. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered in the protection range of the present application. The protection range of the present application should be subject to the protection range of the claims.

Claims

1. A Transformer neural network model training method, characterized in that: Applied to a Transformer neural network model, the Transformer neural network model includes a first normalization module and a second normalization module at the beginning of training, the first normalization module adopts online normalization, and the second normalization module adopts offline normalization, the method includes: Training the Transformer neural network model based on the first normalization module and the second normalization module to obtain a target Transformer neural network model; Among them, the first normalization module outputs at a ratio of 100% at the beginning of training, and the second normalization module outputs at a ratio of 0% at the beginning of training. During the training process, the output ratio of the first normalization module is gradually reduced, and the output ratio of the second normalization module is gradually increased. After the training is completed, the first normalization module outputs at a ratio of 0%, and the second normalization module outputs at a ratio of 100% after the training is completed.

2. The method according to claim 1, characterized in that The gradually decreasing the output ratio of the first normalization module and gradually increasing the output ratio of the second normalization module includes: The output ratio of the first normalization module is gradually decreased based on the step size, and the output ratio of the second normalization module is gradually increased based on the step size.

3. The method according to claim 2, characterized in that The step size is a linear step size, or the step size is a nonlinear step size.

4. The method according to claim 3, characterized in that If the step size is a linear step size, the step size is determined by the number of training times.

5. The method according to any one of claims 1 to 4, characterized in that The second normalization module includes a first branch and a second branch. The output of the second normalization module includes the output of the first branch and the output of the second branch. The output of the first branch is the output obtained after the input is normalized in an offline manner. The output of the second branch is the output obtained after learning the input based on preset learning parameters.

6. The method according to any one of claims 1 to 5, characterized in that The Transformer neural network model further includes an attention module and a multi-layer perceptron (MLP) module. The method further includes: The second normalization module in the target Transformer neural network model is fused with the linear layer adjacent to the second normalization module in the attention module, and the second normalization module in the target Transformer neural network model is fused with the linear layer adjacent to the second normalization module in the MLP module.

7. The method according to claim 6, characterized in that The attention module is a multi-head attention MHSA module.

8. The method according to claim 6, characterized in that The attention module adopts linear attention, the attention module adopts ReLU function as the similarity function, the attention module also includes a depth-wise separable convolution DWC module, and the output of the attention module also includes the output of the DWC module.

9. A Transformer neural network model, characterized in that The Transformer neural network model includes a second normalization module, wherein, The Transformer neural network model is obtained by training based on the first normalization module and the second normalization module. The Transformer neural network model includes the first normalization module and the second normalization module at the beginning of training. The first normalization module adopts online normalization, and the second normalization module adopts offline normalization. The first normalization module outputs at a ratio of 100% at the beginning of training, and the second normalization module outputs at a ratio of 0% at the beginning of training. During the training process, the output ratio of the first normalization module is gradually reduced, and the output ratio of the second normalization module is gradually increased. After the training is completed, the first normalization module outputs at a ratio of 0%, and the second normalization module outputs at a ratio of 100% after the training is completed.

Images