Model expansion method, device, equipment, storage medium and computer program product
By monitoring and dynamically adding trained feedforward modules, the problem of limited adaptability during the pre-training of large language models is solved, thereby improving the stability and adaptability of the model, and enhancing output accuracy and user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING XIAOMI MOBILE SOFTWARE CO LTD
- Filing Date
- 2024-12-26
- Publication Date
- 2026-06-26
AI Technical Summary
In the pre-training process of large language models in the existing technology, the addition of a feedforward module with fixed parameters leads to limited model adaptability and discontinuous training process, affecting the stability and adaptability of the model.
During model training, the data compression capacity of the feedforward module is monitored, and new feedforward modules determined based on the already trained feedforward modules are dynamically added. Training continues using the second training data until the target number is reached, ensuring the continuity and stability of the model parameters.
It improves the model's utilization of training data, enhances the model's adaptability and output accuracy, reduces the waste of training resources, and improves the model's adaptability and user experience in different application scenarios.
Smart Images
Figure CN122287709A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer technology, and in particular to a model expansion method, apparatus, device, storage medium, and computer program product. Background Technology
[0002] Large Language Models (LLMs) have achieved remarkable results in tasks such as translation, text processing, and sentiment analysis.
[0003] In related technologies, when pre-training a large language model, training begins with a small initial model, and feedforward modules are gradually added to the model to continue training, thereby expanding the model's scale. Furthermore, the added feedforward modules are determined based on a fixed number of parameters from the original, untrained model. Pre-training a large language model in this way leads to limited adaptability of the pre-trained model. Summary of the Invention
[0004] To overcome the problems existing in related technologies, this disclosure provides a model expansion method, apparatus, device, storage medium, and computer program product.
[0005] According to a first aspect of the present disclosure, a model expansion method is provided, the model including m decoding modules, the m decoding modules including n feedforward modules, where m and n are positive integers, the method comprising: during model training of the model using first training data, monitoring the data compression capacity of each of the k feedforward modules, wherein the data compression capacity characterizes the number of model parameters carried by the model training, k ≤ n, and k is a positive integer; in response to detecting the presence of a first feedforward module among the k feedforward modules, adding a second feedforward module to the decoding module containing the first feedforward module to expand the feedforward modules included in the decoding module, wherein the first feedforward module is a feedforward module whose data compression capacity is greater than or equal to a preset capacity threshold, and the second feedforward module is determined based on the first feedforward module; based on the expanded feedforward modules, continuing to train the model using second training data, and monitoring the data compression capacity of each of the k feedforward modules; repeating the above process until the feedforward modules in the decoding module are expanded to a target number of feedforward modules.
[0006] In one implementation, the second feedforward module is determined based on the first feedforward module in the following manner: a new feedforward module with model parameters equal to the model parameters of the first feedforward module is created to obtain the second feedforward module.
[0007] In one embodiment, the method further includes: during model training of the model, after each preset number of model parameter adjustments, saving the trained model parameters, including the model parameters of the feedforward module; the model parameters of the first feedforward module are determined as follows: determining the model parameters most recently saved before the existence of the first feedforward module is detected; and determining the model parameters of the first feedforward module from the most recently saved model parameters.
[0008] In one embodiment, the model parameters of the feedforward module are determined based on the parameters of the x-layer linear layer, the parameters of the linear layer are based on matrix representation, and the linear layer is used to perform data compression; the data compression capacity is the sum of the ranks of the matrices corresponding to the parameters of each linear layer in the x-layer, where x is a positive integer.
[0009] In one implementation, the k feedforward modules are selected from the feedforward modules included in the decoding module based on a gating mechanism.
[0010] In one implementation, the k feedforward modules for the i-th monitoring are determined from h feedforward modules based on a first gating mechanism, and the k feedforward modules for the (i+1)-th monitoring are determined from j feedforward modules based on a second gating mechanism; wherein, h is less than j, and the number of gating parameters corresponding to the first gating mechanism is less than the number of gating parameters corresponding to the second gating mechanism.
[0011] In one implementation, the complexity of the first training data is lower than that of the second training data.
[0012] According to a second aspect of the present disclosure, a model expansion apparatus is provided, the model including m decoding modules, the m decoding modules including n feedforward modules, where m and n are positive integers, the model expansion apparatus comprising: a monitoring unit, configured to monitor the data compression capacity of each of the k feedforward modules during model training using first training data, wherein the data compression capacity characterizes the size of the number of model parameters carried by the model training, k ≤ n, and k is a positive integer; and a processing unit, configured to respond to the detection of storage in the k feedforward modules. In the first feedforward module, a second feedforward module is added to the decoding module containing the first feedforward module to expand the feedforward modules included in the decoding module. The first feedforward module is a feedforward module whose data compression capacity is greater than or equal to a preset capacity threshold, and the second feedforward module is determined based on the first feedforward module. Based on the expanded feedforward modules, the model is trained again using the second training data, and the data compression capacity of each of the k feedforward modules is monitored. The above process is repeated until the feedforward modules in the decoding module are expanded to the target number of feedforward modules.
[0013] In one implementation, the processing unit is configured to determine a second feedforward module based on a first feedforward module by creating a new feedforward module whose model parameters are the model parameters of the first feedforward module, thereby obtaining the second feedforward module.
[0014] In one embodiment, during the model training process, after each preset number of model parameter adjustments, the processing unit is further configured to save the model parameters after model training, including the model parameters of the feedforward module; determine the model parameters most recently saved before the presence of the first feedforward module was detected; and determine the model parameters of the first feedforward module from the most recently saved model parameters.
[0015] In one implementation, the model parameters of the feedforward module are determined based on the parameters of the x-layer linear layer, the parameters of the linear layer are based on matrix representation, and the linear layer is used to perform data compression; the processing unit is used to determine the data compression capacity by determining the sum of the ranks of the matrices corresponding to the parameters of each linear layer in the x-layer as the data compression capacity, where x is a positive integer.
[0016] In one implementation, the monitoring unit is used to select k feedforward modules from the feedforward modules included in the decoding module based on a gating mechanism.
[0017] In one embodiment, the monitoring unit is configured to select k feedforward modules from the feedforward modules included in the decoding module based on a gating mechanism in the following manner: determining the k feedforward modules for the i-th monitoring from h feedforward modules based on a first gating mechanism, and determining the k feedforward modules for the (i+1)-th monitoring from j feedforward modules based on a second gating mechanism; wherein, h is less than j, and the number of gating parameters corresponding to the first gating mechanism is less than the number of gating parameters corresponding to the second gating mechanism.
[0018] In one implementation, the complexity of the first training data is lower than that of the second training data.
[0019] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the model extension method described in the first aspect or any embodiment of the first aspect.
[0020] According to a fourth aspect of the present disclosure, a storage medium is provided that stores instructions which, when executed by a processor of an electronic device, enable the electronic device to perform the model extension method described in the first aspect or any embodiment of the first aspect.
[0021] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the model extension method described in the first aspect or any embodiment of the first aspect.
[0022] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: During model training using the first training data, the data compression capacity of each of the k feedforward modules is monitored. If a first feedforward module is detected to have a data compression capacity greater than or equal to a preset capacity threshold, a second feedforward module is added to the decoding module containing the first feedforward module, thereby expanding the model during training. Furthermore, since the second feedforward module is determined based on the already trained first feedforward module, there is no need to retrain the feedforward modules. This not only improves the model's utilization of training data and saves training resources, but also enhances the continuity of the model's pre-training process, thereby improving the model's generalization ability to new input data. This allows the model to adapt to diverse data, improves its adaptability to different application scenarios, and ultimately increases the accuracy of the model's output data, thus enhancing the user experience.
[0023] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description
[0024] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.
[0025] Figure 1 This is a schematic diagram illustrating the principle of a model extension method in a related art according to an exemplary embodiment.
[0026] Figure 2 This is a flowchart illustrating a model extension method according to an exemplary embodiment.
[0027] Figure 3 This is a schematic diagram illustrating the principle of a model expansion method according to an exemplary embodiment.
[0028] Figure 4 This is a flowchart illustrating a method for determining a second feedforward module according to an exemplary embodiment.
[0029] Figure 5 This is a flowchart illustrating a method for determining model parameters of a first feedforward module according to an exemplary embodiment.
[0030] Figure 6 This is a flowchart illustrating a method for determining data compression capacity according to an exemplary embodiment.
[0031] Figure 7 This is a flowchart illustrating a model selection feedforward module for model training according to an exemplary embodiment.
[0032] Figure 8 This is a flowchart illustrating, according to an exemplary embodiment, the selection of k feedforward modules to process training data through a gating mechanism at different stages of training.
[0033] Figure 9 This is a flowchart illustrating model training using different training data according to an exemplary embodiment.
[0034] Figure 10 This is a block diagram illustrating a model expansion device according to an exemplary embodiment.
[0035] Figure 11 This is a block diagram illustrating an apparatus for model expansion according to an exemplary embodiment.
[0036] Figure 12 This is a block diagram illustrating an apparatus for model expansion according to an exemplary embodiment. Detailed Implementation
[0037] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure.
[0038] The model augmentation method provided in this disclosure is applied to model augmentation scenarios during model pre-training. The model involved in this disclosure can be a large language model.
[0039] The model disclosed herein is a self-attention mechanism model (Transformer), which typically includes N decoding modules (Transformer Decoder Blocks). These decoding modules include an attention module and a feedforward module. The feedforward module is also known as an expert module.
[0040] In related technologies, when pre-training a model, training typically begins with an initial model containing fewer feedforward modules than the target number. During training, initial feedforward modules are added to the initial model to bring the total number of feedforward modules in the model up to the target number. The number of feedforward modules directly impacts the model's data compression capacity. Data compression capacity, also known as capacity or effective capacity, can be understood as the ability of a model, or its decoding or feedforward modules, to effectively process data in its assigned task. Figure 1 This is a schematic diagram illustrating the principle of a model extension method in a related art according to an exemplary embodiment. For example... Figure 1 As shown, pre-training begins with a first initial feedforward module in the decoding module, and during pre-training, a second initial feedforward module is added to the decoding module to expand the number of feedforward modules in the decoding module. However, although this method can increase the number of feedforward modules in the model, since each added feedforward module initializes its model parameters with fixed values, that is, the model parameters of each added second initial feedforward module are the same, the newly added feedforward modules need to be retrained. This may cause the model parameters to be sensitive to small changes in the input data, resulting in insufficient stability.
[0041] In view of this, this disclosure provides a model expansion method. During model training, when a feedforward module (hereinafter referred to as the first feedforward module) with a data compression capacity greater than or equal to a capacity threshold is detected, a new feedforward module (hereinafter referred to as the second feedforward module) is added to the decoding module to expand the feedforward modules included in the decoding module. Furthermore, the second feedforward module is determined based on the first feedforward module. This disclosure expands the model by adding a second feedforward module determined based on the first feedforward module during model training, eliminating the need to retrain the feedforward modules. This not only improves the model's utilization of training data but also enhances the continuity of the model pre-training process. By improving the continuity of the model pre-training process, the model can continuously adapt to new data distributions and the changes in model parameters can be smoother, reducing the possibility of the model getting trapped in local optima or diverging. Smooth changes in model parameters enable the model to learn features in the data more effectively, promoting better generalization. This model expansion method improves the model's adaptability to different application scenarios and enhances the model's output quality, thereby improving the user experience.
[0042] It is understood that the model involved in the embodiments of this disclosure can be used to perform a variety of functions, such as content generation, natural language understanding, multi-turn dialogue, content extraction, knowledge updating, translation, and logical reasoning.
[0043] In this embodiment, the model includes m decoding modules, and each of the m decoding modules includes n feedforward modules, where m and n are positive integers. The n feedforward modules represent the number of feedforward modules included in each of the m decoding modules. Each of the m decoding modules can be a decoding module without feedforward modules, or it can include one or more feedforward modules.
[0044] Figure 2 This is a flowchart illustrating a model extension method according to an exemplary embodiment. For example... Figure 2 As shown, the model expansion method includes the following steps.
[0045] In step S11, during the model training process using the first training data, the data compression capacity of each of the k feedforward modules is monitored.
[0046] Among them, data compression capacity represents the amount of model parameters carried by model training, k≤n, where k is a positive integer.
[0047] In this embodiment, after the first training data is input into the model, the model calls k of the n feedforward modules to compress the first training data. The first training data may be, for example, unlabeled corpus. The model training process refers to the process of adjusting the model parameters to increase the data compression capacity of the feedforward modules as they continuously compress the data.
[0048] In step S12, in response to the detection of the presence of a first feedforward module among the k feedforward modules, a second feedforward module is added to the decoding module where the first feedforward module is located, so as to expand the feedforward modules included in the decoding module.
[0049] The first feedforward module is a feedforward module whose data compression capacity is greater than or equal to a preset capacity threshold.
[0050] In this embodiment, the preset capacity threshold can be a fixed value set according to the structure of the feedforward module itself and the number of model parameters. If the data compression capacity of the feedforward module is greater than or equal to the preset capacity threshold, even if the model continues to call the feedforward module to process training data and adjust model parameters, the data compression capacity of the first feedforward module will not continue to increase. Therefore, a new feedforward module needs to be added to continue to improve the data compression capacity of the model.
[0051] In this embodiment, the second feedforward module is determined based on the first feedforward module. Determining the second feedforward module based on the first feedforward module enables the transfer of training results without requiring retraining of the second feedforward module. Compared to adding an initial feedforward module with fixed model parameters each time, this improves the continuity of model pre-training. Furthermore, the improved continuity of the pre-training process allows the model to better understand and capture data features, performing well on unprocessed data, thereby improving the model's adaptability to different application scenarios and increasing the accuracy of the model's output data.
[0052] In step S13, based on the expanded feedforward module, the model is trained again using the second training data, and the data compression capacity of each of the k feedforward modules is monitored.
[0053] Figure 3 This is a schematic diagram illustrating the principle of a model extension method according to an exemplary embodiment. For example... Figure 3 As shown, this example illustrates a model with one decoding module, which includes one feedforward module, undergoing initial model pre-training. In this case, the data compression capacity of the decoding module increases with the input of the first training data. The data compression capacity of the feedforward module is continuously monitored. When the data compression capacity of the feedforward module is detected to be greater than or equal to a preset capacity threshold, this feedforward module is designated as the first feedforward module. A second feedforward module is added to the decoding module containing the first feedforward module to expand the feedforward modules included in the decoding module. The expanded decoding module performs data compression through the first feedforward module and / or the second feedforward module, and continuously adjusts the model parameters of the feedforward module. The expanded model is trained using second training data, which can be the same as the first training data or different data.
[0054] In step S14, the number of feedforward modules in the decoding module is monitored until the number of feedforward modules in the decoding module is expanded to the target number. If the number of feedforward modules in the decoding module does not reach the target number, steps S11 to S13 are repeated.
[0055] In the model expansion method provided in this disclosure, each added second feedforward module is determined based on the first feedforward module whose data compression capacity is greater than or equal to a preset capacity threshold, thus dynamically setting the model parameters of each added feedforward module. Compared to expanding the model by fixing the model parameters of the added feedforward modules, this method improves the continuity of the model training process. Furthermore, since there is no need to retrain the feedforward modules, it improves the utilization rate of training data and saves training resources. By improving the continuity of the model training process, the changes in model parameters become smoother, reducing the possibility of the model getting trapped in local optima or diverging, and improving the model's stability. Due to the higher stability, the model can continuously retain previously learned knowledge, reducing the forgetting of old knowledge when facing new tasks, and learning stable and effective feature representations. Enabling the model to learn stable and effective feature representations improves its generalization ability to new input data, enhances its adaptability to different application scenarios, and improves the accuracy of the model's output data, thereby improving the user experience.
[0056] In this embodiment of the disclosure, in order to improve the continuity of model pre-training, the model parameters of the added second feedforward module are set according to the model parameters of the already pre-trained feedforward module.
[0057] In one exemplary embodiment, Figure 4 This is a flowchart illustrating a method for determining a second feedforward module according to an exemplary embodiment. Figure 4 As shown, the second feedforward module is determined using the following method:
[0058] In step S21, the data compression capacity of each of the k feedforward modules is monitored.
[0059] In step S22, in response to the detection of the presence of a first feedforward module among the k feedforward modules.
[0060] In step S23, a new feedforward module with model parameters of the first feedforward module is created to obtain the second feedforward module.
[0061] In this embodiment of the disclosure, a new feedforward module is created with model parameters that are the model parameters of the first feedforward module, and a second feedforward module is obtained. This can be understood as copying the model parameters of the first feedforward module to obtain the second feedforward module.
[0062] As training progresses, the data compression capacity of the feedforward module increases. When the data compression capacity of the feedforward module is greater than or equal to a preset capacity threshold, this feedforward module is designated as the first feedforward module. The model parameters of the first feedforward module are copied to a newly created feedforward module, which becomes the second feedforward module. In this way, the second feedforward module inherits the training effect of the first feedforward module, eliminating the need to retrain the second feedforward module.
[0063] Through the above embodiments, the method of dynamically setting the model parameters of the second feedforward module based on the model parameters of the first feedforward module ensures that each added second feedforward module inherits the training effect of the first feedforward module. Compared with the method of expanding the model by adding feedforward modules with fixed model parameters, this ensures that the model can evolve smoothly during training, rather than changing abruptly, resulting in higher continuity in the model training process and higher stability in the obtained model.
[0064] In this embodiment of the disclosure, during the model training process, the model parameters after model training can be saved so that when determining the second feedforward module, the model parameters of the first feedforward module can be determined based on the saved model parameters, thereby quickly determining the second feedforward module and improving the training efficiency of the model.
[0065] The following embodiments further illustrate the method for determining the model parameters of the first feedforward module in this disclosure.
[0066] Figure 5 This is a flowchart illustrating a method for determining model parameters of a first feedforward module according to an exemplary embodiment. Figure 5 As shown, the model parameters of the first feedforward module are determined through the following steps:
[0067] In step S31, during the model training process using the first training data, the data compression capacity of each of the k feedforward modules is monitored.
[0068] In step S32, after each preset number of model parameter adjustments, the model parameters after training are saved.
[0069] The model parameters include those of the feedforward module.
[0070] In step S33, in response to the detection of the presence of a first feedforward module among the k feedforward modules, the model parameters most recently saved before the presence of the first feedforward module was detected are determined.
[0071] In step S34, the model parameters of the first feedforward module are determined from the most recently saved model parameters.
[0072] In this embodiment, model training using the first training data requires continuous adjustment of model parameters to increase the model's data compression capacity. A pre-set number of model parameter adjustments is set, and the current model parameters are saved each time the model undergoes this number of adjustments. When a first feedforward module is detected, the model parameters of the most recently saved first feedforward module are found from the most recently saved model parameters, and the model parameters of the second feedforward module are determined based on these first feedforward module parameters.
[0073] In this embodiment, the number of adjustments can be set to a fixed value as needed, or dynamically set according to the frequency of the first feedforward module. This embodiment does not impose any specific limitations.
[0074] Through the above embodiments, by saving the model parameters after each model parameter adjustment, the efficiency of model pre-training is improved while ensuring the continuity of the model pre-training process and avoiding the slowdown in data processing caused by repeatedly determining the first model parameters.
[0075] The following embodiments further illustrate the method for determining data compression capacity in this disclosure.
[0076] Figure 6 This is a flowchart illustrating a method for determining data compression capacity according to an exemplary embodiment. The model parameters of the feedforward module involved in this disclosure embodiment are determined based on the parameters of an x-layer linear layer, where the parameters are represented by a matrix. The linear layer is used to perform data compression. For example... Figure 6 As shown, the data compression capacity is determined in the following way:
[0077] In step S41, during the model training process using the first training data, the data compression capacity of each of the k feedforward modules is monitored.
[0078] In step S42, the sum of the ranks of the matrices corresponding to the parameters of each linear layer in layer x is used to determine the data compression capacity. Here, x is a positive integer.
[0079] In step S43, in response to detecting that the data compression capacity is greater than the capacity threshold, a second feedforward module is added to the decoding module to expand the feedforward module included in the decoding module.
[0080] In this embodiment of the disclosure, each linear layer in the feedforward module is essentially a matrix A with p rows and q columns. p×qEach row of the matrix corresponds to an output neuron, and each column corresponds to an input neuron; therefore, p < q. Taking a feedforward module consisting of two linear layers as an example, the rank of the matrix corresponding to each linear layer is calculated, such as r(A1) and r(A2) for the two linear layers. The data compression amount C of this feedforward module is C = r(A1) + (A2). When the data compression amount C is greater than or equal to the capacity threshold, it indicates that the data compression capacity of this feedforward module has reached its limit, and the feedforward module included in the decoding module needs to be expanded. Furthermore, the newly added second feedforward module is determined based on the first feedforward module whose data compression capacity has reached the set threshold.
[0081] Through the above embodiments, by using the rank of the matrix corresponding to each linear layer to represent the data compression capacity, it is possible to monitor in real time whether the data compression capacity of the corresponding feedforward module has reached the set threshold, and then add the second feedforward module in a timely manner. This avoids the problems of redundant model parameter learning and wasted training data caused by the untimely addition of the feedforward module, and improves the efficiency of model pre-training.
[0082] Understandably, the model includes multiple decoding modules, each of which contains multiple feedforward modules. Each feedforward module is used to handle a specific task. Therefore, during pre-training, it is necessary to manage the training data input to each feedforward module to reduce the redundancy in model parameter learning.
[0083] In this embodiment of the disclosure, during model pre-training, to reduce learning redundancy among feedforward modules, whenever training data is input, the model selects to call some feedforward modules for training, rather than continuously activating all feedforward modules. For example, if the model has n feedforward modules, k feedforward modules can be called to expand the model, and the outputs of the k feedforward networks are aggregated into the output of the decoding module.
[0084] In this embodiment, k feedforward modules are invoked based on a gating mechanism.
[0085] The following embodiments further illustrate the process of model training by calling the feedforward module in this disclosure.
[0086] In this embodiment of the disclosure, Figure 7 This is a flowchart illustrating a model selection feedforward module used for model training, according to an exemplary embodiment. Figure 7 As shown, the model trains by calling the feedforward module through the following steps:
[0087] In step S51, the model is trained using the first training data.
[0088] In step S52, k feedforward modules are selected from the feedforward modules included in the decoding module based on the gating mechanism.
[0089] In step S53, the data compression capacity of each of the k feedforward modules is monitored.
[0090] Through the above embodiments, it is ensured that when the input training data can be processed, some feedforward modules are called for data compression. Compared with the method of calling all feedforward modules at the same time to learn the training data, the phenomenon of redundant model learning is reduced, enabling the model to learn more knowledge, thereby improving the performance of the obtained model, reducing resource usage and accelerating model training.
[0091] Understandably, during pre-training, the number of feedforward modules in the model gradually increases. If the gating mechanism is not adjusted, newly added feedforward modules may not be trained, and the model's data compression capacity may not continue to improve. The following embodiments further illustrate the method of selecting k feedforward modules through a gating mechanism at different training stages:
[0092] In this embodiment of the disclosure, Figure 8 This is a flowchart illustrating, according to an exemplary embodiment, the selection of k feedforward modules to process training data through a gating mechanism at different stages of training. For example... Figure 8 As shown, k feedforward modules are selected through a gating mechanism at different stages of training using the following method:
[0093] In step S61, k feedforward modules are determined from h feedforward modules based on the first gating mechanism.
[0094] Among them, h feedforward modules are the feedforward modules included in the decoding module during the i-th monitoring.
[0095] In step S62, in response to the detection of the presence of a first feedforward module among the k feedforward modules, a second feedforward module is added to the decoding module where the first feedforward module is located, so as to expand the feedforward modules included in the decoding module.
[0096] During the i-th monitoring process, after the first feedforward module is detected, a second feedforward module is added to the decoding module where the first feedforward module is located. At this time, the number of feedforward modules included in the decoding module increases from h to j, and the (i+1)-th monitoring is performed.
[0097] In step S63, k feedforward modules are determined from j feedforward modules based on the second gating mechanism.
[0098] Among them, j feedforward modules are the feedforward modules included in the decoding module during the (i+1)th monitoring.
[0099] Where h is less than j, the number of gating parameters corresponding to the first gating mechanism is less than the number of gating parameters corresponding to the second gating mechanism.
[0100] In this embodiment, the model typically does not call all feedforward modules for data compression at all times. Instead, it calls feedforward modules based on the content of the training data and a gating mechanism. After adding a feedforward module to the decoding module, gating parameters are added to call the newly added feedforward module for training data processing, ensuring that the newly added feedforward module can participate in training in a timely manner, which helps reduce redundant learning. Therefore, when adding a second feedforward module to the decoding module, gating parameters corresponding to the second feedforward module also need to be added to the gating mechanism. The model selects k feedforward modules to be activated through the gating mechanism with added gating parameters, and uses the outputs of the selected k feedforward modules as the output of the decoding module. The gating mechanism can be implemented through one or more linear layers, and the added gating parameters can be determined through random initialization. For example, in the i-th monitoring, there are h feedforward modules in the model, so the first gating mechanism determines the k feedforward modules from the h feedforward modules. When performing the (i+1)th or (i+u)th monitoring, there are j feedforward modules in the model. At this time, the k feedforward modules are determined from the j feedforward modules through a second gating mechanism with added gating parameters. h, j, k, and i are all positive integers.
[0101] Through the above embodiments, each time a feedforward module is added to the decoding module, a new gating parameter is added to the gating mechanism so that the newly added feedforward module can be called to process training data, reducing the phenomenon of redundant model learning, enabling the model to carry more knowledge, and improving the performance of the obtained model.
[0102] In the embodiments of this disclosure, as the number of feedforward modules included in the decoding module of the model increases, the complexity of the data that the model can process gradually increases. If training data of the same complexity is used for model training throughout the entire pre-training stage, the model's computing power may be insufficient to effectively process the complex data input into the model, leading to a waste of training data. Therefore, increasing the complexity of the training data according to the progress of training can improve the utilization rate of training resources. The following embodiments of this disclosure further illustrate the method of training the model with different training data.
[0103] In this embodiment of the disclosure, Figure 9 This is a flowchart illustrating model training using different training data, according to an exemplary embodiment. For example... Figure 9 As shown, the model is trained using different training data through the following steps:
[0104] In step S71, during the model training process using the first training data, the data compression capacity of each of the k feedforward modules is monitored.
[0105] In step S72, in response to the detection of the presence of a first feedforward module among the k feedforward modules, a second feedforward module is added to the decoding module where the first feedforward module is located, so as to expand the feedforward modules included in the decoding module.
[0106] In step S73, based on the expanded feedforward module, the model is trained again using second training data with a complexity higher than that of the first training data, and the data compression capacity of each of the k feedforward modules is monitored.
[0107] In this embodiment, when the decoding module has only one feedforward module, the model is trained using first training data with lower complexity. After adding a second feedforward module, the model is trained using second training data. The second training data has higher complexity than the first training data. In this embodiment, it can be configured that after adding a predetermined number of second feedforward modules, second training data with higher complexity than the first training data is selected for model training.
[0108] Through the above embodiments, by increasing the complexity of training data as the model's data compression capacity increases, compared to pre-training methods that always use training data of the same complexity, the risk of the model failing to effectively handle complex input data, leading to training failure or getting stuck in local optima, is avoided. This approach enables the model to establish basic feature recognition capabilities and gradually learn more advanced abstract features, thereby improving the model's performance on unprocessed data. Furthermore, in the initial training phase, using simple first training data and gradually introducing more complex second training data as the model includes more feedforward modules can achieve better training results with limited computing resources, reducing resource consumption. In addition, gradually introducing complex training data after the model achieves good performance on simple data can also accelerate the convergence speed of the entire training process.
[0109] Based on the same concept, this disclosure also provides a model expansion device 100.
[0110] It is understood that the model expansion device 100 provided in this disclosure includes hardware structures and / or software modules corresponding to each function in order to achieve the above-mentioned functions. In conjunction with the units and algorithm steps of the various examples disclosed in this disclosure, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of this disclosure.
[0111] Figure 10 This is a block diagram of a model expansion device 100 according to an exemplary embodiment. The model includes m decoding modules, and the m decoding modules include n feedforward modules, where m and n are positive integers. (Refer to...) Figure 10 The model expansion device includes a monitoring unit 101 and a processing unit 102.
[0112] The monitoring unit 101 is used to monitor the data compression capacity of each of the k feedforward modules during the model training process using the first training data. The data compression capacity represents the size of the model parameters carried by the model training, k≤n, and k is a positive integer.
[0113] The processing unit 102, in response to detecting the presence of a first feedforward module among the k feedforward modules, adds a second feedforward module to the decoding module containing the first feedforward module to expand the feedforward modules included in the decoding module. The first feedforward module is a feedforward module whose data compression capacity is greater than or equal to a preset capacity threshold, and the second feedforward module is determined based on the first feedforward module. Based on the expanded feedforward modules, the model continues to be trained using the second training data, and the data compression capacity of each of the k feedforward modules is monitored. The above process is repeated until the feedforward modules in the decoding module are expanded to the target number of feedforward modules.
[0114] In one embodiment, the processing unit 102 is configured to determine the second feedforward module based on the first feedforward module by creating a new feedforward module whose model parameters are the model parameters of the first feedforward module, thereby obtaining the second feedforward module.
[0115] In one embodiment, during the model training process, after each preset number of model parameter adjustments, the processing unit 102 is further configured to save the model parameters after model training, including the model parameters of the feedforward module; determine the model parameters most recently saved before the presence of the first feedforward module was detected; and determine the model parameters of the first feedforward module from the most recently saved model parameters.
[0116] In one embodiment, the model parameters of the feedforward module are determined based on the parameters of the x-layer linear layers. The parameters of the linear layers are represented by matrices, and the linear layers are used to perform data compression. The processing unit 102 is used to determine the data compression capacity by determining the sum of the ranks of the matrices corresponding to the parameters of each linear layer in the x-layer as the data compression capacity. Where x is a positive integer.
[0117] In one embodiment, the monitoring unit 101 is used to select k feedforward modules from the feedforward modules included in the decoding module based on a gating mechanism.
[0118] In one embodiment, the monitoring unit 101 is used to select k feedforward modules from the feedforward modules included in the decoding module based on a gating mechanism in the following manner: the k feedforward modules for the i-th monitoring are determined from h feedforward modules based on a first gating mechanism, and the k feedforward modules for the (i+1)-th monitoring are determined from j feedforward modules based on a second gating mechanism; wherein, h is less than j, and the number of gating parameters corresponding to the first gating mechanism is less than the number of gating parameters corresponding to the second gating mechanism.
[0119] In one embodiment, the complexity of the first training data is lower than that of the second training data.
[0120] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.
[0121] Figure 11 This is a block diagram illustrating a device 200 for model expansion according to an exemplary embodiment. For example, device 200 may be an electronic device such as a terminal. Examples include mobile phones, computers, digital broadcasting terminals, messaging devices, game consoles, tablet devices, medical devices, fitness equipment, personal digital assistants, etc.
[0122] Reference Figure 11 The device 200 may include one or more of the following components: processing component 202, memory 204, power component 206, multimedia component 208, audio component 210, input / output (I / O) interface 212, sensor component 214, and communication component 216.
[0123] Processing component 202 typically controls the overall operation of device 200, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 202 may include one or more modules to facilitate interaction between processing component 202 and other components. For example, processing component 202 may include a multimedia module to facilitate interaction between multimedia component 208 and processing component 202.
[0124] Memory 204 is configured to store various types of data to support the operation of device 200. Examples of such data include instructions for any application or method operating on device 200, contact data, phonebook data, messages, pictures, videos, etc. Memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.
[0125] The power supply component 206 provides power to the various components of the device 200. The power supply component 206 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 200.
[0126] Multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 208 includes a front-facing camera and / or a rear-facing camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.
[0127] Audio component 210 is configured to output and / or input audio signals. For example, audio component 210 includes a microphone (MIC) configured to receive external audio signals when device 200 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 204 or transmitted via communication component 216. In some embodiments, audio component 210 also includes a speaker for outputting audio signals.
[0128] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.
[0129] Sensor assembly 214 includes one or more sensors for providing status assessments of various aspects of device 200. For example, sensor assembly 214 may detect the on / off state of device 200, the relative positioning of components such as the display and keypad of device 200, changes in the position of device 200 or a component of device 200, the presence or absence of user contact with device 200, the orientation or acceleration / deceleration of device 200, and temperature changes of device 200. Sensor assembly 214 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 214 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 214 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.
[0130] Communication component 216 is configured to facilitate wired or wireless communication between device 200 and other devices. Device 200 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 216 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 216 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
[0131] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.
[0132] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, which can be executed by a processor 220 of the device 200 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.
[0133] Figure 12 This is a block diagram illustrating an apparatus 300 for model expansion according to an exemplary embodiment. For example, apparatus 300 may be provided as a server. (Refer to...) Figure 12 The device 300 includes a processing component 322, which further includes one or more processors, and memory resources represented by memory 332 for storing instructions, such as application programs, that can be executed by the processing component 322. The application programs stored in memory 332 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 322 is configured to execute instructions to perform the model extension method described above.
[0134] Device 300 may also include a power supply component 326 configured to perform power management of device 300, a wired or wireless network interface 350 configured to connect device 300 to a network, and an input / output (I / O) interface 358. Device 300 may operate on an operating system stored in memory 332, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.
[0135] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described model extension method.
[0136] It is understood that in this disclosure, "multiple" refers to two or more, and other quantifiers are similar. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. The singular forms "a," "the," and "the" are also intended to include the plural forms unless the context clearly indicates otherwise.
[0137] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.
[0138] It is further understood that although operations are described in a specific order in the accompanying drawings in the embodiments of this disclosure, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all of the shown operations to be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.
[0139] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.
[0140] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.
Claims
1. A model extension method, characterized in that, The model includes m decoding modules, of which n are feedforward modules, where m and n are positive integers. The method includes: During the model training process using the first training data, the data compression capacity of each of the k feedforward modules is monitored, wherein the data compression capacity represents the size of the model parameters carried by the model training, k≤n, and k is a positive integer; In response to the detection of the presence of a first feedforward module among the k feedforward modules, a second feedforward module is added to the decoding module where the first feedforward module is located to expand the feedforward modules included in the decoding module. The first feedforward module is a feedforward module whose data compression capacity is greater than or equal to a preset capacity threshold, and the second feedforward module is determined based on the first feedforward module. Based on the expanded feedforward module, the model is further trained using the second training data, and the data compression capacity of each of the k feedforward modules is monitored. Repeat the above process until the feedforward modules in the decoding module are expanded to the target number of feedforward modules.
2. The model expansion method according to claim 1, characterized in that, The second feedforward module is determined based on the first feedforward module in the following manner: A new feedforward module with the same model parameters as the first feedforward module is created, resulting in the second feedforward module.
3. The model expansion method according to claim 2, characterized in that, The method further includes: During the model training process, after each preset number of model parameter adjustments, the trained model parameters are saved, including the model parameters of the feedforward module. The model parameters of the first feedforward module are determined in the following manner: Determine the model parameters most recently saved before the presence of the first feedforward module was detected; The model parameters of the first feedforward module are determined from the most recently saved model parameters.
4. The model extension method according to any one of claims 1 to 3, characterized in that, The model parameters of the feedforward module are determined based on the parameters of the x-layer linear layer, the parameters of which are based on matrix representation, and the linear layer is used to perform data compression. The data compression capacity is the sum of the ranks of the matrices corresponding to the parameters of each linear layer in layer x, where x is a positive integer.
5. The model expansion method according to claim 1, characterized in that, The k feedforward modules are selected from the feedforward modules included in the decoding module based on a gating mechanism.
6. The model expansion method according to claim 5, characterized in that, The k feedforward modules for the i-th monitoring are determined from the h feedforward modules based on the first gating mechanism, and the k feedforward modules for the (i+1)-th monitoring are determined from the j feedforward modules based on the second gating mechanism. Where h is less than j, and the number of gate parameters corresponding to the first gate mechanism is less than the number of gate parameters corresponding to the second gate mechanism.
7. The model expansion method according to claim 1, characterized in that, The complexity of the first training data is lower than that of the second training data.
8. A model expansion device, characterized in that, The model includes m decoding modules, of which n are feedforward modules, where m and n are positive integers. The model expansion device includes: The monitoring unit is used to monitor the data compression capacity of each of the k feedforward modules during the model training process using the first training data, wherein the data compression capacity represents the size of the number of model parameters carried by the model training, k≤n, and k is a positive integer; The processing unit, in response to detecting the presence of a first feedforward module among the k feedforward modules, adds a second feedforward module to the decoding module containing the first feedforward module to expand the feedforward modules included in the decoding module. The first feedforward module is one whose data compression capacity is greater than or equal to a preset capacity threshold, and the second feedforward module is determined based on the first feedforward module. Based on the expanded feedforward modules, the model continues to be trained using the second training data, and the data compression capacity of each of the k feedforward modules is monitored. The above process is repeated until the feedforward modules in the decoding module are expanded to the target number of feedforward modules.
9. The model expansion device according to claim 8, characterized in that, The processing unit is used to determine the second feedforward module based on the first feedforward module in the following manner: A new feedforward module with the same model parameters as the first feedforward module is created, resulting in the second feedforward module.
10. The model expansion device according to claim 9, characterized in that, During the model training process, after each preset number of model parameter adjustments, the processing unit is also used to save the model parameters after training, including the model parameters of the feedforward module. Determine the model parameters most recently saved before the presence of the first feedforward module was detected; The model parameters of the first feedforward module are determined from the most recently saved model parameters.
11. The model expansion device according to any one of claims 8 to 10, characterized in that, The model parameters of the feedforward module are determined based on the parameters of the x-layer linear layer, the parameters of which are based on matrix representation, and the linear layer is used to perform data compression. The processing unit is used to determine the data compression capacity in the following manner: The sum of the ranks of the matrices corresponding to the parameters of each linear layer in layer x is determined as the data compression capacity, where x is a positive integer.
12. The model expansion device according to claim 8, characterized in that, The monitoring unit is used to select k feedforward modules from the feedforward modules included in the decoding module based on a gating mechanism.
13. The model expansion device according to claim 12, characterized in that, The monitoring unit is used to select k feedforward modules from the feedforward modules included in the decoding module based on a gating mechanism in the following manner: Based on the first gating mechanism, k feedforward modules for the i-th monitoring are determined from h feedforward modules, and based on the second gating mechanism, k feedforward modules for the (i+1)-th monitoring are determined from j feedforward modules. Where h is less than j, and the number of gate parameters corresponding to the first gate mechanism is less than the number of gate parameters corresponding to the second gate mechanism.
14. The model expansion device according to claim 8, characterized in that, The complexity of the first training data is lower than that of the second training data.
15. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured as follows: The model expansion method according to any one of claims 1 to 7.
16. A storage medium, characterized in that, The storage medium stores instructions that, when executed by an electronic device processor, enable the electronic device to perform the model extension method according to any one of claims 1 to 7.
17. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the model extension method according to any one of claims 1 to 7.