Methods, devices, storage media and electronic devices for recognizing text information

Abstract

This application provides a method for recognizing text information. The method includes: detecting the training process of a language model for a target text recognition scenario, wherein the target text recognition scenario is a text information recognition scenario belonging to a target domain; upon detecting that the training process is complete, acquiring a first language model obtained during the training process and a first optimizer for the first language model; extending the first network layer of the first language model to obtain a second language model, and updating the first optimizer to obtain a second optimizer for the second language model; and training the second language model using the second optimizer and text samples belonging to the target domain to obtain a target language model. This application addresses the problem of low training efficiency for large language models in related technologies, and can improve the training efficiency of large language models.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically, to a method, apparatus, storage medium, and electronic device for recognizing text information. Background Technology

[0002] With the increasing demand for large-scale language models, their training process has received growing attention. Generally, large language models have a large number of network layers and parameters. Currently, the common method for training large language models is direct training: constructing an initial model according to the required number of layers, then training the initial model with a large number of training samples to obtain the final large-scale language model. However, due to the large number of layers and parameters required for large-scale language models, this training method results in significant training pressure and low efficiency. Summary of the Invention

[0003] This application provides a method, apparatus, storage medium, and electronic device for recognizing text information, in order to at least solve the problem of low training efficiency of large language models in related technologies.

[0004] According to one embodiment of this application, a method for recognizing text information is provided, including:

[0005] The training process of training a language model for a target text recognition scenario is detected, wherein the target text recognition scenario is a text information recognition scenario belonging to the target domain;

[0006] If the training process is detected to be complete, the first language model obtained by the training process and the first optimizer of the first language model are obtained, wherein the first optimizer is used to record the iterative process of the network parameters of all network layers included in the first language model when the first language model is obtained.

[0007] The first network layer of the first language model is extended to obtain a second language model, and the first optimizer is updated to obtain a second optimizer for the second language model. The second language model includes all network layers and the second network layer included in the first language model. The network parameters of the second network layer are the same as those of the first network layer. The second optimizer is used to record the iteration process of the network parameters of all network layers included in the second language model and the iteration process of the network parameters of the first network layer when the first language model is obtained corresponding to the second network layer.

[0008] The second language model is trained using the second optimizer and text samples belonging to the target domain to obtain a target language model, wherein the target language model is used to identify text information belonging to the target domain.

[0009] In one exemplary embodiment, training the second language model using the second optimizer and text samples belonging to the target domain includes:

[0010] The reference iterative process of extracting the network parameters of the first network layer from the second optimizer yields the third optimizer;

[0011] The second language model is trained using the second optimizer, the third optimizer, and text samples belonging to the target domain.

[0012] In one exemplary embodiment, training the second language model using the second optimizer, the third optimizer, and text samples belonging to the target domain includes:

[0013] The second language model is trained N times using the third optimizer and text samples belonging to the target domain, and the second optimizer is updated according to the N rounds of training to obtain the third language model and the fourth optimizer, where N is an integer greater than 0;

[0014] The third language model is trained for M rounds using the fourth optimizer and text samples belonging to the target domain, where M is an integer greater than 0.

[0015] In an exemplary embodiment, the step of training the second language model N times using the third optimizer and text samples belonging to the target domain, and updating the second optimizer based on the N rounds of training to obtain the third language model and the fourth optimizer, includes:

[0016] In each of the N rounds of training, the first network layer in the second language model is trained P times using the third optimizer and text samples belonging to the target domain to obtain the fourth language model and the fifth optimizer, where P is an integer greater than 0;

[0017] The network parameters of the second network layer in the fourth language model are updated using the network parameters of the first network layer in the fourth language model to obtain the fifth language model. The iterative process of the network parameters of the first network layer recorded in the fifth optimizer is used to update the iterative process of the network parameters of the first and second network layers recorded in the second optimizer to obtain the sixth optimizer.

[0018] The fifth language model is trained K times using the sixth optimizer and text samples belonging to the target domain to obtain the sixth language model and the seventh optimizer, where k is an integer greater than 0;

[0019] Obtain the current training round number in the N rounds of training;

[0020] When the current training round of the second language model is the Nth round of training of the second language model, the sixth language model is determined as the third language model, and the seventh optimizer is determined as the fourth optimizer;

[0021] If the current training rounds for the second language model are any rounds other than the Nth round, the sixth language model is determined as the second language model, and the iterative process for the network parameters of the first network layer recorded in the third optimizer is updated using the iterative process for the network parameters of the first network layer recorded in the seventh optimizer.

[0022] In an exemplary embodiment, updating the iterative process for the network parameters of the first network layer recorded in the third optimizer using the iterative process for the network parameters of the first network layer recorded in the seventh optimizer includes:

[0023] Obtain the first weight corresponding to the first optimization parameters of the first network layer stored in the third optimizer, and the second weight corresponding to the second optimization parameters of the first network layer stored in the seventh optimizer, wherein the optimization parameters are used to indicate the iterative process of the network parameters of the corresponding network layer;

[0024] The first optimization parameter is weighted using the first weight to obtain the third optimization parameter, and the second optimization parameter is weighted using the second weight to obtain the fourth optimization parameter;

[0025] The optimization parameters of the first network layer stored in the third optimizer are updated to the sum of the third optimization parameter and the fourth optimization parameter.

[0026] In an exemplary embodiment, extending the first network layer of the first language model to obtain the second language model includes:

[0027] Obtain the target number of extended layers for the first network layer;

[0028] The first network layer is copied according to the target number of expansion layers to obtain the second network layer with the target number of expansion layers;

[0029] The second network layer, which is connected in sequence, is added after the first network layer of the first language model to obtain the second language model.

[0030] In one exemplary embodiment, obtaining the target number of extended layers for the first network layer includes:

[0031] Obtain the target loss value of the first language model, wherein the target loss value is used to characterize the difference between the recognition response result of the first language model to the target sample and the recognition response result of the target sample annotation;

[0032] The target number of expansion layers corresponding to the target loss value is determined from the loss value and the number of expansion layers with corresponding systems.

[0033] In an exemplary embodiment, updating the first optimizer to obtain a second optimizer for the second language model includes:

[0034] The first information in the first optimizer is copied according to the target number of expansion layers for the first network layer to obtain the second information, wherein the first information is used to indicate the iterative process of the network parameters of the first network layer.

[0035] The second information is added to the first optimizer at the position corresponding to the second network layer to obtain the second optimizer.

[0036] According to another embodiment of this application, a text information recognition device is provided, comprising:

[0037] The detection module is used to detect the training process of the language model for the target text recognition scenario, wherein the target text recognition scenario is a text information recognition scenario belonging to the target domain.

[0038] The acquisition module is used to acquire the first language model obtained by the training process and the first optimizer of the first language model when the training process is detected to be completed. The first optimizer is used to record the iterative process of the network parameters of all network layers included in the first language model when the first language model is obtained.

[0039] An extension module is used to extend the first network layer of the first language model to obtain a second language model, and to update the first optimizer to obtain a second optimizer for the second language model. The second language model includes all network layers and the second network layer included in the first language model. The network parameters of the second network layer are the same as those of the first network layer. The second optimizer is used to record the iteration process of the network parameters of all network layers included in the second language model and the iteration process of the network parameters of the first network layer when obtaining the first language model corresponding to the second network layer.

[0040] The training module is used to train the second language model using the second optimizer and text samples belonging to the target domain to obtain a target language model, wherein the target language model is used to identify text information belonging to the target domain.

[0041] According to yet another embodiment of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and the computer program is configured to perform the steps in any of the above method embodiments when it is run.

[0042] According to yet another embodiment of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0043] This application obtains a first language model trained for a target text recognition scenario, and a first optimizer for the first language model. By extending the network layers based on the first language model, the extended second language model can inherit the training results of the first language model while meeting the requirements for the number of network layers. The first optimizer is then updated to obtain a second optimizer for the second language model. By training the second language model using the second optimizer and text samples belonging to the target domain, the second language model can converge more quickly. Therefore, this addresses the problem of low training efficiency for large language models in related technologies, and improves the training efficiency of large language models. Attached Figure Description

[0044] Figure 1 This is a hardware structure block diagram of a server device for a text information recognition method according to an embodiment of this application;

[0045] Figure 2 This is a flowchart of a text information recognition method according to an embodiment of this application;

[0046] Figure 3 This is a schematic diagram illustrating the configuration of an optional second language model according to an embodiment of this application;

[0047] Figure 4 This is a schematic diagram of one round of training in an optional N-round training according to an embodiment of this application;

[0048] Figure 5 This is a schematic diagram illustrating an optional first language model extended to obtain a second language model according to an embodiment of this application;

[0049] Figure 6 This is a structural block diagram of a text information recognition device according to an embodiment of this application. Detailed Implementation

[0050] The embodiments of this application will be described in detail below with reference to the accompanying drawings and examples.

[0051] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0052] The methods and embodiments provided in this application can be executed on a server device or a similar computing device. Taking running on a server device as an example, Figure 1 This is a hardware structure block diagram of a server device for a text information recognition method according to an embodiment of this application. For example... Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The server device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the server equipment described above. For example, the server equipment may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0053] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the text information recognition method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to server devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0054] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the server device. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0055] This embodiment provides a method for recognizing text information. Figure 2 This is a flowchart of a text information recognition method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0056] Step S202: Detect the training process of training a language model for a target text recognition scenario, wherein the target text recognition scenario is a text information recognition scenario belonging to the target domain.

[0057] Step S204: When it is detected that the training process has been completed, obtain the first language model obtained by the training process and the first optimizer of the first language model, wherein the first optimizer is used to record the iterative process of the network parameters of all network layers included in the first language model when the first language model is obtained.

[0058] Step S206: Expand the first network layer of the first language model to obtain a second language model, and update the first optimizer to obtain a second optimizer for the second language model. The second language model includes all network layers and the second network layer included in the first language model. The network parameters of the second network layer are the same as those of the first network layer. The second optimizer is used to record the iteration process of the network parameters of all network layers included in the second language model and the iteration process of the network parameters of the first network layer when obtaining the first language model corresponding to the second network layer.

[0059] Step S208: The second language model is trained using the second optimizer and text samples belonging to the target domain to obtain a target language model, wherein the target language model is used to identify text information belonging to the target domain.

[0060] Through the above steps, this application obtains a first language model trained for a target text recognition scenario, and a first optimizer for the first language model. By extending the network layers based on the first language model, the extended second language model can inherit the training results of the first language model while meeting the requirements for the number of network layers. The first optimizer is then updated to obtain a second optimizer for the second language model. Furthermore, by training the second language model using the second optimizer and text samples belonging to the target domain, the second language model can converge more quickly. Therefore, this addresses the problem of low training efficiency for large language models in related technologies, and improves the training efficiency of large language models.

[0061] Optionally, the above-mentioned text information recognition method can be deployed, but is not limited to, on a server or terminal device used to run the above-mentioned text information recognition method. The server or terminal device can, but is not limited to, execute the above-mentioned text information recognition method after receiving a text recognition request to recognize target text in the target domain. That is, it can execute the training process of training a language model for the target text recognition scenario, and obtain a first language model and a first optimizer of the first language model obtained during the training process. By expanding the first network layer of the first language model, a second language model is obtained, and the first optimizer is updated to obtain a second optimizer. The second language model is then trained using the second optimizer and text samples in the target domain. After training, a target language model capable of recognizing text in the target domain is obtained. The target language model can then be used to respond to text recognition requests and recognize text information in the target text. Alternatively, the server or terminal can also execute the model training operations included in the above-mentioned text information recognition method at a fixed model training cycle. The trained target language model can then be used to respond to text recognition requests after receiving a text recognition request to recognize target text in the target domain and recognize text information in the target text using the target language model. Optionally, in this embodiment, the recognition of text information may include, but is not limited to, operations such as text generation, text translation, and instruction generation of the target text. For example, the semantics of the target text may be analyzed to generate the text content requested by the target text, or the semantics and context of the target text may be analyzed to translate the target text, or the semantics and context of the target text may be analyzed to generate control instructions requested by the target text to control the operating status of other devices, etc. This solution does not limit this.

[0062] In the embodiment provided in step S204 above, the detection of the training process of training a language model for the target text recognition scenario includes, but is not limited to, detecting the training progress of the current language model, or detecting the degree of matching between the complexity of the currently trained language model and the requirements, and increasing the training scale of the language model if it is confirmed that a more complex language model is needed.

[0063] Optionally, in this embodiment, a language model refers to a probabilistic model used to describe language, which can predict the probability of the next word or a segment of text. Language models can be used to evaluate the rationality, grammatical correctness, and consistency of sentences, and can also be used in natural language processing tasks such as part-of-speech tagging, speech recognition, and machine translation. Common language models include n-gram models, recurrent neural network (RNN) models, long short-term memory (LSTM) models, and Transformer models. In this application, the type of language model is not specifically limited.

[0064] Optionally, in this embodiment, the optimizer refers to an algorithm used to optimize model parameters. In the fields of machine learning and deep learning, optimizers are used to adjust the parameters of a model so that the model can better fit the training data and achieve better performance on the test data. Common optimizers include gradient descent, stochastic gradient descent, Adam, RMSprop, etc. These optimizers update model parameters in different ways to minimize the loss function and improve the model's performance. In this application, the language model is trained by combining the iterative process recorded by the optimizer with text training samples. This application does not specifically limit the type of optimizer involved.

[0065] Optionally, in this application example, the network layer may be used, but is not limited to, to perform operations such as weighting, combining, transforming, feature processing, and data generation on the input data. The network layer may include, but is not limited to, input layers, hidden layers, output layers, fully connected layers, convolutional layers, pooling layers, recurrent layers, etc.

[0066] Optionally, in this embodiment, the iterative process refers to the process of gradually approaching or reaching a goal by repeatedly executing a series of steps or operations. The iterative process typically includes steps such as initialization, loop execution, and state updates, until a specific condition is met.

[0067] Optionally, in this embodiment, the obtained first language model is a small-scale language model with fewer model layers and a shorter training process.

[0068] In the embodiment provided in step S204 above, a small language model with a smaller number of network layers and a simpler training process is pre-trained during the entire training process of the large language model. Then, the large language model can be trained on the basis of the small language model. The large language model absorbs the training results of the small language model and is then trained, which can effectively speed up the training process and improve the efficiency of language model training.

[0069] Optionally, in the embodiments of this application, the first network layer may be, but is not limited to, a set of network layers of a certain type included in the language model, or it may be a sub-layer included in a set of network layers of a certain type. For example, the first network model may be a set of convolutional layers in the language model used to perform convolution operations, or it may be any one of multiple convolutional layers included in the set of convolutional layers. This solution does not limit this.

[0070] Optionally, in this embodiment, extending the first network layer of the first language model to obtain the second language model includes, but is not limited to, copying the first network layer to extend the second network layer based on the first network layer. In this embodiment, when extending the first network layer, it can be extended according to the extension requirements of the number of network layers. For example, if the extension requirement is to extend the first network layer to a second network layer with a target value, the first network layer can be directly extended by copying the target value. This method is suitable for cases where the target value is small. When the number of layers extended in a single step is large, it is not conducive to the convergence of the training of the second language model. Alternatively, when extending the first network layer, it can also be copied and extended according to the target number of extension layers. The target number of extension layers can be a fixed value set based on language model training experience. The target number of extension layers can ensure that the convergence rate of the second language model meets the target convergence rate condition.

[0071] Optionally, in this embodiment, the optimizer of the network model is a tool used to adjust the model parameters to minimize the loss function. The optimizer records the iterative process corresponding to the network parameters of each network layer in the language model.

[0072] Optionally, in this embodiment, updating the first optimizer may be an iterative process in which the network parameters of the first network layer are copied to each second network layer in the first optimizer based on the network layer expansion of the first network layer of the first language model.

[0073] In the embodiment provided in step S208 above, training the second language model using the second optimizer and text samples belonging to the target domain can be achieved through multiple rounds of iterative training using the second optimizer and text samples from the target domain. Specifically, after inputting a text sample from the text set into the second language model, the second optimizer is used to update the model parameters of the second language model based on the output of the second language model, thereby obtaining an updated second language model. Other text samples are then extracted from the text set to train the updated second language model. This training process is repeated multiple times to obtain the trained target language model.

[0074] Optionally, in this embodiment, training the second language model using the second optimizer and text samples belonging to the target domain can also be performed by alternating iterative training of the first network layer and the set of network layers composed of the first and second network layers in the second language model based on the second optimizer and text samples. That is, firstly, the network parameters of the first network layer in the second language model are updated using the second optimizer and text samples from the target domain, and the updated network parameters of the second network layer in the second language model are updated using the updated results. Then, the network parameters of the set of first and second network layers in the second language model are trained using the second optimizer and text samples from the target domain. After updating the network parameters of the set of first and second network layers, the operation of updating the network parameters of the first network layer in the second language model using the second optimizer and text samples from the target domain is performed again. This alternating iterative training of the first network layer and the set of network layers composed of the first and second network layers is repeated multiple times.

[0075] As an optional embodiment, training the second language model using the second optimizer and text samples belonging to the target domain includes:

[0076] The reference iterative process of extracting the network parameters of the first network layer from the second optimizer yields the third optimizer;

[0077] The second language model is trained using the second optimizer, the third optimizer, and text samples belonging to the target domain.

[0078] Optionally, in this embodiment, the optimizer stores the iterative process corresponding to the network parameters of each network layer in the language model.

[0079] Optionally, in this embodiment, training the second language model using the second optimizer, the third optimizer, and text samples from the target domain can be achieved by first using the third optimizer to perform multiple rounds of iterative training on the network parameters of the first network layer in the second language model. During the iterative training, the network parameters of the second network layer are locked, and only the network parameters of the first network layer are trained. Since the number of parameter iterations is small, the network parameters of the first network layer can converge quickly. After the multiple rounds of iterative training on the first network parameters are completed, the iterative update results of the first network parameters can be used to refresh the network parameters of the second network layer in the second network model, as well as refresh the iterative process of all network layers in the second optimizer. Then, the updated second optimizer and text samples from the target domain are used to train the updated first and second network layers of the second language model, thereby enabling the entire second language model to be trained based on the training results of the first network layer, which accelerates the convergence speed of the entire second language model.

[0080] Optionally, in this embodiment, Figure 3 This is a schematic diagram illustrating the configuration of an optional second language model, such as... Figure 3 As shown, the second language model includes all parts 1-10. The combination of 1, 4, 7, and 10 can constitute the original first language model. The second optimizer records the reference iteration process of all network parameters in 1-10. The iteration process of network parameters 1, 4, 7, and 10 is extracted to obtain the third optimizer.

[0081] As an optional embodiment, training the second language model using the second optimizer, the third optimizer, and text samples belonging to the target domain includes:

[0082] The second language model is trained N times using the third optimizer and text samples belonging to the target domain, and the second optimizer is updated according to the N rounds of training to obtain the third language model and the fourth optimizer, where N is an integer greater than 0;

[0083] The third language model is trained for M rounds using the fourth optimizer and text samples belonging to the target domain, where M is an integer greater than 0.

[0084] Optionally, in this embodiment, since the third optimizer only records the reference iteration process of the network parameters of the first network layer, the network parameters of the second network layer in the second language model can be locked during the N rounds of training of the second language model using the third optimizer and text samples from the target domain. That is, the N rounds of training only update and iterate the network parameters of the first network layer in the second language model.

[0085] Optionally, in this embodiment, the fourth optimizer is obtained by updating the second optimizer based on N rounds of training. Therefore, the fourth optimizer stores the iterative process of iterating the network parameters of the first and second network layers included in the third language model. Thus, when training the third language model using the fourth optimizer and text samples belonging to the target domain, it is necessary to update the network parameters of the first and second network layers included in the third language model simultaneously.

[0086] Through the above steps, when training a second language model with a large number of layers, the network parameters of the first network layer in the second language model are first trained for N rounds using a third optimizer. After training, the second optimizer is updated, and the updated third language model is trained using the updated optimizer after N rounds of training. That is, the first network parameters are trained for N rounds first, and the network parameters of the second network layer are locked during the training process. Since the N rounds of training only update the network parameters of the first network layer, the amount of network parameter updates is reduced, and the convergence speed of the first network parameters is accelerated. Then, after the N rounds of training, the updated second optimizer and text samples from the target domain are used to train the first and second network layers of the updated second language model. This allows the entire second language model to be trained based on the training results of the first network layer, thus accelerating the convergence speed of the entire second language model.

[0087] As an optional embodiment, the step of training the second language model N times using the third optimizer and text samples belonging to the target domain, and updating the second optimizer based on the N rounds of training to obtain the third language model and the fourth optimizer, includes:

[0088] In each of the N rounds of training, the first network layer in the second language model is trained P times using the third optimizer and text samples belonging to the target domain to obtain the fourth language model and the fifth optimizer, where P is an integer greater than 0;

[0089] The network parameters of the second network layer in the fourth language model are updated using the network parameters of the first network layer in the fourth language model to obtain the fifth language model. The iterative process of the network parameters of the first network layer recorded in the fifth optimizer is used to update the iterative process of the network parameters of the first and second network layers recorded in the second optimizer to obtain the sixth optimizer.

[0090] The fifth language model is trained K times using the sixth optimizer and text samples belonging to the target domain to obtain the sixth language model and the seventh optimizer, where k is an integer greater than 0;

[0091] Obtain the current training round number in the N rounds of training;

[0092] When the current training round of the second language model is the Nth round of training of the second language model, the sixth language model is determined as the third language model, and the seventh optimizer is determined as the fourth optimizer;

[0093] If the current training rounds for the second language model are any rounds other than the Nth round, the sixth language model is determined as the second language model, and the iterative process for the network parameters of the first network layer recorded in the third optimizer is updated using the iterative process for the network parameters of the first network layer recorded in the seventh optimizer.

[0094] Optionally, in this embodiment, the sixth optimizer stores the iterative process of iterating the network parameters of the first and second network layers included in the fifth language model. Therefore, when training the fifth language model using the sixth optimizer and text samples belonging to the target domain, it is necessary to update the network parameters of the first and second network layers included in the fifth language model simultaneously.

[0095] Optionally, in this embodiment, in each of the N training rounds, the ratio of P training sessions to K training sessions can be a fixed ratio set based on historical training experience. In order to improve the model convergence speed and training efficiency, the value of P is set to be greater than the value of K.

[0096] Optionally, in this embodiment, such as Figure 4 The diagram shown illustrates one round of training in an optional N-round training scenario. Figure 4 As shown, the first network layer of the second language model is trained P times using training text resources and a third optimizer corresponding to the first network layer in the second language model, resulting in a fourth language model and a fifth optimizer. The network parameters of the second network layer in the fourth language model are updated based on the network parameters of the first network layer obtained in the fourth language model, resulting in a fifth language model. The network parameter iteration processes of the first network layer recorded in the fifth optimizer are then updated in the second optimizer, along with the network parameter iteration processes of the first and second network layers, resulting in a sixth optimizer. Thus, the initially obtained fifth language model and sixth optimizer have absorbed the training results of this P round of training. Then, the fifth language model is trained K times based on the training text resources and the obtained sixth optimizer, resulting in the output results of this round of training in N rounds—the sixth language model and a seventh optimizer. The obtained sixth language model and seventh optimizer have absorbed the training results of the P and K rounds of training.

[0097] Based on the above, in each round of training of the second language model using the third optimizer and text samples belonging to the target domain, the first network layer and the set of network layers composed of the first and second network layers are trained alternately. That is, the first network layer is trained P times, and the network parameters of the second network layer and the second optimizer are updated according to the training results to obtain the sixth optimizer. This allows the sixth optimizer and the fifth language model to integrate the P training results of the first network model. The network parameters of the updated first and second network layers are then updated using the sixth optimizer. The fifth language model is trained k times using the sixth optimizer and text samples from the target domain, and the network parameters in the third optimizer and the second network layer are updated after k training. This allows the next round of training in the N rounds to inherit the parameter training results of the set of first and second network models, thereby improving the training accuracy of the model and accelerating the model convergence speed.

[0098] As an optional embodiment, updating the iterative process of the network parameters of the first network layer recorded in the third optimizer using the iterative process of the network parameters of the first network layer recorded in the seventh optimizer includes:

[0099] Obtain the first weight corresponding to the first optimization parameters of the first network layer stored in the third optimizer, and the second weight corresponding to the second optimization parameters of the first network layer stored in the seventh optimizer, wherein the optimization parameters are used to indicate the iterative process of the network parameters of the corresponding network layer;

[0100] The first optimization parameter is weighted using the first weight to obtain the third optimization parameter, and the second optimization parameter is weighted using the second weight to obtain the fourth optimization parameter;

[0101] The optimization parameters of the first network layer stored in the third optimizer are updated to the sum of the third optimization parameter and the fourth optimization parameter.

[0102] Optionally, in the embodiments of this application, the first weight and the second weight can be fixed values ​​based on requirements and values, or they can be weight values ​​corresponding to the model convergence or model loss value determined based on the model convergence or model loss value. This solution does not limit this.

[0103] Through the above steps, during the alternating iterative training of the first network layer and the network layer set composed of the first and second network layers, when the training results of the network layer set are passed to the training process of the first network layer, a weighted approach is adopted, thereby balancing the influence of the training results of the network layer set on the parameter training of the first network layer during the alternating iterative training process.

[0104] As an optional embodiment, the step of extending the first network layer of the first language model to obtain the second language model includes:

[0105] Obtain the target number of extended layers for the first network layer;

[0106] The first network layer is copied according to the target number of expansion layers to obtain the second network layer with the target number of expansion layers;

[0107] The second network layer, which is connected in sequence, is added after the first network layer of the first language model to obtain the second language model.

[0108] Optionally, in this embodiment, the first network layer is used as the teacher model, and the set of network layers consisting of the second network layer (obtained by copying the first network layer) and the first network layer is used as the student model. This effectively transfers the knowledge and generative capabilities learned by the teacher model from massive datasets to the student model. If the number of layers expanded at one time is too large, the student model's convergence speed will slow down, and it will be difficult to converge. The final convergence loss value will also be significantly higher than the final loss value of the student model pre-trained from scratch. To address this, we designed a method to gradually increase the model size. Each time, the number of layers in the student model is expanded to 3-4 times the number of layers in the teacher model. This not only effectively increases the model size but also ensures that the final convergence loss of the student model is not much different from that of direct pre-training. In this way, only two iterations are needed to expand the number of parameters of a pre-trained teacher model to 9 times the original, significantly reducing the computational cost required for training the student model. In other words, if we need to quickly train a student model with 100 times the number of parameters of the teacher model, only 4 iterations are needed to obtain a massive model that inherits the knowledge of the teacher model and also possesses capabilities such as thought chains and context learning, which are only found in large models.

[0109] Optionally, in this embodiment, Figure 5 A schematic diagram illustrating an optional extension of a first language model to obtain a second language model is shown, such as... Figure 5 As shown, after determining the current target number of expansion layers, the first network layer is copied, and the copied second network layer is connected to the corresponding first network layer to obtain the second language model.

[0110] By replicating the first network layer according to the target number of layers through the above steps, the model with a larger number of layers inherits the training results of the first language model with a smaller number of layers, thereby improving the training efficiency of the large model.

[0111] As an optional embodiment, obtaining the target number of extended layers for the first network layer includes:

[0112] Obtain the target loss value of the first language model, wherein the target loss value is used to characterize the difference between the recognition response result of the first language model to the target sample and the recognition response result of the target sample annotation;

[0113] The target number of expansion layers corresponding to the target loss value is determined from the loss value and the number of expansion layers with corresponding systems.

[0114] By taking the above steps, the target loss value of the first language model is obtained, and the number of target expansions of the first network layer is determined based on the loss value of the first language model. This avoids the impact of too many expansion layers on model training and convergence speed, and improves model training efficiency.

[0115] As an optional embodiment, updating the first optimizer to obtain a second optimizer for the second language model includes:

[0116] The first information in the first optimizer is copied according to the target number of expansion layers for the first network layer to obtain the second information, wherein the first information is used to indicate the iterative process of the network parameters of the first network layer.

[0117] The second information is added to the first optimizer at the position corresponding to the second network layer to obtain the second optimizer.

[0118] In this application, to meet the training requirements of language models with a large number of layers, a small model (hereinafter referred to as the teacher model, corresponding to the first network layer in the first language model mentioned above) is first trained. This small model has been pre-trained on a TB-level dataset, but due to the limited parameter size of the model, the teacher model cannot possess good cognitive chain and context learning capabilities. The specific implementation process is as follows:

[0119] 1. We initialize the teacher model's parameters using a constant expansion method (corresponding to the expansion of the first network layer mentioned above), along the layer direction, for the student model (corresponding to the first and second network layers in the second language model mentioned above). For example, to expand a 4-layer pre-trained teacher model to a 12-layer student model, we assign the parameters of the teacher model's first layer to the student model's layers 1-3, and so on. We can use the teacher model to initialize the parameters of all layers in the student model. In addition to the model parameters, we also initialize the optimizer states corresponding to the parameters. The network structure with shared parameters is shown in the figure below, where the orange grid represents the location of the teacher model's parameters, and all grids represent the location of the student model's parameters. The teacher model and the student model have their own optimizers.

[0120] 2. After initializing the parameters and optimizer, the teacher model and student model share parameters. This means the teacher model consists of layers 1, 4, 7, and 10 of the student model. A new optimizer is then created to store the optimizer state of the teacher model. At this point, both the teacher and student models have their own optimizers, learning rates, and other parameters.

[0121] 3. First, the teacher model is trained and updated n_1 times. During the forward propagation process, only the layers containing the teacher model are propagated. For example, after the first layer completes its calculation, the output is passed to the fourth layer, skipping the calculations of layers 2 and 3. When updating model parameters, the optimizer of the teacher model is used. Since the optimizer of the teacher model only includes the optimizer states of layers 1, 4, 7, and 10, only the parameters of the teacher model are updated. Because the calculations of layers 2 and 3 are skipped, the gradient values ​​of layers 2 and 3 are also not calculated during the backpropagation process, reducing memory usage and saving computation.

[0122] 4. Pass the optimizer state of the teacher model to the corresponding network layer of the student model.

[0123] 5. After the student model obtains a new optimizer state, it performs n-2 iterations of updates. During the student model's iterative updates, forward propagation passes through all layers of the student model, and the parameters are updated using the student model's optimizer, which in turn updates the parameters of all layers in the model.

[0124] 6. After completing the iterative update of the student model, pass the state of the corresponding teacher model layer in the student model optimizer state to the teacher model optimizer.

[0125] 7. Repeat steps 1-6 to iteratively update the parameters and optimizer state of the teacher and student models.

[0126] Using the methods described above, we can effectively transfer the knowledge and generative capabilities learned by the teacher model from massive datasets to the student model. However, if the number of layers expanded too much at once, the student model's convergence speed will slow down, making convergence difficult and resulting in a significantly higher final loss value compared to a student model pre-trained from scratch. Therefore, we designed a method to gradually increase the model size. Each time, the number of layers in the student model is increased to 3-4 times the number of layers in the teacher model. This not only effectively increases the model size but also ensures that the final convergence loss of the student model is not significantly different from that of direct pre-training. In this way, only two iterations are needed to increase the number of parameters in a pre-trained teacher model to nine times its original size, significantly reducing the computational cost of training the student model. In other words, if we need to quickly train a student model with 100 times the number of parameters of the teacher model, only four iterations are needed to obtain a massive model that inherits the knowledge of the teacher model and possesses capabilities such as thought chains and contextual learning, which are typically only found in large models.

[0127] In addition, because the shared parameter method is used, the teacher model and the student model share some parameters, which saves the GPU memory resources occupied by the teacher model for these parameters, and can support the training of larger student models.

[0128] In the alternating iterative updates of the teacher and student models, the teacher model's parameter size limitation restricts the student model's capabilities in later stages. To address this, through multiple rounds of experimentation, we used alternating iterative updates for the first 20% of the training process. Once the student model had fully inherited the teacher model's knowledge, we began training the student model independently, ceasing updates to the teacher model. This approach allows for the simultaneous improvement of the teacher model's cognitive abilities, such as its thought processes, during the earlier student model training phases; and avoids the limitation imposed by the teacher model's parameter size on the student model's capabilities during the subsequent independent student model training phases.

[0129] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0130] This embodiment also provides a text information recognition device, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0131] Figure 6 This is a structural block diagram of a text information recognition device according to an embodiment of this application, such as... Figure 6 As shown, the device includes:

[0132] The detection module is used to detect the training process of the language model for the target text recognition scenario, wherein the target text recognition scenario is a text information recognition scenario belonging to the target domain.

[0133] The acquisition module is used to acquire the first language model obtained by the training process and the first optimizer of the first language model when the training process is detected to be completed. The first optimizer is used to record the iterative process of the network parameters of all network layers included in the first language model when the first language model is obtained.

[0134] An extension module is used to extend the first network layer of the first language model to obtain a second language model, and to update the first optimizer to obtain a second optimizer for the second language model. The second language model includes all network layers and the second network layer included in the first language model. The network parameters of the second network layer are the same as those of the first network layer. The second optimizer is used to record the iteration process of the network parameters of all network layers included in the second language model and the iteration process of the network parameters of the first network layer when obtaining the first language model corresponding to the second network layer.

[0135] The training module is used to train the second language model using the second optimizer and text samples belonging to the target domain to obtain a target language model, wherein the target language model is used to identify text information belonging to the target domain.

[0136] Based on the above, a first language model trained for a target text recognition scenario and a first optimizer for the first language model are obtained. By extending the network layers based on the first language model, the extended second language model can inherit the training results of the first language model while meeting the requirements for the number of network layers. The first optimizer is then updated to obtain a second optimizer for the second language model. By training the second language model using the second optimizer and text samples belonging to the target domain, the second language model can converge more quickly. Therefore, this approach can solve the problem of low training efficiency for large language models in related technologies and improve the training efficiency of large language models.

[0137] Optionally, the training module includes:

[0138] An extraction unit is used to extract a reference iterative process of the network parameters of the first network layer from the second optimizer to obtain a third optimizer;

[0139] The training unit is used to train the second language model using the second optimizer, the third optimizer, and text samples belonging to the target domain.

[0140] Optionally, the training unit is further configured to:

[0141] The second language model is trained N times using the third optimizer and text samples belonging to the target domain, and the second optimizer is updated according to the N rounds of training to obtain the third language model and the fourth optimizer, where N is an integer greater than 0;

[0142] The third language model is trained for M rounds using the fourth optimizer and text samples belonging to the target domain, where M is an integer greater than 0.

[0143] Optionally, the training unit is further configured to:

[0144] The step of training the second language model N times using the third optimizer and text samples belonging to the target domain, and updating the second optimizer based on the N rounds of training to obtain the third language model and the fourth optimizer, includes:

[0145] In each of the N rounds of training, the first network layer in the second language model is trained P times using the third optimizer and text samples belonging to the target domain to obtain the fourth language model and the fifth optimizer, where P is an integer greater than 0;

[0146] The network parameters of the second network layer in the fourth language model are updated using the network parameters of the first network layer in the fourth language model to obtain the fifth language model. The iterative process of the network parameters of the first network layer recorded in the fifth optimizer is used to update the iterative process of the network parameters of the first and second network layers recorded in the second optimizer to obtain the sixth optimizer.

[0147] The fifth language model is trained K times using the sixth optimizer and text samples belonging to the target domain to obtain the sixth language model and the seventh optimizer, where k is an integer greater than 0;

[0148] Obtain the current training round number in the N rounds of training;

[0149] When the current training round of the second language model is the Nth round of training of the second language model, the sixth language model is determined as the third language model, and the seventh optimizer is determined as the fourth optimizer;

[0150] If the current training rounds for the second language model are any rounds other than the Nth round, the sixth language model is determined as the second language model, and the iterative process for the network parameters of the first network layer recorded in the third optimizer is updated using the iterative process for the network parameters of the first network layer recorded in the seventh optimizer.

[0151] Optionally, the training unit is further configured to:

[0152] The step of updating the iterative process of the network parameters of the first network layer recorded in the third optimizer using the iterative process of the network parameters of the first network layer recorded in the seventh optimizer includes:

[0153] Obtain the first weight corresponding to the first optimization parameters of the first network layer stored in the third optimizer, and the second weight corresponding to the second optimization parameters of the first network layer stored in the seventh optimizer, wherein the optimization parameters are used to indicate the iterative process of the network parameters of the corresponding network layer;

[0154] The first optimization parameter is weighted using the first weight to obtain the third optimization parameter, and the second optimization parameter is weighted using the second weight to obtain the fourth optimization parameter;

[0155] The optimization parameters of the first network layer stored in the third optimizer are updated to the sum of the third optimization parameter and the fourth optimization parameter.

[0156] Optionally, the extension module includes:

[0157] The acquisition unit is used to acquire the target number of extended layers for the first network layer;

[0158] The first replication unit is used to replicate the first network layer according to the target number of expansion layers to obtain the second network layer with the target number of expansion layers;

[0159] The first addition unit is used to add the second network layer, which is connected in sequence, after the first network layer of the first language model to obtain the second language model.

[0160] Optionally, the acquisition unit is further configured to:

[0161] Obtain the target loss value of the first language model, wherein the target loss value is used to characterize the difference between the recognition response result of the first language model to the target sample and the recognition response result of the target sample annotation;

[0162] The target number of expansion layers corresponding to the target loss value is determined from the loss value and the number of expansion layers with corresponding systems.

[0163] Optionally, the extension module further includes:

[0164] The second copying unit is used to copy the first information in the first optimizer according to the target number of expansion layers for the first network layer, to obtain the second information, wherein the first information is used to indicate the iterative process of the network parameters of the first network layer.

[0165] The second adding unit adds the second information to the first optimizer at the position corresponding to the second network layer, thereby obtaining the second optimizer.

[0166] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0167] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when run.

[0168] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0169] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0170] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0171] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0172] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0173] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method for recognizing text information, characterized in that, include: The training process of training a language model for a target text recognition scenario is detected, wherein the target text recognition scenario is a text information recognition scenario belonging to the target domain; If the training process is detected to be complete, the first language model obtained by the training process and the first optimizer of the first language model are obtained, wherein the first optimizer is used to record the iterative process of the network parameters of all network layers included in the first language model when the first language model is obtained. The first network layer of the first language model is extended to obtain a second language model, and the first optimizer is updated to obtain a second optimizer for the second language model. The second language model includes all network layers and the second network layer included in the first language model. The network parameters of the second network layer are the same as those of the first network layer. The second optimizer is used to record the iteration process of the network parameters of all network layers included in the second language model and the iteration process of the network parameters of the first network layer when the first language model is obtained corresponding to the second network layer. The second language model is trained using the second optimizer and text samples belonging to the target domain to obtain a target language model, wherein the target language model is used to identify text information belonging to the target domain.

2. The method according to claim 1, characterized in that, The step of training the second language model using the second optimizer and text samples belonging to the target domain includes: The reference iterative process of extracting network parameters of the first network layer from the second optimizer is used to obtain the third optimizer; the second language model is trained using the second optimizer, the third optimizer, and text samples belonging to the target domain.

3. The method according to claim 2, characterized in that, The step of training the second language model using the second optimizer, the third optimizer, and text samples belonging to the target domain includes: The second language model is trained N times using the third optimizer and text samples belonging to the target domain, and the second optimizer is updated according to the N rounds of training to obtain the third language model and the fourth optimizer, where N is an integer greater than 0; The third language model is trained for M rounds using the fourth optimizer and text samples belonging to the target domain, where M is an integer greater than 0.

4. The method according to claim 3, characterized in that, The step of training the second language model N times using the third optimizer and text samples belonging to the target domain, and updating the second optimizer based on the N rounds of training to obtain the third language model and the fourth optimizer, includes: In each of the N rounds of training, the first network layer in the second language model is trained P times using the third optimizer and text samples belonging to the target domain to obtain the fourth language model and the fifth optimizer, where P is an integer greater than 0; The network parameters of the second network layer in the fourth language model are updated using the network parameters of the first network layer in the fourth language model to obtain the fifth language model. The iterative process of the network parameters of the first network layer recorded in the fifth optimizer is used to update the iterative process of the network parameters of the first and second network layers recorded in the second optimizer to obtain the sixth optimizer. The fifth language model is trained K times using the sixth optimizer and text samples belonging to the target domain to obtain the sixth language model and the seventh optimizer, where K is an integer greater than 0; Obtain the current training round number in the N rounds of training; When the current training round of the second language model is the Nth round of training of the second language model, the sixth language model is determined as the third language model, and the seventh optimizer is determined as the fourth optimizer; If the current training rounds for the second language model are any rounds other than the Nth round, the sixth language model is determined as the second language model, and the iterative process for the network parameters of the first network layer recorded in the third optimizer is updated using the iterative process for the network parameters of the first network layer recorded in the seventh optimizer.

5. The method according to claim 4, characterized in that, The step of updating the iterative process of the network parameters of the first network layer recorded in the third optimizer using the iterative process of the network parameters of the first network layer recorded in the seventh optimizer includes: Obtain the first weight corresponding to the first optimization parameters of the first network layer stored in the third optimizer, and the second weight corresponding to the second optimization parameters of the first network layer stored in the seventh optimizer, wherein the optimization parameters are used to indicate the iterative process of the network parameters of the corresponding network layer; The first optimization parameter is weighted using the first weight to obtain the third optimization parameter, and the second optimization parameter is weighted using the second weight to obtain the fourth optimization parameter; The optimization parameters of the first network layer stored in the third optimizer are updated to the sum of the third optimization parameter and the fourth optimization parameter.

6. The method according to claim 1, characterized in that, The step of extending the first network layer of the first language model to obtain the second language model includes: Obtain the target number of extended layers for the first network layer; The first network layer is copied according to the target number of expansion layers to obtain the second network layer with the target number of expansion layers; The second network layer, which is connected in sequence, is added after the first network layer of the first language model to obtain the second language model.

7. The method according to claim 6, characterized in that, Obtaining the target number of extended layers for the first network layer includes: Obtain the target loss value of the first language model, wherein the target loss value is used to characterize the difference between the recognition response result of the first language model to the target sample and the recognition response result of the target sample annotation; The target expansion layer number corresponding to the target loss value is determined from the loss value and expansion layer number that have a corresponding relationship.

8. The method according to claim 1, characterized in that, The step of updating the first optimizer to obtain a second optimizer for the second language model includes: The first information in the first optimizer is copied according to the target number of expansion layers for the first network layer to obtain the second information, wherein the first information is used to indicate the iterative process of the network parameters of the first network layer. The second information is added to the first optimizer at the position corresponding to the second network layer to obtain the second optimizer.

9. A text information recognition device, characterized in that, include: The detection module is used to detect the training process of the language model for the target text recognition scenario, wherein the target text recognition scenario is a text information recognition scenario belonging to the target domain. An acquisition module is used to acquire, upon detecting that the training process has been completed, a first language model obtained by the training process and a first optimizer of the first language model, wherein the first optimizer is used to record the iterative process of the network parameters of all network layers included in the first language model when the first language model is obtained; an expansion module is used to expand the first network layer of the first language model to obtain a second language model, and update the first optimizer to obtain a second optimizer of the second language model, wherein the second language model includes all network layers and a second network layer included in the first language model, the network parameters of the second network layer are the same as the network parameters of the first network layer, and the second optimizer is used to record the iterative process of the network parameters of all network layers included in the second language model and the iterative process of the network parameters of the first network layer corresponding to the second network layer when the first language model is obtained; The training module is used to train the second language model using the second optimizer and text samples belonging to the target domain to obtain a target language model, wherein the target language model is used to identify text information belonging to the target domain.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of claims 1 to 8.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 8.

Images