Text processing method and device based on deep learning model and electronic equipment

Abstract

The invention discloses a text processing method and device based on a deep learning model and electronic equipment, the method is applied to the field of artificial intelligence, and the method comprises the following steps: receiving a first text input by a target user; inputting the first text into a target model, and generating a target text for replying to the first text through the target model; and displaying the target text through the target device to respond to the first text input by the target user. Through application of the method and the device, the problem that the model training time is relatively long and the request response efficiency of the model is relatively low due to the fact that a long time needs to be spent on pruning the model according to the equipment condition and training the pruned model when the expanded deep learning model is adopted to respond to the user request in the related technology is solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and specifically, to a text processing method, device and electronic device based on a deep learning model. Background Art

[0002] The training and reasoning of large-scale language models both have high requirements for computing resources. In terms of training, training a large language model with tens of billions or hundreds of billions of parameters requires a lot of GPU training time, which makes training larger models (such as hundreds of billions of parameters) an unattainable goal and slows down the speed of model updates. In terms of reasoning, large-scale language models have a large amount of video memory usage and reasoning latency, which makes it difficult to deploy the model on resource-constrained devices and the reasoning speed will also be slow.

[0003] In order to reduce the resource consumption of model training and reduce the resource usage and inference latency of model inference, a feasible way is to expand the network parameters of the model during the training process, that is, to expand the model, then prune the expanded model, and finally use the pruned model to process the questions input by the user, generate reply text, and respond to the questions input by the user according to the reply text. However, the existing model pruning method directly deletes some parameters by evaluating the importance of parameters. After that, it generally takes a long time of training to achieve an effect close to the original model. The efficiency is very low, which reduces the efficiency of the model in responding to user requests. In addition, the existing dynamic network method applied to large-scale language models can only prune part of the network structure (that is, delete some layers), which is not flexible enough and can only compress the model to a few fixed parameter scales, resulting in reduced accuracy of the model during inference.

[0004] When an expanded deep learning model is used in related technologies to respond to user requests, it takes a long time to prune the model according to the device conditions and train the pruned model, resulting in a long model training time and low efficiency in the model responding to requests. No effective solution has been proposed yet. Summary of the invention

[0005] The main purpose of the present application is to provide a text processing method, device and electronic device based on a deep learning model, so as to solve the problem in the related art that when an expanded deep learning model is used to respond to user requests, it takes a long time to prune the model according to the device conditions and train the pruned model, resulting in a long model training time and low efficiency in model response to requests.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a text processing method based on a deep learning model is provided, the method comprising: receiving a first text input by a target user; inputting the first text into a target model, and generating a target text for replying to the first text through the target model, wherein the target model is a model that is pre-trained during model training, the network parameters of the model are expanded after pre-training, and the network parameters of the model are pruned during model inference; and the target text is displayed through a target device in response to the first text input by the target user.

[0007] Furthermore, the target model is trained by the following steps: pre-training the preset model with a preset data set until the model converges to obtain a first model, wherein the preset data set is a data set constructed based on the user's historical question and answer text; expanding the network parameters of the first model by copying the network parameters, and training the expanded first model to obtain a second model; calculating the occupied space of the model based on the device information of the target device to obtain a target value; compressing the network parameters of the second model based on the target value to obtain the target model.

[0008] Furthermore, the network parameters of the first model are expanded by copying the network parameters, including: determining N first parameter matrices based on a preset numerical matrix, wherein the preset numerical matrix is ​​a parameter matrix of the first network structure in the first model, and N is a positive integer; updating the first model based on the N first parameter matrices to obtain a third model; determining M second network structures based on the second network structure, wherein the second network structure is a network structure in the third model, and M is a positive integer; updating the third model based on the M second network structures to obtain the expanded first model.

[0009] Furthermore, the network parameters of the second model are compressed according to the target value to obtain the target model, including: calculating the pruning quantity according to the target value; pruning the N first parameter matrices in the second model according to the pruning quantity to obtain a fourth model; pruning the M second network structures in the fourth model according to a preset quantity to obtain the target model.

[0010] Furthermore, the first model includes a router module. After determining N first parameter matrices according to a preset numerical matrix, the method also includes: determining a weight value of each of the N first parameter matrices through the router module; when detecting that the first data is input into the N first network structures corresponding to the N first parameter matrices, calculating the output results of the N first network structures according to the weight value of each first parameter matrix.

[0011] Furthermore, after determining M second network structures based on the second network structure, the method also includes: when detecting that the second data is input into the M second network structures, obtaining the output results of the M second network structures; inputting the output results of the M second network structures into a preset mapping matrix, and calculating the loss function value of each second network structure in the M second network structures; and updating the network parameters in the expanded first model according to the loss function value of each second network structure.

[0012] Furthermore, the pruning amount is calculated according to the target value, including: obtaining training samples, and constructing a calibration set according to the training samples; for each of the M second network structures in the second model, inputting the calibration set into the N first parameter matrices in the second network structure, and obtaining the weight value of each first parameter matrix in the N first parameter matrices; determining the pruning amount according to the weight value of each first parameter matrix in each second network structure, wherein the pruning amount is an integer greater than a preset value.

[0013] Furthermore, after pruning the N first parameter matrices in the second model according to the pruning quantity to obtain a fourth model, the method also includes: calculating the weight value of each first parameter matrix in the fourth model, wherein the sum of the weight values ​​of each first parameter matrix is ​​a preset value; and training each first parameter matrix and the weight value of each first parameter matrix in the fourth model using training samples to update each first parameter matrix in the fourth model.

[0014] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a text processing device based on a deep learning model is provided, and the device includes: a receiving unit, which is used to receive a first text input by a target user; a generating unit, which is used to input the first text into a target model, and generate a target text for replying to the first text through the target model, wherein the target model is a model that is pre-trained during model training, the network parameters of the model are expanded after pre-training, and the network parameters of the model are pruned during model inference; a responding unit, which is used to display the target text through a target device in response to the first text input by the target user.

[0015] Furthermore, the device also includes: a training unit, used to pre-train a preset model using a preset data set until the model converges to obtain a first model, wherein the preset data set is a data set constructed based on the user's historical question and answer text; an expansion unit, used to expand the network parameters of the first model by copying the network parameters, and train the expanded first model to obtain a second model; a calculation unit, used to calculate the occupied space of the model based on the device information of the target device to obtain a target value; a compression unit, used to compress the network parameters of the second model based on the target value to obtain the target model.

[0016] Furthermore, the expansion unit includes: a first determination subunit, used to determine N first parameter matrices based on a preset numerical matrix, wherein the preset numerical matrix is ​​a parameter matrix of the first network structure in the first model, and N is a positive integer; a first update subunit, used to update the first model based on the N first parameter matrices to obtain a third model; a second determination subunit, used to determine M second network structures based on the second network structure, wherein the second network structure is the network structure in the third model, and M is a positive integer; a second update subunit, used to update the third model based on the M second network structures to obtain the expanded first model.

[0017] Furthermore, the compression unit includes: a first calculation subunit, used to calculate the pruning amount according to the target value; a first pruning processing subunit, used to prune the N first parameter matrices in the second model according to the pruning amount to obtain a fourth model; a second pruning processing subunit, used to prune the M second network structures in the fourth model according to a preset number to obtain the target model.

[0018] Furthermore, the first model includes a router module, and the extension unit also includes: a third determination subunit, which is used to determine the weight value of each of the N first parameter matrices through the router module after determining the N first parameter matrices according to a preset numerical matrix; and a second calculation subunit, which is used to calculate the output results of the N first network structures according to the weight value of each first parameter matrix when it is detected that the first data is input into the N first parameter matrices corresponding to the N first parameter matrices.

[0019] Furthermore, the expansion unit also includes: an acquisition subunit, which is used to obtain output results of the M second network structures after determining M second network structures based on the second network structure, when it is detected that the second data is input into the M second network structures; a third calculation subunit, which is used to input the output results of the M second network structures into a preset mapping matrix, and calculate the loss function value of each second network structure in the M second network structures; and a third updating subunit, which is used to update the network parameters in the expanded first model according to the loss function value of each second network structure.

[0020] Furthermore, the first computing subunit includes: a construction module, used to obtain training samples and construct a calibration set based on the training samples; an acquisition module, used to input the calibration set into the N first parameter matrices in the second network structure for each of the M second network structures in the second model, and obtain the weight value of each first parameter matrix in the N first parameter matrices; a determination module, used to determine the pruning number based on the weight value of each first parameter matrix in each second network structure, wherein the pruning number is an integer greater than a preset value.

[0021] Furthermore, the compression unit also includes: a fourth calculation subunit, used to prune the N first parameter matrices in the second model according to the pruning amount to obtain a fourth model, and then calculate the weight value of each first parameter matrix in the fourth model, wherein the sum of the weight values ​​of each first parameter matrix is ​​a preset value; a fourth updating subunit, used to train each first parameter matrix in the fourth model and the weight value of each first parameter matrix using training samples to update each first parameter matrix in the fourth model.

[0022] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a computer program product is provided, including a computer program, which, when executed by a processor, implements any one of the above-mentioned text processing methods based on deep learning models, and which, when executed by a processor, implements the steps of the text processing method based on deep learning models described in each embodiment of the present application.

[0023] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes stored computer instructions, wherein when the computer instructions are executed by a processor, any one of the above-mentioned text processing methods based on the deep learning model is implemented.

[0024] In order to achieve the above-mentioned purpose, according to one aspect of the present application, an electronic device is provided, comprising one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement any one of the above-mentioned text processing methods based on deep learning models.

[0025] Through the present application, the following steps are adopted: receiving a first text input by a target user; inputting the first text into a target model, and generating a target text for replying to the first text through the target model, wherein the target model is a model that is pre-trained during model training, the network parameters of the model are expanded after pre-training, and the network parameters of the model are pruned during model inference; the target text is displayed through a target device to respond to the first text input by the target user, thereby solving the problem in the related art that when an expanded deep learning model is used to respond to user requests, it takes a long time to prune the model according to the device situation and train the pruned model, resulting in a long model training time and low efficiency in model response to requests. By pre-training the model before model training and expanding the network parameters of the pre-trained model, the parameters of the pre-trained model can initialize the model parameters more efficiently, so that the model training cost after the network parameters are expanded is greatly reduced, and the computing power consumption during model training is reduced. At the same time, by reducing the network parameters of the model according to the method of expanding the network parameters of the model during model inference, the resource occupancy and inference delay of model inference are reduced while maintaining the model accuracy. This avoids the problem of using a lot of time to train the pruned model, which leads to low model response efficiency, and achieves the effect of improving the efficiency of the model in responding to user requests. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0027] Figure 1 is a flowchart of a text processing method based on a deep learning model provided according to Embodiment 1 of the present application;

[0028] Figure 2 This is a schematic diagram of an optional text processing method based on a deep learning model provided in Example 1 of the present application. Figure 1 ;

[0029] Figure 3 This is a schematic diagram of an optional text processing method based on a deep learning model provided in Example 1 of the present application. Figure 2 ;

[0030] Figure 4 This is a schematic diagram of an optional text processing method based on a deep learning model provided in Example 1 of the present application. Figure 3 ;

[0031] Figure 5 is a schematic diagram of a text processing device based on a deep learning model provided according to Embodiment 2 of the present application;

[0032] Figure 6 This is a schematic diagram of a text processing electronic device based on a deep learning model provided according to Example 5 of the present application. DETAILED DESCRIPTION

[0033] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0034] It should be noted that the user information (including but not limited to user device information, user personal information, collected data, used data, generated data, processed data, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, collected information, used information, generated information, processed information, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data are in compliance with the relevant laws, regulations and standards of relevant countries and regions, necessary confidentiality measures are taken, and public order and good morals are not violated, and corresponding operation portals are provided for users to choose to authorize or refuse. For example, an interface is set between this system and relevant users or organizations. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or organization.

[0035] It should be noted that this application provides users with corresponding operation entrances for them to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered.

[0036] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0037] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0038] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0039] Transformer: It is a neural network architecture based on the self-attention mechanism. Through the self-attention mechanism and the layer-by-layer stacked encoder-decoder structure, it realizes the ability to globally model the input sequence and generate the target sequence.

[0040] Text pre-training: Text pre-training refers to the process of training a model on a large-scale text dataset. This initial training phase allows the model to learn general features and representations from text data, which can then be fine-tuned for specific tasks.

[0041] Mixture of Experts (MoE): A mixture of expert models is a machine learning technique in which multiple expert networks are used to divide the problem space into homogeneous regions. A significant advantage of mixture of expert models is that they can be effectively pre-trained with far fewer computational resources than dense models. This means that the size of the model can be significantly expanded under the same computational budget.

[0042] Early Stopping: It is a technique used in deep neural network (DNN) inference to reduce redundant calculations and increase inference speed. The core idea is to add some judgment conditions in the middle layer of the network so that the results can be output in advance when simple samples are identified without having to execute the deep structure of the entire network.

[0043] Embodiment 1

[0044] The present invention is described below in conjunction with preferred implementation steps. Figure 1 is a flowchart of a text processing method based on a deep learning model provided in Example 1 of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0045] Step S101: receiving a first text input by a target user.

[0046] The text processing method based on the deep learning model provided in the first embodiment of the present application can be applied to a target device, which is a device that interacts with a user. For example, the user inputs questions or other text information in the target device, the target device receives the text information input by the user, and processes the text information input by the user through the target model to generate a reply text, and displays the reply text in the target device in response to the text information input by the user.

[0047] In the first embodiment, in order to respond to the user request (ie, reply to the first text input by the user), it is necessary to receive the first text input by the user, generate a reply text according to the first text input by the user, and respond to the user request according to the reply text.

[0048] It should be noted that, in addition to being used for general knowledge question and answer, the above-mentioned target model can also be used for code generation, knowledge mounting, long article writing, intelligent data retrieval, intelligent customer service (or digital human), email generation and many other projects, which are not specifically limited in this embodiment 1. Therefore, the first text input by the user can be "Please generate a Python code for a convolutional neural network model for image classification.", "Please explain what is a long short-term memory network (LSTM) and its application in time series prediction.", "Please write a paper on the application of deep learning in medical image analysis.", "I need to obtain user behavior data for the past year, including click-through rate and conversion rate.", "I need an AI customer service robot that can answer common questions." and "Please generate an automatic reply email to inform users that the order has been shipped." and other text information.

[0049] Step S102, inputting the first text into the target model, and generating a target text for replying to the first text through the target model, wherein the target model is a model that is pre-trained during model training, the network parameters of the model are expanded after pre-training, and the network parameters of the model are pruned during model inference.

[0050] In the prior art, in order to reduce the resource consumption of model training, a feasible way is to expand the network parameters of the model during the model training process, that is, to expand the model, then prune the expanded model, and finally use the pruned model to process the questions input by the user during the model reasoning process, generate a reply text, and respond to the questions input by the user according to the reply text. However, the existing model pruning method directly deletes some parameters by evaluating the importance of parameters, and then generally requires a long period of training to achieve an effect close to the original model, which is very inefficient.

[0051] In the first embodiment of the present invention, it is necessary to pre-train the model before model training, and expand the network parameters of the pre-trained model. During model inference, the network parameters of the model are correspondingly reduced according to the method of expanding the network parameters to obtain the above-mentioned target model, and the target model is used to generate a reply text, thereby combining the two processes of model expansion and model scaling, and realizing the mutual conversion from a small model to a large model, thereby accelerating the reasoning speed of the model without affecting the accuracy of the model, reducing the response time of the model to respond to user requests, and improving the response efficiency of the model.

[0052] Step S103: displaying the target text via the target device in response to the first text input by the target user.

[0053] In this first embodiment, in order to meet the user request, the reply text generated by the target model (i.e., the target text mentioned above) needs to be displayed in the target device that interacts with the target user, and the target text is displayed to the target user through the target device in response to the first text input by the target user.

[0054] In summary, the text processing method based on the deep learning model provided in the first embodiment of the present application receives a first text input by a target user; inputs the first text into a target model, and generates a target text for replying to the first text through the target model, wherein the target model is a model that is pre-trained during model training, the network parameters of the model are expanded after pre-training, and the network parameters of the model are pruned during model inference; the target text is displayed through a target device to respond to the first text input by the target user, thereby solving the problem in the related art that when an expanded deep learning model is used to respond to a user request, it takes a long time to prune the model according to the device situation and train the pruned model, resulting in a long model training time and low efficiency in the model responding to requests. By pre-training the model before model training and expanding the network parameters of the pre-trained model, the parameters of the pre-trained model can initialize the model parameters more efficiently, so that the model training cost after the network parameters are expanded is greatly reduced, and the computing power consumption during model training is reduced. At the same time, by reducing the network parameters of the model according to the method of expanding the network parameters of the model during model inference, the resource occupancy and inference delay of model inference are reduced while maintaining the model accuracy. This avoids the problem of using a lot of time to train the pruned model, which leads to low model response efficiency, and achieves the effect of improving the efficiency of the model in responding to user requests.

[0055] Optionally, in the text processing method based on the deep learning model provided in the first embodiment of the present application, the above-mentioned target model is trained by the following steps: pre-training the preset model with a preset data set until the model converges to obtain a first model, wherein the preset data set is a data set constructed based on the user's historical question and answer text; expanding the network parameters of the first model by copying the network parameters, and training the expanded first model to obtain a second model; calculating the occupied space of the model based on the device information of the target device to obtain the target value; compressing the network parameters of the second model based on the target value to obtain the target model.

[0056] In the first embodiment, in order to improve the model response efficiency, a preset model with a smaller network parameter amount can be trained first until the model converges to obtain a first model, and the network parameters of the first model are expanded, and the first model is expanded into a large model with a larger parameter amount (i.e., the expanded first model mentioned above) and continued to be trained until the model converges to obtain the above-mentioned second model. During the model training process, the network parameters of the pre-trained first model can provide good initialization network parameters for the expanded first model, so that the training cost of the expanded first model is much lower than training from scratch.

[0057] Then, in order to reduce the resource usage and inference delay of model inference, the actual amount of computing resources can be determined according to the device information of the target device, and the appropriate model size, that is, the above-mentioned target value, can be determined according to the actual amount of computing resources. Finally, according to the target value, the second model with a larger amount of network parameters is scaled into a target model with a smaller network parameter scale, and at the same time, the target model has the performance of the second model.

[0058] In the first embodiment, the transformer model (hereinafter referred to as the Transformer model) is used as an example to explain the expansion and scaling process of the model network parameters, that is, the Transformer model is used as the above-mentioned preset model. It should be noted that in actual production activities, other deep learning models can be used as preset models according to actual production conditions, and no specific restrictions are made in the first embodiment.

[0059] Optionally, in the text processing method based on the deep learning model provided in Example 1 of the present application, the network parameters of the first model are expanded by copying the network parameters, including: determining N first parameter matrices based on a preset numerical matrix, wherein the preset numerical matrix is ​​the parameter matrix of the first network structure in the first model, and N is a positive integer; updating the first model based on the N first parameter matrices to obtain a third model; determining M second network structures based on the second network structure, wherein the second network structure is the network structure in the third model, and M is a positive integer; updating the third model based on the M second network structures to obtain the expanded first model.

[0060] In the first embodiment, in order to reduce the model training cost, the pre-trained first model can be expanded horizontally and vertically.

[0061] In an optional embodiment, the process of horizontally expanding the Transformer model can be as follows: Figure 2 As shown, by copying N copies of the self-attention of the first model (such as Figure 2 The network parameter matrix of the attn1, attn2, attn3 and attn4 modules (i.e., the first parameter matrix mentioned above) and the MLP module (e.g. Figure 2 The network parameter matrix of MLP1, MLP2, MLP3 and MLP4 in the first model (i.e., the first parameter matrix mentioned above) is used to form K independent expert modules. For example, the W of the self-attention module in the first model can be q , W k , W v , W, these four network parameter matrices are copied N times, and we get Where i = 1, ..., N. For the input vector x of the self-attention module, the calculation process of the self-attention module can be shown as formula 1 to formula 4,

[0062]

[0063] h (i) =attn(Q (i) , K (i) , V (i) ) (three)

[0064]

[0065] Among them, y (i) represents the output result of the i-th self-attention module, W q , W k , W v Respectively represent the Q, K, and V mapping matrices in the self-attention module, with a size of d×d, W o represents the output mapping matrix of the self-attention module, with a size of d×d, where d represents the size of the hidden layer and h (i) Usually represents the output after processing by the i-th self-attention module. The first model is updated according to the network parameter matrix of the self-attention module after N copies and the network parameter matrix of the MLP module after N copies, and the N first parameter matrices to obtain the third model.

[0066] In an optional embodiment, the process of vertically expanding the Transformer model can be as follows: Figure 3As shown in the figure, for a Transformer model with L layers, a model with 2L layers (such as Figure 3 For example, for the MLP module in the third model (i.e., the second network structure mentioned above), the two matrices W1 and W2 of the third model are copied M times (i.e., the M second network structures mentioned above), and the result is Where i = 1, ..., M. For the input vector x of the MLP module, the calculation process of the MLP module can be shown as formula 5,

[0067]

[0068] Among them, y (i) represents the output result of the i-th MLP module, Represents the two mapping matrices of the MLP layer, w1∈R d×4d , W2∈R 4d×d , d represents the size of the hidden layer, and f represents the calculation function corresponding to the MLP module. The third model is updated according to the M second network structures to obtain the expanded first model.

[0069] Optionally, in the text processing method based on the deep learning model provided in the first embodiment of the present application, the network parameters of the second model are compressed according to the target value to obtain the target model, including: calculating the pruning quantity according to the target value; pruning the N first parameter matrices in the second model according to the pruning quantity to obtain a fourth model; pruning the M second network structures in the fourth model according to a preset quantity to obtain the target model.

[0070] In the first embodiment, in order to reduce the video memory occupancy and inference latency of inference, the amount of network parameters that need to be pruned (i.e., the above-mentioned pruning amount) can be determined based on the actual computing resources of the target device used by the user, and the trained second model can be pruned according to the pruning amount.

[0071] In an optional embodiment, before deploying the model, the size to be compressed, i.e., the above-mentioned target value, can be determined based on the video memory size and computing resource limitations of the target device used by the target user. Then, the occupied space of the second model is compressed to the target value through horizontal compression and vertical compression.

[0072] In an optional embodiment, lateral compression is the inverse process of lateral expansion. During model inference, the number of pruning can be calculated according to the target value, that is, the number of network parameters that need to be pruned, and the N first parameter matrices horizontally expanded from the second model are pruned according to the number of pruning, and some expert models of some layers are pruned to obtain the above-mentioned fourth model, thereby making the N expert models sparse and reducing video memory usage and inference latency.

[0073] Longitudinal compression is the reverse process of longitudinal expansion. After pruning the N first parameter matrices of the second model, the M second network structures in the fourth model can be pruned. Because during the training process of the second model, the first L layers and the overall model (M×L layers) in the second model have the ability to generate tokens. Therefore, exemplarily, during model inference, only the first L layers can be used for inference, that is, the network structure of the i+Lth layer (i=1, 2,…, (M-1)×L) in the fourth model is pruned to obtain the above-mentioned target model. It should be noted that 2×L layers can also be used for inference during model inference, and there is no specific limitation in the first embodiment.

[0074] Optionally, in the text processing method based on the deep learning model provided in the first embodiment of the present application, the above-mentioned first model includes a router module. After determining N first parameter matrices according to a preset numerical matrix, the above-mentioned method also includes: determining the weight value of each first parameter matrix in the N first parameter matrices through the router module; when it is detected that the first data is input into the N first parameter matrices corresponding to the N first network structures, the output results of the N first network structures are calculated according to the weight value of each first parameter matrix.

[0075] In the first embodiment, in order to avoid the negative impact of the expanded network parameters on the accuracy of the target model, after the network parameters are horizontally expanded, the output results of the N first network structures corresponding to the N first parameter matrices can be determined by adjusting the weight values ​​of the N first parameter matrices.

[0076] In an optional embodiment, after the vector x is input into the N first network structures, a router module G is added, and the router module calculates the weight value corresponding to each first network structure according to the vector x, and performs weighted summation on the output results of the N first network structures to obtain the output results of the N first network structures. Exemplarily, the calculation process of calculating the output results of the N first network structures can be shown as Formula 6 and Formula 7,

[0077] G = softmax(W g x) (vi)

[0078]

[0079] Among them, G represents the weight matrix of N first network structures, W g Represents the above router module, W g ∈R d×N , i represents the i-th first network structure, y (i) represents the output vector of the i-th first network structure, y represents the output result of N first network structures, G i Represents the weight value of the i-th first network structure, and softmax represents normalization processing.

[0080] Optionally, in the text processing method based on the deep learning model provided in the first embodiment of the present application, after determining M second network structures based on the second network structure, the above method also includes: when it is detected that the second data is input into the M second network structures, obtaining the output results of the M second network structures; inputting the output results of the M second network structures into a preset mapping matrix, and calculating the loss function value of each second network structure in the M second network structures; and updating the network parameters in the expanded first model based on the loss function value of each second network structure.

[0081] In the first embodiment, in order to maintain the accuracy of the model when scaling the model network parameters after the model expands the network parameters, after copying M second network structures, the output vector of each of the M second network structures can be input into the lm head mapping matrix respectively, and the loss function value of the output vector of each second network structure can be calculated using an independent language model loss module respectively, and the loss function value of the output vector of each second network structure is used to update the network parameters in the expanded first model, so that each of the M second network structures has the ability to generate tokens.

[0082] Optionally, in the text processing method based on the deep learning model provided in the first embodiment of the present application, the pruning amount is calculated according to the target value, including: obtaining training samples, and constructing a calibration set based on the training samples; for each of the M second network structures in the second model, the calibration set is input into the N first parameter matrices in the second network structure, and the weight value of each first parameter matrix in the N first parameter matrices is obtained; and the pruning amount is determined according to the weight value of each first parameter matrix in each second network structure.

[0083] In the first embodiment, in order to ensure the accuracy of the model after pruning, the number to be pruned, ie, the above-mentioned pruning number, can be determined according to the compression target size (ie, the above-mentioned target value).

[0084] In an optional embodiment, before model training, a small number of samples can be randomly selected from the above training set to construct the above calibration set. The sample data in the calibration set is input into the N first parameter matrices of each second network structure of the M second network structures in the second model to obtain the weight value of each first parameter matrix in each second network structure.

[0085] Then, the sum of the weight values ​​assigned by the routing module to each first parameter matrix in the M second network structures is counted to obtain the sum of the weight values ​​of each first parameter matrix in the N first parameter matrices. The sum of the weight values ​​of each first parameter matrix in the N first parameter matrices is sorted in order from small to large, and the P first parameter matrices with the smallest sum of weight values ​​in the N first parameter matrices are pruned.

[0086] It should be noted that after pruning the N first parameter matrices in the M second network structures, it is necessary to ensure that at least one first parameter matrix is ​​retained in each second network structure.

[0087] Optionally, in the text processing method based on the deep learning model provided in the first embodiment of the present application, after pruning the N first parameter matrices in the second model according to the pruning number to obtain the fourth model, the above method also includes: calculating the weight value of each first parameter matrix in the fourth model, wherein the sum of the weight values ​​of each first parameter matrix is ​​a preset value; and training each first parameter matrix and the weight value of each first parameter matrix in the fourth model using training samples to update each first parameter matrix in the fourth model.

[0088] In the first embodiment, in order to ensure the accuracy of the model after pruning, when calculating the weight of the routing module, only the retained first parameter matrix can be normalized (softmax) so that the sum of the weight values ​​of all the first network structures corresponding to all the retained first parameter matrices is 1. Then, a small amount of sample data can be used to train each first parameter matrix and the weight value of each first parameter matrix in the fourth model, and each first parameter matrix in the fourth model can be updated to reduce the performance loss after the model pruning process.

[0089] Optionally, in the first embodiment, the process of using the target model after the network parameters are extended and scaled to respond to the user request of the target user can be as follows: Figure 4 shown.

[0090] First, a randomly initialized Transformer model (i.e., the preset model mentioned above) is pre-trained from scratch, and then trained to convergence using a data set containing 1T token to obtain model M1 (i.e., the first model mentioned above). Exemplarily, in this embodiment, a model structure with 7 billion (abbreviated as 7B) parameters is used as the preset model. The model has 30 transformer layers, a hidden layer size of 4096, 32 attention heads per layer, a vocabulary size of 120,000, and the Megatron-Deepspeed framework is used for model training, and the training data set size is 1T token.

[0091] Then, each Transformer layer in the model M1 containing 7B parameters is horizontally expanded to obtain the third model mentioned above, where N=2 is set during the horizontal expansion process. The third model is trained with a preset training set. After a short period of training, it can be observed that the loss of the third model decreases, and the rate of decrease is faster than the rate of decrease when directly training the model containing 7B parameters. After the model M1 is expanded, the model parameter volume is 13B. Continue to train 200B tokens on the pre-training data set until the model converges, and obtain the model M2 with twice the parameter scale.

[0092] Secondly, the Transformer layer of the model M2 with 13B parameters is vertically expanded to obtain the above-mentioned expanded model. After a short period of training, the loss can be observed to decrease, and the rate of decrease is faster than the rate of decrease when directly training the model with 13B parameters. The model parameter volume after vertical expansion is 25B. Continue to train 100B tokens on the pre-training dataset until the model converges, and obtain the model M3 with a parameter scale expanded four times.

[0093] Secondly, according to the video memory information and computing resources of the target device, the model compression size (i.e., the above-mentioned target value) is determined, and the number of pruning is determined. The second model is horizontally pruned and vertically early stopped to compress the occupied space of the second model to the target value, and the compressed model M3' is obtained. In addition, the model M3' is trained on a small amount of data for a small number of steps to improve the performance of the model M3'.

[0094] Finally, the model M3' is deployed in the target device so as to interact with the target user through the model M3' in the target device. For example, the target user inputs a first text in the target device, and the target text is generated through the model M3', and the target text is displayed in the target device in response to the first text input by the user.

[0095] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0096] Embodiment 2

[0097] Embodiment 2 of the present application also provides a text processing device based on a deep learning model. It should be noted that the text processing device based on a deep learning model in Embodiment 2 of the present application can be used to execute the text processing method based on a deep learning model provided in Embodiment 1 of the present application. The text processing device based on a deep learning model provided in Embodiment 2 of the present application is introduced below.

[0098] Figure 5 is a schematic diagram of a text processing device based on a deep learning model according to the second embodiment of the present application. Figure 5 As shown, the device includes: a receiving unit 501, a generating unit 502 and a responding unit 503.

[0099] Specifically, the receiving unit 501 is configured to receive a first text input by a target user.

[0100] The generation unit 502 is used to input the first text into the target model, and generate a target text for replying to the first text through the target model, wherein the target model is a model that is pre-trained during model training, the network parameters of the model are expanded after pre-training, and the network parameters of the model are pruned during model inference.

[0101] The responding unit 503 is configured to display the target text via the target device in response to the first text input by the target user.

[0102] The text processing device based on the deep learning model provided in the second embodiment of the present application receives the first text input by the target user through the receiving unit 501; the generating unit 502 inputs the first text into the target model, and generates the target text for replying to the first text through the target model, wherein the target model is a model that is pre-trained during model training, the network parameters of the model are expanded after pre-training, and the network parameters of the model are pruned during model inference; the responding unit 503 displays the target text through the target device to respond to the first text input by the target user, which solves the problem in the related art that when the expanded deep learning model is used to respond to the user request, it takes a long time to prune the model according to the device situation and train the pruned model, resulting in a long model training time and low efficiency in model response to requests. By pre-training the model before model training and expanding the network parameters of the pre-trained model, the parameters of the pre-trained model can initialize the model parameters more efficiently, so that the model training cost after the network parameters are expanded is greatly reduced, and the computing power consumption during model training is reduced. At the same time, by reducing the network parameters of the model according to the method of expanding the network parameters of the model during model inference, the resource occupancy and inference delay of model inference are reduced while maintaining the model accuracy. This avoids the problem of using a lot of time to train the pruned model, which leads to low model response efficiency, and achieves the effect of improving the efficiency of the model in responding to user requests.

[0103] Optionally, in the text processing device based on the deep learning model provided in Example 2 of the present application, the above-mentioned device also includes: a training unit, used to pre-train the preset model using a preset data set until the model converges to obtain a first model, wherein the preset data set is a data set constructed based on the user's historical question and answer text; an expansion unit, used to expand the network parameters of the first model by copying the network parameters, and train the expanded first model to obtain a second model; a calculation unit, used to calculate the occupied space of the model based on the device information of the target device to obtain a target value; a compression unit, used to compress the network parameters of the second model based on the target value to obtain a target model.

[0104] Optionally, in the text processing device based on the deep learning model provided in Example 2 of the present application, the above-mentioned expansion unit includes: a first determination subunit, used to determine N first parameter matrices based on a preset numerical matrix, wherein the preset numerical matrix is ​​the parameter matrix of the first network structure in the first model, and N is a positive integer; a first updating subunit, used to update the first model based on the N first parameter matrices to obtain a third model; a second determination subunit, used to determine M second network structures based on the second network structure, wherein the second network structure is the network structure in the third model, and M is a positive integer; a second updating subunit, used to update the third model based on the M second network structures to obtain the expanded first model.

[0105] Optionally, in the text processing device based on the deep learning model provided in Example 2 of the present application, the above-mentioned compression unit includes: a first calculation subunit, used to calculate the pruning amount according to the target value; a first pruning processing subunit, used to prune the N first parameter matrices in the second model according to the pruning amount to obtain a fourth model; a second pruning processing subunit, used to prune the M second network structures in the fourth model according to a preset number to obtain a target model.

[0106] Optionally, in the text processing device based on the deep learning model provided in Example 2 of the present application, the above-mentioned first model includes a router module, and the expansion unit also includes: a third determination subunit, which is used to determine the weight value of each first parameter matrix in the N first parameter matrices through the router module after determining the N first parameter matrices according to a preset numerical matrix; and a second calculation subunit, which is used to calculate the output results of the N first network structures according to the weight value of each first parameter matrix when detecting that the first data is input into the N first parameter matrices corresponding to the N first network structures.

[0107] Optionally, in the text processing device based on the deep learning model provided in Example 2 of the present application, the above-mentioned expansion unit also includes: an acquisition subunit, which is used to obtain the output results of the M second network structures after determining M second network structures based on the second network structure, when it is detected that the second data is input into the M second network structures; a third calculation subunit, which is used to input the output results of the M second network structures into a preset mapping matrix, and calculate the loss function value of each second network structure in the M second network structures; a third update subunit, which is used to update the network parameters in the expanded first model according to the loss function value of each second network structure.

[0108] Optionally, in the text processing device based on the deep learning model provided in Example 2 of the present application, the above-mentioned first computing subunit includes: a construction module, used to obtain training samples and construct a calibration set based on the training samples; an acquisition module, used to input the calibration set into the N first parameter matrices in the second network structure for each of the M second network structures in the second model, and obtain the weight value of each first parameter matrix in the N first parameter matrices; a determination module, used to determine the number of pruning based on the weight value of each first parameter matrix in each second network structure, wherein the number of pruning is an integer greater than a preset value.

[0109] Optionally, in the text processing device based on the deep learning model provided in Example 2 of the present application, the above-mentioned compression unit also includes: a fourth calculation subunit, used to prune the N first parameter matrices in the second model according to the pruning number to obtain the fourth model, and then calculate the weight value of each first parameter matrix in the fourth model, wherein the sum of the weight values ​​of each first parameter matrix is ​​a preset value; a fourth updating subunit, used to train each first parameter matrix in the fourth model and the weight value of each first parameter matrix using training samples to update each first parameter matrix in the fourth model.

[0110] The text processing device based on the deep learning model includes a processor and a memory. The above-mentioned receiving unit 501, generating unit 502 and responding unit 503 are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0111] The processor contains a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the efficiency of the deep learning model in responding to user requests can be improved by adjusting the kernel parameters.

[0112] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0113] Embodiment 3 of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements a text processing method based on a deep learning model.

[0114] Embodiment 4 of the present invention provides a processor, which is used to run a program, wherein the program executes a text processing method based on a deep learning model when running.

[0115] like Figure 6As shown, embodiment 5 of the present invention provides an electronic device, the device includes a processor, a memory, and a program stored in the memory and executable on the processor, and the processor implements the following steps when executing the program: receiving a first text input by a target user; inputting the first text into a target model, and generating a target text for replying to the first text through the target model, wherein the target model is a model that is pre-trained during model training, the network parameters of the model are expanded after pre-training, and the network parameters of the model are pruned during model inference; and the target text is displayed through a target device in response to the first text input by the target user.

[0116] When the processor executes the program, the following steps are also implemented: the above-mentioned target model is trained by the following steps: pre-training the preset model with a preset data set until the model converges to obtain a first model, wherein the preset data set is a data set constructed based on the user's historical question and answer text; expanding the network parameters of the first model by copying the network parameters, and training the expanded first model to obtain a second model; calculating the occupied space of the model based on the device information of the target device to obtain the target value; compressing the network parameters of the second model based on the target value to obtain the target model.

[0117] When the processor executes the program, the following steps are also implemented: the network parameters of the first model are expanded by copying the network parameters, including: determining N first parameter matrices based on a preset numerical matrix, wherein the preset numerical matrix is ​​a parameter matrix of a first network structure in the first model, and N is a positive integer; updating the first model based on the N first parameter matrices to obtain a third model; determining M second network structures based on the second network structure, wherein the second network structure is a network structure in the third model, and M is a positive integer; updating the third model based on the M second network structures to obtain the expanded first model.

[0118] When the processor executes the program, the following steps are also implemented: compressing the network parameters of the second model according to the target value to obtain the target model, including: calculating the pruning quantity according to the target value; pruning the N first parameter matrices in the second model according to the pruning quantity to obtain the fourth model; pruning the M second network structures in the fourth model according to a preset quantity to obtain the target model.

[0119] When the processor executes the program, the following steps are also implemented: the above-mentioned first model includes a router module, and after determining N first parameter matrices according to a preset numerical matrix, the above-mentioned method also includes: determining the weight value of each first parameter matrix in the N first parameter matrices through the router module; when detecting that the first data is input into the N first parameter matrices corresponding to the N first network structures, calculating the output results of the N first network structures according to the weight value of each first parameter matrix.

[0120] When the processor executes the program, the following steps are also implemented: after determining M second network structures based on the second network structure, the above method also includes: when detecting that the second data is input into the M second network structures, obtaining the output results of the M second network structures; inputting the output results of the M second network structures into a preset mapping matrix, and calculating the loss function value of each second network structure in the M second network structures; updating the network parameters in the expanded first model according to the loss function value of each second network structure.

[0121] When the processor executes the program, the following steps are also implemented: calculating the number of pruning based on the target value, including: obtaining training samples, and constructing a calibration set based on the training samples; for each of the M second network structures in the second model, inputting the calibration set into the N first parameter matrices in the second network structure, and obtaining the weight value of each first parameter matrix in the N first parameter matrices; determining the number of pruning based on the weight value of each first parameter matrix in each second network structure, wherein the number of pruning is an integer greater than a preset value.

[0122] When the processor executes the program, the following steps are also implemented: after pruning the N first parameter matrices in the second model according to the pruning amount to obtain the fourth model, the above method also includes: calculating the weight value of each first parameter matrix in the fourth model, wherein the sum of the weight values ​​of each first parameter matrix is ​​a preset value; and training each first parameter matrix in the fourth model and the weight value of each first parameter matrix using training samples to update each first parameter matrix in the fourth model.

[0123] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0124] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that is initialized with the following method steps: receiving a first text input by a target user; inputting the first text into a target model, and generating a target text for replying to the first text through the target model, wherein the target model is a model that is pre-trained during model training, the network parameters of the model are expanded after pre-training, and the network parameters of the model are pruned during model inference; and the target text is displayed through a target device in response to the first text input by the target user.

[0125] When executed on a data processing device, it is also suitable for executing a program initialized with the following method steps: the above-mentioned target model is trained by the following steps: pre-training the preset model with a preset data set until the model converges to obtain a first model, wherein the preset data set is a data set constructed based on the user's historical question and answer text; expanding the network parameters of the first model by copying the network parameters, and training the expanded first model to obtain a second model; calculating the occupied space of the model based on the device information of the target device to obtain the target value; compressing the network parameters of the second model based on the target value to obtain the target model.

[0126] When executed on a data processing device, it is also suitable for executing an initialized program having the following method steps: expanding the network parameters of the first model by copying the network parameters, including: determining N first parameter matrices based on a preset numerical matrix, wherein the preset numerical matrix is ​​a parameter matrix of a first network structure in the first model, and N is a positive integer; updating the first model based on the N first parameter matrices to obtain a third model; determining M second network structures based on the second network structure, wherein the second network structure is a network structure in the third model, and M is a positive integer; updating the third model based on the M second network structures to obtain the expanded first model.

[0127] When executed on a data processing device, it is also suitable for executing a program that initializes the following method steps: compressing the network parameters of the second model according to the target value to obtain the target model, including: calculating the pruning amount according to the target value; pruning the N first parameter matrices in the second model according to the pruning amount to obtain the fourth model; pruning the M second network structures in the fourth model according to a preset number to obtain the target model.

[0128] When executed on a data processing device, it is also suitable for executing an initialized program having the following method steps: the above-mentioned first model includes a router module, and after determining N first parameter matrices according to a preset numerical matrix, the above-mentioned method also includes: determining the weight value of each first parameter matrix in the N first parameter matrices through the router module; when detecting that the first data is input into the N first parameter matrices corresponding to the N first network structures, calculating the output results of the N first network structures according to the weight value of each first parameter matrix.

[0129] When executed on a data processing device, it is also suitable for executing a program initialized with the following method steps: after determining M second network structures based on the second network structure, the above method also includes: when detecting that the second data is input into the M second network structures, obtaining the output results of the M second network structures; inputting the output results of the M second network structures into a preset mapping matrix, and calculating the loss function value of each second network structure in the M second network structures; updating the network parameters in the expanded first model according to the loss function value of each second network structure.

[0130] When executed on a data processing device, it is also suitable for executing an initialized program having the following method steps: calculating the number of pruning according to a target value, including: obtaining training samples, and constructing a calibration set according to the training samples; for each of the M second network structures in the second model, inputting the calibration set into the N first parameter matrices in the second network structure, and obtaining the weight value of each first parameter matrix in the N first parameter matrices; determining the number of pruning according to the weight value of each first parameter matrix in each second network structure, wherein the number of pruning is an integer greater than a preset value.

[0131] When executed on a data processing device, it is also suitable for executing a program initialized with the following method steps: after pruning the N first parameter matrices in the second model according to the pruning amount to obtain a fourth model, the above method also includes: calculating the weight value of each first parameter matrix in the fourth model, wherein the sum of the weight values ​​of each first parameter matrix is ​​a preset value; and training each first parameter matrix in the fourth model and the weight value of each first parameter matrix using training samples to update each first parameter matrix in the fourth model.

[0132] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0133] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0134] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0135] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0136] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0137] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0138] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0139] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0140] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0141] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A text processing method based on a deep learning model, characterized in that: include: receiving a first text input by a target user; Inputting the first text into a target model, and generating a target text for replying to the first text through the target model, wherein the target model is a model that is pre-trained during model training, network parameters of the model are expanded after pre-training, and network parameters of the model are pruned during model inference; The target text is displayed by a target device in response to the first text input by the target user.

2. The method according to claim 1, characterized in that The target model is trained by the following steps: Pre-training a preset model using a preset data set until the model converges to obtain a first model, wherein the preset data set is a data set constructed based on the user's historical question and answer texts; Expanding the network parameters of the first model by copying the network parameters, and training the expanded first model to obtain a second model; Calculate the occupied space of the model according to the device information of the target device to obtain the target value; The network parameters of the second model are compressed according to the target values ​​to obtain the target model.

3. The method according to claim 2, characterized in that Expanding the network parameters of the first model by copying the network parameters includes: Determine N first parameter matrices according to a preset numerical matrix, wherein the preset numerical matrix is ​​a parameter matrix of the first network structure in the first model, and N is a positive integer; Updating the first model according to the N first parameter matrices to obtain a third model; Determine M second network structures according to the second network structure, wherein the second network structure is the network structure in the third model, and M is a positive integer; The third model is updated according to the M second network structures to obtain the expanded first model.

4. The method according to claim 3, characterized in that Compressing the network parameters of the second model according to the target value to obtain the target model includes: Calculating the number of pruning branches according to the target value; Pruning the N first parameter matrices in the second model according to the pruning quantity to obtain a fourth model; The M second network structures in the fourth model are pruned according to a preset number to obtain the target model.

5. The method according to claim 3, characterized in that: The first model includes a router module. After determining N first parameter matrices according to a preset numerical matrix, the method further includes: Determine, by the router module, a weight value of each of the N first parameter matrices; When it is detected that the first data is input into the N first network structures corresponding to the N first parameter matrices, the output results of the N first network structures are calculated according to the weight value of each first parameter matrix.

6. The method according to claim 3, characterized in that After determining the M second network structures according to the second network structure, the method further includes: When it is detected that the second data is input into the M second network structures, obtaining output results of the M second network structures; Input the output results of the M second network structures into a preset mapping matrix, and calculate the loss function value of each second network structure in the M second network structures; The network parameters in the expanded first model are updated according to the loss function value of each second network structure.

7. The method according to claim 4, characterized in that The number of pruning branches is calculated according to the target value, including: Acquire training samples, and construct a calibration set based on the training samples; For each of the M second network structures in the second model, input the calibration set into the N first parameter matrices in the second network structure, and obtain a weight value of each first parameter matrix in the N first parameter matrices; The pruning amount is determined according to the weight value of each first parameter matrix in each second network structure, wherein the pruning amount is an integer greater than a preset value.

8. The method according to claim 4, characterized in that After pruning the N first parameter matrices in the second model according to the pruning amount to obtain a fourth model, the method further includes: Calculating a weight value of each first parameter matrix in the fourth model, wherein the sum of the weight values ​​of each first parameter matrix is ​​a preset value; Each first parameter matrix and the weight value of each first parameter matrix in the fourth model are trained using training samples to update each first parameter matrix in the fourth model.

9. A text processing device based on a deep learning model, characterized in that: include: A receiving unit, configured to receive a first text input by a target user; A generating unit, configured to input the first text into a target model, and generate a target text for replying to the first text through the target model, wherein the target model is a model that is pre-trained during model training, network parameters of the model are expanded after pre-training, and network parameters of the model are pruned during model inference; The response unit is used to display the target text through the target device in response to the first text input by the target user.

10. An electronic device, characterized in that: It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the text processing method based on the deep learning model as described in any one of claims 1 to 8.

Images