Text processing method and device

Abstract

The present application provides a text processing method and device. The method includes: obtaining a text to be processed; inputting the text to be processed into a text recognition model for processing, and obtaining a recognition result of the text recognition model on the text to be processed. The text recognition model is obtained by performing knowledge distillation processing on a pre-trained model for text processing based on labeled training data and a guiding task corresponding to text recognition, and performing knowledge distillation processing again based on the knowledge distillation processing result and unlabeled training data. Since the text recognition model in this method uses more data training than the model obtained only based on training data, the text processing method recognizes text more accurately. In addition, since the text recognition model used by the text processing method is obtained through knowledge distillation, the text processing method is easier to apply.

Description

Background Art

[0002] In many text recognition tasks such as natural language inference, named entity recognition, and reading comprehension, pre-trained models for text processing can be applied to handle the above tasks.

[0003] However, existing pre-trained models such as ELMO (Embeddings Language Models) are huge in size. The huge model size makes it time-consuming to apply pre-trained models for text processing and difficult to apply in actual scenarios. Therefore, it is very important to find an accurate and easy-to-apply text recognition method. Summary of the invention

[0004] The present application aims to provide a text processing method and device, which can accurately and easily perform text recognition.

[0005] According to one aspect of an embodiment of the present application, a text processing method is provided, including: obtaining a text to be processed; inputting the text to be processed into a text recognition model for processing to obtain a recognition result of the text to be processed by the text recognition model, wherein the text recognition model is obtained by performing knowledge distillation processing on a pre-trained model for text processing based on labeled training data and a guiding task corresponding to text recognition, and performing knowledge distillation processing again based on the knowledge distillation processing result and unlabeled training data.

[0006] According to one aspect of an embodiment of the present application, a text processing device is provided, including: an acquisition module, used to acquire a text to be processed; a processing module, used to input the text to be processed into a text recognition model for processing, and obtain a recognition result of the text to be processed by the text recognition model, wherein the text recognition model is based on labeled training data and a guiding task corresponding to text recognition, and a pre-trained model for text processing is subjected to knowledge distillation processing, and the knowledge distillation processing is performed again based on the knowledge distillation processing result and unlabeled training data.

[0007] In some embodiments of the present application, based on the aforementioned scheme, the text processing device also includes: a model generation module, which is used to perform knowledge distillation processing on the pre-trained model based on the labeled training data and the guiding task to obtain a guiding model; generate a distance function corresponding to the guiding task according to the correspondence between the output of the pre-trained model and the output of the guiding model; perform knowledge distillation processing on the guiding model according to the unlabeled training data and the distance function to obtain a pre-recognition model, so as to generate the text recognition model according to the pre-recognition model.

[0008] In some embodiments of the present application, based on the aforementioned scheme, the model generation module is configured to: generate a universal function according to the correspondence between the output of the guidance model and the output of the pre-recognition model; and generate the text recognition model according to the universal function.

[0009] In some embodiments of the present application, based on the aforementioned scheme, the model generation module is further configured to: obtain general parameters of the pre-recognition model according to the general function; initialize the general parameters; and obtain the text recognition model according to the initialized general parameters.

[0010] In some embodiments of the present application, based on the aforementioned scheme, the model generation module is further configured to: establish a general model according to the initialized general parameters; perform knowledge distillation processing on the general model based on the training data corresponding to the specified text recognition task to obtain the text recognition model.

[0011] In some embodiments of the present application, based on the aforementioned scheme, the model generation module is also configured to: train the pre-trained model based on training data corresponding to a specified text recognition task; perform knowledge distillation processing on the training results based on the training data corresponding to the specified text recognition task to obtain a specific model; generate a specific function based on the correspondence between the output of the pre-trained model and the output of the specific model; and generate the text recognition model based on the general function and the specific function.

[0012] In some embodiments of the present application, based on the aforementioned scheme, the model generation module is also configured to: obtain general parameters of the pre-recognition model according to the general function, and initialize the general parameters; obtain specific parameters of the specific model according to the specific function, and initialize the specific parameters; obtain the text recognition model according to the initialized general parameters and the initialized specific parameters.

[0013] In some embodiments of the present application, based on the aforementioned scheme, the model generation module is also configured to: obtain the shared parameters of the text recognition model according to the initialized general parameters; obtain the task parameters of the text recognition model according to the initialized specific parameters; and establish the text recognition model according to the shared parameters and the task parameters.

[0014] In some embodiments of the present application, based on the aforementioned solution, the model generation module is further configured to: perform linear processing on the general function and the specific function to obtain the text recognition model.

[0015] According to one aspect of an embodiment of the present application, a computer-readable program medium is provided, which stores computer program instructions. When the computer program instructions are executed by a computer, the computer executes any of the above methods.

[0016] According to one aspect of an embodiment of the present application, an electronic device is provided, including: a processor; a memory, wherein computer-readable instructions are stored in the memory, and when the computer-readable instructions are executed by the processor, the method described in any one of the above items is implemented.

[0017] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0018] In the technical solutions provided by some embodiments of the present application, by obtaining the text to be processed, the text to be processed is input into the text recognition model for processing, and the recognition result of the text recognition model on the text to be processed is obtained. Among them, the text recognition model is obtained by using a semi-supervised learning method to perform knowledge distillation on a pre-trained model for text processing. First, based on the labeled training data and the guiding task corresponding to the text recognition, the pre-trained model for text processing is subjected to knowledge distillation processing, and the processing result of the knowledge distillation can guide the unlabeled data to perform knowledge distillation; secondly, the text recognition model is obtained by performing knowledge distillation processing again according to the knowledge distillation processing result and the unlabeled training data. Since the text recognition model is obtained by training data and unlabeled data, more data training is used compared to the model obtained only from the training data, so the text recognition model is more accurate. In addition, since the text recognition model is obtained by knowledge distillation, the size of the text recognition model is smaller than that of the existing pre-trained model for text processing, and it is more convenient to apply. Based on the text recognition model, the text to be processed can be recognized, and the processing result of the text to be processed can be obtained more quickly and accurately. Therefore, the text processing method can accurately and conveniently perform text recognition.

[0019] It should be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings herein are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the description, are used to explain the principles of the present application.

[0021] Figure 1A A schematic diagram showing an exemplary system architecture to which the technical solution of the embodiments of the present application can be applied;

[0022] Figure 1B A schematic diagram showing a data sharing system to which the technical solution of an embodiment of the present application can be applied;

[0023] Figure 1C A schematic diagram showing a node storing program behavior data where a blockchain according to an embodiment of the present application can be applied;

[0024] Figure 2 A flowchart of a text processing method according to an embodiment of the present application is schematically shown;

[0025] Figure 3 A flowchart schematically shows a process of establishing a text recognition model in the text processing method according to an embodiment of the present application;

[0026] Figure 4 A schematic diagram schematically shows a process of generating a text recognition model based on a pre-recognition model according to an embodiment of the present application;

[0027] Figure 5 A flowchart of a process of generating a text recognition model according to a universal function according to an embodiment of the present application is schematically shown;

[0028] Figure 6 A flowchart schematically shows a process of obtaining a specific function according to an embodiment of the present application;

[0029] Figure 7 A flowchart schematically illustrates a process of generating a text recognition model according to a general function and a specific function according to an embodiment of the present application;

[0030] Figure 8 A block diagram of a text processing device according to an embodiment of the present application is schematically shown;

[0031] Fig. 9 The figure is a hardware diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0032] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0033] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0034] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0035] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0036] Figure 1A A schematic diagram of an exemplary system architecture 100A to which the technical solution of the embodiments of the present application can be applied is shown.

[0037] like Figure 1A As shown, the system architecture 100A may include a terminal device 101A (the terminal device may be one or more of a smart phone, a tablet computer, a portable computer, and a desktop computer), a network 102A, and a server 103A. The network 102A is used to provide a medium for a communication link between the terminal device 101A and the server 103A. The network 102A may include various connection types, such as a wired communication link, a wireless communication link, and the like.

[0038] It should be understood that Figure 1A The number of terminal devices 101A, networks 102A and servers 103A in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices 101A, networks 102A and servers 103A. For example, the server 103A may be a server cluster composed of multiple servers.

[0039] In one embodiment of the present application, the server 103A obtains the text to be processed, inputs the text to be processed into the text recognition model for processing, and obtains the recognition result of the text recognition model on the text to be processed. Among them, the text recognition model is obtained by using a semi-supervised learning method to perform knowledge distillation on a pre-trained model for text processing. First, the pre-trained model for text processing is subjected to knowledge distillation based on labeled training data and a guiding task corresponding to text recognition, and the processing result of knowledge distillation can guide the knowledge distillation of unlabeled data; secondly, the text recognition model is obtained by performing knowledge distillation again based on the knowledge distillation processing result and the unlabeled training data. Since the text recognition model is obtained through training data and unlabeled data, more data training is used compared to the model obtained only based on training data, so the text recognition model is more accurate. In addition, since the text recognition model is obtained through knowledge distillation, the size of the text recognition model is smaller than that of the existing pre-trained model for text processing, and it is more convenient to apply. Based on the text recognition model, the text to be processed can be recognized, and the processing result of the text to be processed can be obtained more quickly and accurately. Therefore, the text processing method can accurately and conveniently perform text recognition.

[0040] It should be noted that the text processing method provided in the embodiment of the present application is generally executed by the server 103A, and accordingly, the text processing device is generally set in the server 103A. However, in other embodiments of the present application, the terminal device 101A may also have similar functions as the server 103A, thereby executing the text processing method provided in the embodiment of the present application.

[0041] Figure 1B A schematic diagram of an exemplary data sharing system 100B to which the technical solution of an embodiment of the present invention can be applied is shown.

[0042] See also Figure 1BThe data sharing system 100B shown, the data sharing system 100B refers to a system for sharing data between nodes, and the data sharing system may include multiple nodes 101B, multiple nodes 101B may refer to multiple game terminal devices 101A in the data sharing system, and multiple nodes 101B may also refer to game servers 103A in the data sharing system. Each node 101B can receive input information when performing normal work, and maintain the shared data in the data sharing system based on the received input information. In order to ensure the information intercommunication in the data sharing system, there may be an information connection between each node in the data sharing system, and the nodes can transmit information through the above-mentioned information connection. For example, when any node in the data sharing system receives input information, other nodes in the data sharing system obtain the input information according to the consensus algorithm, and store the input information as data in the shared data, so that the data stored on all nodes in the data sharing system are consistent.

[0043] Each node in the data sharing system has a corresponding node identifier, and each node in the data sharing system can store the node identifiers of other nodes in the data sharing system, so that the generated blocks can be broadcast to other nodes in the data sharing system according to the node identifiers of other nodes. A node identifier list as shown in the following table can be maintained in each node, and the node name and node identifier are stored in the node identifier list accordingly. Among them, the node identifier can be an IP (Internet Protocol, a protocol for interconnecting networks) address and any other information that can be used to identify the node. Table 1 only takes the IP address as an example for illustration.

[0044] Node Name Node ID Node 1 117.114.151.174 Node 2 117.116.189.145 … … Node N 119.123.789.258

[0045] Table 1

[0046] Each node in the data sharing system stores the same blockchain. The blockchain consists of multiple blocks, see Figure 1B The blockchain consists of multiple blocks. The genesis block includes a block header and a block body. The block header stores the input information characteristic value, version number, timestamp and difficulty value, and the block body stores the input information; the next block of the genesis block takes the genesis block as the parent block, and the next block also includes a block header and a block body. The block header stores the input information characteristic value of the current block, the block header characteristic value, version number, timestamp and difficulty value of the parent block, and so on, so that the block data stored in each block in the blockchain is associated with the block data stored in the parent block, ensuring the security of the input information in the block.

[0047] When generating each block in the blockchain, see Figure 1CWhen the node where the blockchain is located receives the input information, it verifies the input information. After the verification is completed, the input information is stored in the memory pool and the hash tree used to record the input information is updated. After that, the update timestamp is updated to the time when the input information is received, and different random numbers are tried to calculate the eigenvalue multiple times so that the calculated eigenvalue can satisfy the following formula:

[0048] SHA256(SHA256(version+prev_hash+merkle_root+ntime+nbits+x)) <TARGET

[0049] Among them, SHA256 is the eigenvalue algorithm used to calculate the eigenvalue; version (version number) is the version information of the relevant block protocol in the blockchain; prev_hash is the block header eigenvalue of the parent block of the current block; merkle_root is the eigenvalue of the input information; ntime is the update time of the update timestamp; nbits is the current difficulty, which is a fixed value within a period of time and is determined again after exceeding the fixed time period; x is a random number; TARGET is the eigenvalue threshold, which can be determined based on nbits.

[0050] In this way, when the random number that satisfies the above formula is calculated, the information can be stored accordingly, the block header and block body can be generated, and the current block can be obtained. Subsequently, the node where the blockchain is located sends the newly generated block to other nodes in the data sharing system according to the node identification of other nodes in the data sharing system. Other nodes verify the newly generated block and add the newly generated block to the blockchain stored in them after the verification is completed.

[0051] The input information stored in the data sharing system 100B may be labeled training data, unlabeled training data, etc.

[0052] The implementation details of the technical solution of the embodiment of the present application are described in detail below:

[0053] Figure 2 The flowchart of the text processing method according to an embodiment of the present application is schematically shown. The execution subject of the text processing method may be a server, for example, Figure 1A Server 103A shown in .

[0054] Reference Figure 2 As shown, the text processing method at least includes steps S210 to S220, which are described in detail as follows:

[0055] In step S210, the text to be processed is obtained.

[0056] In one embodiment of the present application, the text to be processed may be a language text, which may be a text language text or a voice language text.

[0057] In one embodiment of the present application, the text to be processed may be an image text, an audio text, or a video text.

[0058] In step S220, the text to be processed is input into the text recognition model for processing to obtain the recognition result of the text to be processed by the text recognition model. The text recognition model is obtained by performing knowledge distillation processing on a pre-trained model used for text processing based on labeled training data and a guiding task corresponding to text recognition, and performing knowledge distillation processing again based on the knowledge distillation processing result and the unlabeled training data.

[0059] In one embodiment of the present application, the text to be processed may be a language text, and the text recognition model may perform semantic recognition on the text to be processed, thereby identifying the instruction corresponding to the text to be processed so that the instruction can be executed.

[0060] In one embodiment of the present application, the guiding task corresponding to the text recognition may be a designated text recognition task.

[0061] In one embodiment of the present application, the labeled training data may be training data corresponding to a specified text recognition task.

[0062] Using a specified text recognition task as a guide task and using the training data corresponding to the specified text recognition task can accurately provide guidance for the training of unlabeled data.

[0063] In one embodiment of the present application, the guiding task corresponding to the text recognition may be a task similar to the designated text recognition task.

[0064] In one embodiment of the present application, the labeled training data may be training data corresponding to a task similar to a specified text recognition task.

[0065] Adopting tasks similar to the specified text recognition task and using training data corresponding to tasks similar to the specified text recognition task can avoid data coupling while providing guidance for the training of unlabeled data.

[0066] In one embodiment of the present application, an execution object of a specified text recognition task may be obtained, and a text recognition task that is the same as the execution object of the specified text recognition task may be regarded as a task similar to the specified text recognition task.

[0067] In one embodiment of the present application, the specified text recognition task may be to recognize the time in the text. For example, the text to be processed may be a voice command "What time is it now?" received by the terminal device 101A. The server 103A may confirm that the voice command is related to the clock based on the keyword "what time" contained in the voice command. Tasks similar to the specified text recognition task may be tasks related to the clock, such as turning off the alarm clock and setting the alarm clock.

[0068] In one embodiment of the present application, the training data corresponding to the task similar to the specified text recognition task is the commonly used text when performing the task similar to the specified text recognition task. The commonly used text may be the text used in the historical task execution record.

[0069] In one embodiment of the present application, the unlabeled training data is large-scale unlabeled data, and the unlabeled training data may be words in Wikipedia.

[0070] In one embodiment of the present application, the pre-trained model used for text processing may be a pre-trained model such as ELMO (Embeddings Language Models), BERT (Bidirectional Encoder Representations from Transfoemers), or openAI GPT (Generative Pre-trained Transformer).

[0071] exist Figure 2In the illustrated embodiment, by obtaining the text to be processed, the text to be processed is input into the text recognition model for processing, and the recognition result of the text to be processed by the text recognition model is obtained. Among them, the text recognition model is obtained by using a semi-supervised learning method to perform knowledge distillation on a pre-trained model for text processing. First, the pre-trained model for text processing is subjected to knowledge distillation based on labeled training data and a guiding task corresponding to text recognition, and the processing result of knowledge distillation can guide the knowledge distillation of unlabeled data; secondly, the text recognition model is obtained by performing knowledge distillation again based on the knowledge distillation processing result and the unlabeled training data. Since the text recognition model is obtained through training data and unlabeled data, more data training is used compared to the model obtained only based on training data, so the text recognition model is more accurate. In addition, since the text recognition model is obtained through knowledge distillation, the size of the text recognition model is smaller than that of the existing pre-trained model for text processing, and it is more convenient to apply. Based on the text recognition model, the text to be processed can be recognized, and the processing result of the text to be processed can be obtained more quickly and accurately. Therefore, the text processing method can accurately and conveniently perform text recognition.

[0072] In one embodiment of the present application, Figure 2 Before step S220 shown in the figure, that is, before the text to be processed is input into the text recognition model for processing, the text processing method of the present application can be performed as follows Figure 3 Steps S310 to S330 shown in the figure establish a text recognition model in the text processing method:

[0073] In step S310, knowledge distillation is performed on the pre-trained model based on the labeled training data and the guidance task to obtain a guidance model.

[0074] In step S320, a distance function corresponding to the guidance task is generated according to the corresponding relationship between the output of the pre-trained model and the output of the guidance model.

[0075] In one embodiment of the present application, the labeled training data can be input into the pre-trained model and the guided model respectively to obtain the output of the pre-trained model and the output of the guided model, and then a distance function corresponding to the guided task is generated based on the correspondence between the output of the pre-trained model and the output of the guided model.

[0076] In one embodiment of the present application, the distance function corresponding to the guidance task may be a loss function between the pre-trained model and the guidance model.

[0077] In one embodiment of the present application, the loss function is formed by a linear combination of a sharing function and a task function, wherein the sharing function part reflects the common characteristics of natural text in the data used for knowledge distillation, and the task function part reflects the task characteristics corresponding to the data used for knowledge distillation. Since the loss function between the pre-trained model and the guided model is obtained based on labeled data, the task function part in the loss function can better reflect the characteristics of the guided task corresponding to the labeled data, so the loss function can guide unlabeled data for knowledge distillation.

[0078] Continue to refer to Figure 3 In step S330, the guide model is subjected to knowledge distillation processing according to the unlabeled training data and the distance function to obtain a pre-recognition model, so as to generate a text recognition model according to the pre-recognition model.

[0079] In one embodiment of the present application, the pre-recognition model may be used as a text recognition model.

[0080] exist Figure 3 In the illustrated embodiment, the pre-trained model is subjected to knowledge distillation based on the labeled training data and the guided task to obtain the guided model. The guided model can guide the unlabeled data to further perform knowledge distillation on the guided model, which increases the training data of the guided model, so that the obtained guided model can better express the characteristics of the guided task while also better expressing the general characteristics of the text, so that the guided model can more accurately perform text recognition. On the other hand, since the guided model is obtained through knowledge distillation, the guided model has fewer parameters and is easy to apply.

[0081] In one embodiment of the present application, in step S330, the process of generating a text recognition model according to the pre-recognition model may include: Figure 4 Steps S410 to S420 shown:

[0082] In step S410, a universal function is generated according to the corresponding relationship between the output of the guidance model and the output of the pre-recognition model.

[0083] In one embodiment of the present application, unlabeled data may be input into a guidance model and a pre-recognition model, and a universal function may be generated according to the corresponding relationship between the guidance model and the pre-recognition model and the output of the unlabeled data.

[0084] In one embodiment of the present application, the universal function may be a loss function between the guidance model and the pre-recognition model.

[0085] In one embodiment of the present application, since the loss function between the guidance model and the pre-recognition model is obtained based on unlabeled data, the shared function part in the loss function can better reflect the general characteristics of the text.

[0086] In step S420, a text recognition model is generated according to the general function.

[0087] In one embodiment of the present application, since the universal function is obtained by performing knowledge distillation on the guidance model using unlabeled data, the text recognition model generated according to the universal function can reflect both the universal characteristics of the text in the universal function and the characteristics of the guidance task in the guidance function, so the text recognition model can accurately perform text recognition. On the other hand, since the text recognition model is obtained through knowledge distillation, the text recognition model has fewer parameters and is easy to apply.

[0088] In one embodiment of the present application, in step S420, the process of generating a text recognition model according to the general function may include: Figure 5 Steps S510 to S530 shown:

[0089] In step S510, general parameters of the pre-recognition model are obtained according to the general function.

[0090] In one embodiment of the present application, the universal parameters of the pre-recognition model may be obtained by finding the minimum value of the universal function.

[0091] In step S520, general parameters are initialized.

[0092] In one embodiment of the present application, in the pre-recognition model obtained by knowledge distillation, the common parameters of the pre-recognition model include shared parameters and task parameters. The shared parameters carry the common characteristics of the data used when performing knowledge distillation on the pre-recognition model, and the task parameters reflect the task characteristics corresponding to the data used in knowledge distillation. The task parameters in the common parameters can be randomly initialized, and the shared parameters in the common parameters can be retained.

[0093] In step S530, a text recognition model is obtained according to the initialized general parameters.

[0094] In this embodiment, the obtained text recognition model can be enabled to better learn the characteristics of the task corresponding to the data used to train the model established according to the general parameters.

[0095] In one embodiment of the present application, the task parameters in the general parameters can be randomly initialized, including the shared parameters in the general parameters, so that the obtained text recognition model can better learn the characteristics of the task corresponding to the data used to train the general model established according to the general parameters while retaining the general characteristics of the text carried in the general parameters.

[0096] In one embodiment of the present application, a general model can be established according to the initialized general parameters, and the general model can be subjected to knowledge distillation processing based on the training data corresponding to the specified text recognition task to obtain a text recognition model.

[0097] In this embodiment, since the text recognition model obtained according to the initialized general parameters can better learn the characteristics of the task corresponding to the data used to train the model established according to the general parameters, the general model is subjected to knowledge distillation processing using the training data corresponding to the specified text recognition task, and the obtained text recognition model can handle the specified text recognition task well.

[0098] In one embodiment of the present application, Figure 4 Before step S420, that is, before generating a text recognition model according to the general function, a specific function may be generated according to the training data corresponding to the specified text recognition task. Figure 4 In step S420 shown, a text recognition model is generated according to a general function, and the text recognition model may be generated according to a general function and a specific function.

[0099] In this embodiment, since the general function is obtained by performing knowledge distillation on the guidance model using unlabeled data, the text recognition model generated according to the general function can reflect the general characteristics of the text in the general function, and the specific function is obtained according to the training data corresponding to the specified text recognition task, and the specific function can reflect the characteristics of the specified text recognition task. Therefore, the obtained text recognition model can handle the specified text recognition task more accurately.

[0100] In one embodiment of the present application, Figure 6 The steps S610 to S630 shown obtain a specific function:

[0101] In step S610, a pre-trained model is trained based on training data corresponding to a specified text recognition task;

[0102] In step S620, knowledge distillation is performed on the training results based on the training data corresponding to the specified text recognition task to obtain a specific model;

[0103] In step S630, a specific function is generated according to the corresponding relationship between the output of the pre-trained model and the output of the specific model.

[0104] In this embodiment, since the specific function is obtained by performing knowledge distillation on the training results of a pre-trained model trained with training data corresponding to the specified text recognition task using training data corresponding to the specified text recognition task, the specific function can carry the characteristics of the training data corresponding to the specified text recognition task.

[0105] In one embodiment of the present application, training data corresponding to a specified text recognition task may be input into a pre-trained model and a specific model, and a specific function may be generated based on the correspondence between the output of the pre-trained model and the output of the specific model.

[0106] In one embodiment of the present application, the specific function may be a loss function between a pre-trained model and a specific model.

[0107] In one embodiment of the present application, the process of generating a text recognition model according to the general function and the specific function may include: Figure 7 Steps S710 to S730 shown:

[0108] In step S710, general parameters of the pre-recognition model are obtained according to the general function, and the general parameters are initialized.

[0109] In step S720, specific parameters of the specific model are obtained according to the specific function, and the specific parameters are initialized.

[0110] In one embodiment of the present application, the minimum value of a specific function may be obtained to obtain specific parameters of the pre-recognition model.

[0111] In one embodiment of the present application, in a specific model obtained by knowledge distillation, specific parameters include shared parameters and task parameters. The shared parameters carry the characteristics of the training data corresponding to the specified text recognition task used when performing knowledge distillation on the specific model. The task parameters reflect the characteristics of the specified text recognition task used during knowledge distillation. The shared parameters in the specific parameters can be randomly initialized, and the task parameters in the general parameters can be retained.

[0112] Continue to refer to Figure 7 In step S730, a text recognition model is obtained according to the initialized general parameters and the initialized specific parameters.

[0113] In this embodiment, the initialized general parameters can carry the general characteristics of the text in the general parameters, and the initialized specific parameters can carry the characteristics of the training data corresponding to the specified text recognition task. Therefore, the text recognition model obtained according to the initialized general parameters and the initialized specific parameters can accurately handle the specified text recognition task.

[0114] In one embodiment of the present application, the shared parameters of the text recognition model can be obtained according to the initialized general parameters, the task parameters of the text recognition model can be obtained according to the initialized specific parameters, and the text recognition model can be established according to the shared parameters and the task parameters.

[0115] In one embodiment of the present application, the text recognition model obtained by knowledge distillation includes multiple layers of training layers, and the multiple layers of training layers are divided into two parts, wherein the parameters of the training layers of one part represent the characteristics of the task processed by the text recognition model, and the parameters of the training layers of the other part represent the general characteristics of the text. Shared parameters are defined as the parameter part that represents the general characteristics of natural text, and task parameters are defined as the parameter part that represents the characteristics of the task processed by the text recognition model. The shared parameters of the processing model are obtained according to the initialized general function, and the shared parameters can well represent the general characteristics of the text; the task parameters of the text recognition model are obtained according to the initialized specific function, and the task parameters can well represent the characteristics of the specified text recognition task. The text recognition model generated according to the shared parameters and task parameters can well represent both the general characteristics of the text and the characteristics of the specified text recognition task, and can accurately process the specified text recognition task.

[0116] In one embodiment of the present application, the general function can be expressed by the formula Lw=βLu 2 +(1-β)Lt 2 Represents, where Lw represents the universal function obtained by using unlabeled data for knowledge distillation, Lu 2 represents the shared function in the universal function, β represents the weight of the shared function, Lt 2 represents the task function in the general function, and (1-β) represents the weight of the task function. The specific function can be obtained by the formula Lr = αLu 1 +(1-α)Lt 1 Represents, where Lr represents the specific function obtained by knowledge distillation based on labeled training data, Lu 1 represents the shared function in a specific function, α represents the weight of the shared function, Lt 1 represents the task function in the specific function, and (1-α) represents the weight of the task function. The initialized general function can be obtained by the formula Lcw = Lu 2 It means that the specific function after initialization can be expressed by the formula Lcr = Lt 1 It means that the shared parameters can be obtained by taking the minimum value of the initialized universal function, and the task parameters can be obtained by taking the minimum value of the initialized specific function. Multiple training layers are established according to the shared parameters and the task parameters, respectively, to obtain a text recognition model for processing the specified text recognition task.

[0117] In one embodiment of the present application, a text recognition model is obtained based on a general function and a specific function. The general function and the specific function are linearly processed to obtain the text recognition model. The established text recognition model can not only well express the general characteristics of the text, but also well express the characteristics of the specified text recognition task, and can accurately handle the specified text recognition task.

[0118] In one embodiment of the present application, the specific function and the universal function can be linearly added according to the weight, and the obtained text recognition model can process the specified text recognition task more accurately, and the text weight can be set as needed.

[0119] In one embodiment of the present application, the ELMO model, the openAI GPT model, the BERT-12 model with 12 hidden layers, and the BERT-6 model with 6 hidden layers obtained by performing knowledge distillation on the BERT model with 12 hidden layers are used to process the text to be processed. Then, the text recognition model generated by using the BERT model with 12 hidden layers as the pre-training model of the present application is used to process the same text to be processed, and multiple text data sets to be processed are processed. The accuracy of the processing results is shown in Table 2:

[0120]

[0121] Table 2

[0122] According to Table 2, the average accuracy of each method for processing data sets 1-7 is calculated. It can be seen that the average accuracy of the processing results using the ELMO model is 75; the average accuracy of the processing results using the openAI GPT model is 76; the average accuracy of the processing results using the BERT-12 model is 77; the average accuracy of the processing results using the BERT-6 model is 78; the average accuracy of the processing results using the processing method of the present application is 79. It can be seen that the processing method of the present application has the highest accuracy among these processing methods. At the same time, the processing model of the present application has the least number of parameters. Therefore, the application of the processing method of the present application to process text recognition tasks can be faster and more accurate than other processing methods.

[0123] In one embodiment of the present application, the ELMO model, the openAI GPT model, the BERT-12 model with 12 hidden layers, and the BERT-6 model with 6 hidden layers obtained by performing knowledge distillation on the BERT model with 12 hidden layers are used to process the text to be processed. Then, the text recognition model generated by the BERT model with 12 hidden layers as the pre-training model of the present application is used to process the same text to be processed, and multiple text data sets to be processed are processed, and the F1 value of the processing result is represented by Table 3:

[0124]

[0125]

[0126] Table 3

[0127] According to Table 3, the F1 values ​​obtained by processing data sets 1, 4, 5, and 7 by each method are averaged. It can be seen that the average F1 value of the results obtained by the method using the ELMO model is 77.4; the average F1 value of the results obtained by the method using the openAI GPT model is 81.4; the average F1 value of the results obtained by the method using the BERT-12 model is 85.8; the average F1 value of the results obtained by the method using the BERT-6 model is 83.05; the F1 value of the results obtained by applying the processing method of the present application is 85.3. It can be seen that the F1 value of the results obtained by the method used for the BERT-12 model is the highest among the four models, and the F1 value of the results obtained by applying the processing method of the present application is the second. Furthermore, by comparing the parameter amounts of the BERT-12 model and the text recognition model of the present application, the parameter amount of the text recognition model of the present application is less than that of the BERT-12 model. Therefore, applying the processing method of the present application to process text recognition tasks can be faster than applying the BERT-12 model to process text tasks. At the same time, the F1 value of the result obtained by applying the processing method of the present application is also higher. The F1 value is the harmonic mean of the precision and the recall rate, and also represents the accuracy of processing text recognition tasks. Therefore, the processing method of the present application can process text recognition tasks faster than other methods while ensuring accurate processing of text recognition tasks.

[0128] In one embodiment of the present application, the text processing method can be applied to various text recognition tasks such as natural language text reasoning, named entity recognition, and reading comprehension. Taking the application of the text processing method in Tencent Xiaowei's dialogue system as an example, Tencent Xiaowei's dialogue system uses the text processing method of the present application to process the text to be processed, which can identify the user's intention faster and more accurately than using other text processing methods to process the text to be processed, bringing a better experience to the user.

[0129] In one embodiment of the present application, the Tencent Xiaowei speaker receives a user's voice command "Please wake me up at 7:30 tomorrow morning", and recognizes that the designated text recognition task corresponding to the voice command is to recognize the time based on the keywords such as "7:30" and "wake up" in the voice command, and sends the voice command to the text recognition model corresponding to the recognition time. The text recognition model corresponding to the recognition time is obtained by performing knowledge distillation processing on the ELMO model based on clock-related training data and clock-related tasks, and performing knowledge distillation processing again based on the knowledge distillation processing results and unlabeled training data. The text recognition model is used to perform semantic analysis on the voice command, and the analysis results are sent to the controller, so that the controller controls the clock module to set the alarm. The Tencent Xiaowei speaker using the text processing method of the present application successfully sets the alarm after 0.5 seconds, and the Tencent Xiaowei speaker using the ELMO model processing method successfully sets the alarm after 4 seconds. The text processing method of the present application greatly saves the user's time.

[0130] The following describes an apparatus embodiment of the present application, which can be used to execute the text processing method in the above-mentioned embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the above-mentioned embodiment of the text processing method of the present application.

[0131] Figure 8 The block diagram of a text processing device according to an embodiment of the present application is schematically shown.

[0132] Reference Figure 8 As shown, a text processing device 800 according to an embodiment of the present application includes an acquisition module 801 , a processing module 802 and a model generation module 803 .

[0133] In some embodiments of the present application, based on the aforementioned scheme, the acquisition module 801 is used to acquire the text to be processed; the processing module 802 is used to input the text to be processed into the text recognition model for processing, and obtain the recognition result of the text to be processed by the text recognition model. The text recognition model is based on labeled training data and a guiding task corresponding to text recognition. The pre-trained model used for text processing is subjected to knowledge distillation processing, and the knowledge distillation processing is performed again based on the knowledge distillation processing result and the unlabeled training data.

[0134] In some embodiments of the present application, based on the aforementioned scheme, the model generation module 803 is used to perform knowledge distillation processing on the pre-trained model based on labeled training data and the guidance task to obtain a guidance model; generate a distance function corresponding to the guidance task according to the correspondence between the output of the pre-trained model and the output of the guidance model; perform knowledge distillation processing on the guidance model according to the unlabeled training data and the distance function to obtain a pre-recognition model, so as to generate a text recognition model based on the pre-recognition model.

[0135] In some embodiments of the present application, based on the aforementioned scheme, the model generation module 803 is configured to: generate a universal function according to the correspondence between the output of the guidance model and the output of the pre-recognition model; and generate a text recognition model according to the universal function.

[0136] In some embodiments of the present application, based on the aforementioned scheme, the model generation module 803 is further configured to: obtain general parameters of the pre-recognition model according to a general function; initialize the general parameters; and obtain a text recognition model according to the initialized general parameters.

[0137] In some embodiments of the present application, based on the aforementioned scheme, the model generation module 803 is further configured to: establish a general model according to the initialized general parameters; perform knowledge distillation processing on the general model based on the training data corresponding to the specified text recognition task to obtain a text recognition model.

[0138] In some embodiments of the present application, based on the aforementioned scheme, the model generation module 803 is also configured to: train a pre-trained model based on training data corresponding to a specified text recognition task; perform knowledge distillation processing on the training results based on the training data corresponding to the specified text recognition task to obtain a specific model; generate a specific function based on the correspondence between the output of the pre-trained model and the output of the specific model; generate a text recognition model based on the general function and the specific function.

[0139] In some embodiments of the present application, based on the aforementioned scheme, the model generation module 803 is also configured to: obtain general parameters of the pre-recognition model according to a general function and initialize the general parameters; obtain specific parameters of the specific model according to a specific function and initialize the specific parameters; obtain the text recognition model according to the initialized general parameters and the initialized specific parameters.

[0140] In some embodiments of the present application, based on the aforementioned scheme, the model generation module 803 is also configured to: obtain the shared parameters of the text recognition model according to the initialized general parameters; obtain the task parameters of the text recognition model according to the initialized specific parameters; and establish the text recognition model according to the shared parameters and the task parameters.

[0141] In some embodiments of the present application, based on the aforementioned solution, the model generation module 803 is further configured to: perform linear processing on the general function and the specific function to obtain a text recognition model.

[0142] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0143] Refer to the following Fig. 9 An electronic device 90 according to this embodiment of the present application will be described. Fig. 9 The electronic device 90 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0144] like Fig. 9 As shown, the electronic device 90 is in the form of a general computing device. The components of the electronic device 90 may include, but are not limited to: at least one processing unit 91, at least one storage unit 92, a bus 93 connecting different system components (including the storage unit 92 and the processing unit 91), and a display unit 94.

[0145] The storage unit stores program codes, which can be executed by the processing unit 91, so that the processing unit 91 executes the steps described in the above “Example Method” section of this specification according to various exemplary implementations of the present application.

[0146] The storage unit 92 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 921 and / or a cache storage unit 922 , and may further include a read-only storage unit (ROM) 923 .

[0147] The storage unit 92 may also include a program / utility 924 having a set (at least one) of program modules 925, such program modules 925 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0148] Bus 93 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0149] The electronic device 90 may also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may communicate with one or more devices that enable a user to interact with the electronic device 90, and / or may communicate with any device that enables the electronic device 90 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 95. Furthermore, the electronic device 90 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 96. As shown, the network adapter 96 communicates with other modules of the electronic device 90 via a bus 93. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 90, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0150] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0151] According to one embodiment of the present application, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present application can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present application described in the above "Exemplary Method" section of the present specification.

[0152] According to one embodiment of the present application, a program product for implementing the above method may adopt a portable compact disk read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.

[0153] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0154] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0155] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0156] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0157] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present application, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0158] It should be understood that the present application is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be performed without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A text processing method, characterized in that: include: Based on the labeled training data and the guided task, the pre-trained model is subjected to knowledge distillation to obtain the guided model; Generate a distance function corresponding to the guidance task according to the corresponding relationship between the output of the pre-trained model and the output of the guidance model; Performing knowledge distillation processing on the guidance model according to the unlabeled training data and the distance function to obtain a pre-recognition model; generating a universal function according to the corresponding relationship between the output of the guidance model and the output of the pre-recognition model; Obtaining universal parameters of the pre-recognition model according to the universal function; Initializing the general parameters; Establishing a universal model according to the initialized universal parameters; Performing knowledge distillation processing on the general model based on training data corresponding to a specified text recognition task to obtain a text recognition model; Get the text to be processed; The text to be processed is input into a text recognition model for processing to obtain a recognition result of the text to be processed by the text recognition model. The text recognition model is obtained by performing knowledge distillation processing on a pre-trained model used for text processing based on labeled training data and a guiding task corresponding to text recognition, and performing knowledge distillation processing again based on the knowledge distillation processing result and the unlabeled training data.

2. The text processing method according to claim 1, characterized in that: Also includes: Training the pre-trained model based on training data corresponding to a specified text recognition task; Based on the training data corresponding to the specified text recognition task, the training results are processed by knowledge distillation to obtain a specific model; Generate a specific function according to the corresponding relationship between the output of the pre-trained model and the output of the specific model; Generating the text recognition model according to the universal function includes: The text recognition model is generated according to the general function and the specific function.

3. The text processing method according to claim 2, characterized in that: The step of generating the text recognition model according to the general function and the specific function comprises: Obtaining universal parameters of the pre-recognition model according to the universal function, and initializing the universal parameters; Obtaining specific parameters of the specific model according to the specific function, and initializing the specific parameters; The text recognition model is obtained according to the initialized general parameters and the initialized specific parameters.

4. The text processing method according to claim 3, characterized in that: The step of obtaining the text recognition model according to the initialized general parameters and the initialized specific parameters includes: Obtaining shared parameters of the text recognition model according to the initialized common parameters; Obtaining task parameters of the text recognition model according to the initialized specific parameters; The text recognition model is established according to the shared parameters and the task parameters.

5. The text processing method according to claim 2, characterized in that: The step of generating the text recognition model according to the general function and the specific function comprises: The universal function and the specific function are linearly processed to obtain the text recognition model.

6. A text processing device, characterized in that: include: A processing module, used for performing knowledge distillation processing on the pre-trained model based on the labeled training data and the guided task to obtain a guided model; Generate a distance function corresponding to the guidance task according to the corresponding relationship between the output of the pre-trained model and the output of the guidance model; Performing knowledge distillation processing on the guidance model according to the unlabeled training data and the distance function to obtain a pre-recognition model; generating a universal function according to the corresponding relationship between the output of the guidance model and the output of the pre-recognition model; Obtaining universal parameters of the pre-recognition model according to the universal function; Initializing the general parameters; Establishing a universal model according to the initialized universal parameters; Performing knowledge distillation processing on the general model based on training data corresponding to a specified text recognition task to obtain a text recognition model; An acquisition module is used to obtain the text to be processed; The processing module is also used to input the text to be processed into a text recognition model for processing to obtain a recognition result of the text to be processed by the text recognition model. The text recognition model is obtained by performing knowledge distillation processing on a pre-trained model used for text processing based on labeled training data and a guiding task corresponding to text recognition, and performing knowledge distillation processing again based on the knowledge distillation processing result and unlabeled training data.

7. A computer device, characterized in that: include: a memory, wherein a computer program is stored in the memory; A processor, used to load the computer program to implement the text processing method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the text processing method according to any one of claims 1 to 5.

Images