Training Method, Device, Electronic Device and Storage Medium for Text Processing Model

Abstract

The embodiments of the present application provide a method, apparatus, electronic device, and storage medium for training a text processing model, which relate to the fields of cloud technology and artificial intelligence technology. The method for training the text processing model includes: constructing an initial neural network model including a first text processing model and a second text processing model, and jointly training the first text processing model and the second text processing model using a first training data set and a second training data set in a different domain from the first training data set until a preset training end condition is satisfied, so as to obtain a text processing model that meets the requirements. Through joint training, auxiliary training of the first text processing model can be realized. Moreover, using training data sets in different domains helps to expand the domain of the text processing model and improve the accuracy of the model in processing data in different domains.

Description

Technical Field

[0001] This application relates to the fields of cloud technology and artificial intelligence technology. Specifically, this application relates to a method, apparatus, electronic device, and storage medium for training a text processing model. Background Art

[0002] Text processing technology is a technology commonly used in the industrial field for natural language processing. Through a text processing model, corresponding processing is performed on text data according to different application scenarios and actual processing tasks. Among them, text processing can include text matching, text classification, text generation, etc. In the prior art, for a trained text processing model, the processing performance of the model is mainly limited to the field to which the training data of the model belongs. When processing text in an extended field, the effect of text processing is usually not ideal. Summary of the Invention

[0003] Embodiments of this application provide a method, apparatus, electronic device, and storage medium for training a text processing model. Based on this solution, the text matching effect of the text matching model in the extended field can be effectively improved.

[0004] To achieve the above object, the specific technical solutions provided by the embodiments of this application are as follows:

[0005] On the one hand, an embodiment of this application provides a method for training a text processing model. The method includes:

[0006] Construct an initial neural network model. The initial neural network model includes a first text processing model and a second text processing model. Among them, the first text processing model includes a cascaded text feature extraction module and a result prediction module. The second text processing model is connected to the output of the text feature extraction module. The second text processing model includes at least one of a masked language model or a named entity recognition model;

[0007] Obtain a first training data set corresponding to the first text processing model and a second training data set corresponding to the second text processing model. Among them, the first training data set and the second training data set belong to different fields;

[0008] Based on the first training data set and the second training data set, train the initial neural network model until a preset training end condition is met, and use the first text processing model at the end of training as the final text processing model.

[0009] On the other hand, an embodiment of the present invention also provides a text processing method. The method includes:

[0010] Obtain the text to be processed;

[0011] Input the text to be processed into a text processing model to obtain a processing result;

[0012] Based on the processing result, perform corresponding processing;

[0013] Among them, the text processing model is trained based on the method of any implementation manner of the first aspect.

[0014] The embodiment of the present invention also provides a training device for a text processing model, and the device includes:

[0015] A model construction module, configured to construct an initial neural network model. The initial neural network model includes a first text processing model and a second text processing model. Among them, the first text processing model includes a cascaded text feature extraction module and a result prediction module, and the second text processing model is connected to the output of the text feature extraction module. The second text processing model includes at least one of a masked language model or a named entity recognition model;

[0016] A data acquisition module, configured to acquire a first training data set corresponding to the first text processing model and a second training data set corresponding to the second text processing model. Among them, the first training data set and the second training data set belong to different fields;

[0017] A model training module, configured to train the initial neural network model based on the first training data set and the second training data set until a preset training end condition is met, and use the first text processing model at the end of training as the final text processing model.

[0018] The embodiment of the present invention also provides a text processing device, and the device includes:

[0019] A text acquisition module, configured to acquire the text to be processed;

[0020] A model processing module, configured to input the text to be processed into the text processing model to obtain a processing result;

[0021] A result processing module, configured to perform corresponding processing based on the processing result;

[0022] Among them, the text processing model is trained based on the method of any implementation manner of the first aspect.

[0023] The embodiment of the present invention also provides an electronic device, and the electronic device includes one or more processors; a memory; one or more computer programs, where one or more computer programs are stored in the memory and configured to be executed by one or more processors, and one or more computer programs are configured to execute the method shown in the first aspect or the second aspect of the present application.

[0024] An embodiment of the present invention also provides a computer-readable storage medium for storing a computer program. When the computer program runs on a processor, the processor can execute the method as shown in the first aspect or the second aspect of the present application.

[0025] An embodiment of the present invention also provides a computer program product or a computer program. The computer program product or the computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in various alternative implementations of the above text processing model training method or text processing method.

[0026] The beneficial effects brought by the technical solution provided by the present application are:

[0027] The present application provides a method, apparatus, electronic device, and storage medium for training a text processing model. The method for training the text processing model includes: jointly training a first text processing model and a second text processing model using a first training data set and a second training data set in different domains until a preset training end condition is met, thereby obtaining a text processing model that meets the requirements. Through joint training, auxiliary training of the first text processing model can be realized, which helps to improve the accuracy of the trained text processing model in processing data in different domains. Moreover, since the second text processing model is at least one of a masked language model or a named entity recognition model, the training samples do not require sentence pairs and corresponding labels in the extended domain, and the training cost of the model will not increase, and domain expansion without increasing costs can be achieved, thus solving the problems of difficult domain migration and low data processing accuracy when the model processes data in the extended domain, and improving the usage effect of the text processing model in the extended domain. Description of the Drawings

[0028] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the accompanying drawings required for the description in the embodiments of the present application.

[0029] Figure 1 It is a schematic flowchart of a method for training a text processing model provided by an embodiment of the present application;

[0030] Figure 2 It is a schematic diagram of the training process of the second text processing model provided by an embodiment of the present application;

[0031] Figure 3 It is a schematic diagram of the joint alternating training of the first text processing model and the second text processing model provided by an embodiment of the present application;

[0032] Figure 4Schematic flowchart of the text processing method provided by the embodiments of the present application;

[0033] Figure 5 Schematic diagram of the interface for provident fund query provided by the embodiments of the present application;

[0034] Figure 6 Schematic diagram of data processing of the question-and-answer model provided by the embodiments of the present application;

[0035] Figure 7 Schematic diagram of the interface of the question-and-answer model provided by the embodiments of the present application;

[0036] Figure 8 Schematic diagram of the structure of the training device of the text processing model provided by the embodiments of the present application;

[0037] Figure 9 Schematic diagram of the structure of the text processing device provided by the embodiments of the present application;

[0038] Figure 10 Schematic diagram of the structure of an electronic device provided by the embodiments of the present application. Detailed implementation manners

[0039] The embodiments of the present application will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and should not be construed as a limitation of the present application.

[0040] Those skilled in the art of the present technology can understand that unless specifically stated otherwise, the singular forms "a", "an", "the" and "said" used herein may also include the plural forms. It should be further understood that the term "including" used in the specification of the present application means the presence of the described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to other elements, or there may also be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any unit and all combinations of one or more related listed items.

[0041] The embodiments of the present application address the problem that in the prior art, text processing models are limited to the domain of the training dataset, and when processing text data in an extended domain, it is difficult to perform domain migration and the processing effect is not good. The training method for the text processing model provided by the embodiments of the present application uses the first training dataset and the second training dataset in different domains to jointly train the first text processing model and the second text processing model until the preset training end condition is met, thereby obtaining a text processing model that meets the requirements. Through joint training, the auxiliary training of the first text processing model can be achieved, which helps to improve the accuracy of the trained text processing model in processing data from different domains. Moreover, since the second text processing model is a masked language model or a named entity recognition model, the training samples do not require sentence pairs and corresponding labels in the extended domain, and the training cost of the model will not increase, and domain expansion without increasing costs can be achieved, thereby solving the problems of difficult domain migration and low data processing accuracy when the model processes data in the extended domain, and improving the usage effect of the text processing model in the extended domain.

[0042] The present application also provides a text processing method obtained by the training method of the text processing model provided by the embodiments of the present application. Based on this method, the text to be processed in the extended domain of the model can be processed accordingly, and the data processing accuracy is relatively high.

[0043] The solutions provided by the optional embodiments of the present application relate to fields such as artificial intelligence, cloud technology, and big data in computer technology.

[0044] The model training method and the text processing method involved in the embodiments of the present application can be implemented through machine learning in artificial intelligence technology.

[0045] Artificial Intelligence (AI) is to use a digital computer or a machine controlled by a digital computer to simulate, extend, and expand human intelligence, and to perceive the environment, acquire knowledge, and use knowledge to obtain the best results in theory, methods, technologies, and application systems. In other words, artificial intelligence is a comprehensive technology in computer science. It attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a way similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines to enable the machines to have the functions of perception, reasoning, and decision-making.

[0046] Artificial intelligence technology is an interdisciplinary subject, involving a wide range of fields, including both hardware-level technologies and software-level technologies. The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation / interaction systems, and mechatronics. The artificial intelligence technology involved in the embodiments of the present application mainly includes several major directions such as natural language processing technology, machine learning / deep learning.

[0047] Natural Language Processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between humans and computers in natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, the research in this field will involve natural language, that is, the language people use in daily life, so it has a close connection with the research of linguistics. Natural language processing technologies usually include text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0048] Machine Learning (ML) is an interdisciplinary subject that involves multiple disciplines such as probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize the existing knowledge structure to continuously improve their own performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications cover all fields of artificial intelligence. Machine learning and deep learning usually include technologies such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and rote learning.

[0049] The data processing involved in the embodiments of this application can be implemented through cloud technology, and the data calculation involved can be implemented through cloud computing in cloud technology.

[0050] Cloud computing is a computing model that distributes computing tasks on a resource pool composed of a large number of computing devices, enabling various application systems to obtain computing power, storage space, and information services according to needs. The network that provides resources is called the "cloud". The resources in the "cloud" seem to be infinitely expandable to users, and can be obtained at any time, used on demand, expanded at any time, and paid according to usage.

[0051] As a basic capability provider of cloud computing, a cloud computing resource pool (abbreviated as cloud platform, generally called IaaS (Infrastructure as a Service) platform) will be established, and various types of virtual resources will be deployed in the resource pool for external customers to choose and use. The cloud computing resource pool mainly includes: computing devices (virtual machines, including operating systems), storage devices, and network devices.

[0052] According to the logical function division, the PaaS (Platform as a Service) layer can be deployed on the IaaS (Infrastructure as a Service) layer, and the SaaS (Software as a Service) layer can be deployed on the PaaS layer. SaaS can also be deployed directly on IaaS. PaaS is a platform for software operation, such as databases, web containers, etc. SaaS is a variety of business software, such as web portals, SMS mass senders, etc. Generally speaking, SaaS and PaaS are upper layers relative to IaaS.

[0053] Cloud computing refers to the delivery and use model of IT infrastructure, which means obtaining required resources through the network in an on-demand and easily scalable manner; in a broad sense, cloud computing refers to the delivery and use model of services, which means obtaining required services through the network in an on-demand and easily scalable manner. This service can be related to IT and software, the Internet, or other services. Cloud computing is the product of the development and integration of traditional computer and network technologies such as grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing.

[0054] With the development of the Internet, real-time data streams, and the diversification of connected devices, as well as the demand for search services, social networks, mobile commerce, and open collaboration, cloud computing has developed rapidly. Different from the previous parallel distributed computing, the emergence of cloud computing will promote revolutionary changes in the entire Internet model and enterprise management model from a conceptual perspective.

[0055] The training data required for the model training involved in the embodiments of the present application can be big data obtained from the Internet.

[0056] Big data refers to a collection of data that cannot be captured, managed, and processed by conventional software tools within a certain time range. It is a vast, high-growth, and diverse information asset that requires new processing models to have stronger decision-making power, insight discovery ability, and process optimization ability. With the advent of the cloud era, big data has attracted more and more attention. Big data requires special technologies to effectively process a large amount of data tolerated over time. Technologies applicable to big data include massively parallel processing databases, data mining, distributed file systems, distributed databases, cloud computing platforms, the Internet, and scalable storage systems.

[0057] The following uses specific embodiments to elaborate in detail on the technical solutions of this application and how the technical solutions of this application solve the above technical problems. These several specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The following will describe the embodiments of this application with reference to the accompanying drawings.

[0058] The execution subject of the technical solution of this application is a computer device, including but not limited to servers, personal computers, laptops, tablets, smartphones, etc. The computer device includes a user device and a network device. Among them, the user device includes but not limited to computers, smartphones, PADs, etc.; the network device includes but not limited to a single network server, a server group composed of multiple network servers, or a cloud composed of a large number of computers or network servers in cloud computing. Among them, cloud computing is a type of distributed computing, which is composed of a group of loosely coupled computer sets to form a super virtual computer. Among them, the computer device can run alone to implement this application, or can be connected to the network and implement this application through interaction with other computer devices in the network. Among them, the network where the computer device is located includes but not limited to the Internet, wide area network, metropolitan area network, local area network, VPN network, etc.

[0059] The embodiment of this application provides a method for training a text processing model. The execution subject of this method can be any electronic device, such as Figure 1 As shown, this method may include:

[0060] Step S101, constructing an initial neural network model;

[0061] Among them, the initial neural network model includes a first text processing model and a second text processing model. The first text processing model includes a cascaded text feature extraction module and a result prediction module. The second text processing model is connected to the output of the text feature extraction module. The second text processing model includes at least one of a masked language model or a named entity recognition model.

[0062] Among them, the text feature extraction module is used to extract features from the input text, and can be any neural network model with text feature extraction capabilities. For example, it can be a Bidirectional Encoder Representations from Transformers (BERT) model, etc. The result prediction module is used to predict results based on the text features output by the text feature extraction module, and can be any neural network model with result prediction capabilities. Taking text similarity prediction as an example, the result prediction module can be a siamese network, etc. The Masked Language Model is used to predict the masked part of the partially masked text data to obtain the predicted value of the masked part. The Named Entity Recognition (NER) model is used to recognize named entities in the input text to obtain the predicted value of the named entities. Among them, named entities can include but are not limited to personal names, organization names, place names, and all other entities identified by names, and also include numbers, dates, currencies, addresses, etc.

[0063] In the embodiment of the present application, through the second text processing model connected to the output of the text feature extraction module, the training of the text feature extraction module can be assisted, and the encoding and modeling ability of the finally trained text processing model can be improved.

[0064] Step S102, obtain a first training data set corresponding to the first text processing model and a second training data set corresponding to the second text processing model;

[0065] Among them, the first training data set and the second training data set belong to different fields. The first training data set is the training data set corresponding to the first text processing model and is used to train the text feature extraction module and the result prediction module. The second training data set is the training data set corresponding to the second text processing model and is used to train at least one of the Masked Language Model or the Named Entity Recognition model. Optionally, the first training data set can be the data of the original preset field, and the second training data set can be the data of the extended field.

[0066] Training a masked language model or a named entity recognition model using a second training dataset from a different domain than the first training dataset helps improve the encoding and modeling ability of the finally trained text processing model for texts in the extended domain, thereby enhancing the text processing effect of the text processing model for texts in the extended domain. Moreover, since the second text processing model is at least one of a masked language model or a named entity recognition model, the training samples do not require sentence pairs in the extended domain and corresponding labels, which does not increase the training cost of the model, and domain expansion without increasing costs can be achieved, thus solving the problems of difficult domain transfer and low data processing accuracy when the model processes data in the extended domain, and enhancing the usage effect of the text processing model in the extended domain.

[0067] Step S103: Based on the first training dataset and the second training dataset, train the initial neural network model until the preset training end condition is met, and use the first text processing model at the end of training as the final text processing model.

[0068] In the embodiment of this application, an initial neural network model including a first text processing model and a second text processing model is constructed, and the first training dataset and the second training dataset composed of unlabeled data from a different domain than the first training dataset are used to jointly train the first text processing model and the second text processing model, so as to obtain a text processing model that meets the requirements. Through joint training, auxiliary training of the first text processing model can be achieved, which helps improve the accuracy of the trained text processing model in processing data from different domains. Moreover, since unlabeled data from different domains is used, the training cost of the model is not increased, and domain expansion without increasing costs can be achieved, enhancing the usage effect of the text processing model in the extended domain.

[0069] The finally obtained text processing model in the embodiment of this application can be applied to scenarios such as text matching, text classification, and text generation. Correspondingly, the text processing model is any one of a text matching model, a text classification model, and a text generation model, and corresponding output results are obtained based on the specific type of the text processing model for corresponding processing such as text matching, text classification, and text generation.

[0070] In practical applications, a text matching model is a model for processing text matching tasks. Text matching is to calculate the semantic similarity between two texts, and based on the calculation of semantic similarity, to make a matching judgment on whether the text pair is similar or not. Text matching can be applied to a large number of natural language processing tasks, such as information retrieval, question answering systems, paraphrasing questions, dialogue systems, etc. To a large extent, these natural language processing tasks can be abstracted into text matching problems. For example, information retrieval can be reduced to the matching between search terms and document resources, question answering systems can be reduced to the matching between questions and candidate answers, paraphrasing questions can be reduced to the matching between two synonymous sentences, and dialogue systems can be reduced to the matching between the previous dialogue and the reply.

[0071] A text classification model is a model for processing text classification tasks. Text classification means using a computer to automatically classify and label a text data set according to a certain classification system or standard. A text classification model is a model that determines the relationship between document features and text categories based on a labeled training text data set, and can use the text classification model to judge the category of unlabeled text.

[0072] A text generation model is a model that generates new text data based on input text data. The text generation model can be based on text statistics, learn the combination rules between different texts from a large amount of text, and then infer possible combination methods as output based on the input.

[0073] In addition, the specific structure of the text matching model, text classification model, and text generation model can be constructed according to specific needs, and can be any neural network model that can achieve the corresponding functions. This application does not make any limitations in this regard.

[0074] In a possible implementation manner, the second text processing model includes a masked language model and a named entity recognition model respectively connected to the output of the text feature extraction module, and the second training data set includes a third training data set corresponding to the masked language model and a fourth training data set corresponding to the named entity recognition model.

[0075] In practical applications, the second text processing model can include two models, namely a masked language model and a named entity recognition model. Correspondingly, the second training data set includes a third training data set and a fourth training data set respectively corresponding to the masked language model and the named entity recognition model. When training the model, the two models are trained respectively using their corresponding training data sets. In the embodiments of this application, by using the masked language model and the named entity recognition model to assist the training of the text feature extraction module, the encoding and modeling ability of the finally trained text processing model can be improved from two aspects: masked text recognition and named entity recognition, thereby improving the data processing effect of the text processing model.

[0076] In a possible implementation, based on a first training data set and a second training data set, an initial neural network model is trained until a preset training end condition is met, including:

[0077] Based on the first training data set and the second training data set, joint alternating training is performed on a first text processing model and a second text processing model until the loss function corresponding to the first text processing model converges. The training end condition includes the convergence of the loss function corresponding to the first text processing model;

[0078] Wherein, for each training, based on the training data set corresponding to the model branch of this training and the text processing result of the training data set obtained through the model branch, the value of the loss function corresponding to the model branch is determined, and the model parameters of the model branch are adjusted based on the value of the loss function corresponding to the model branch. The model branch is the first text processing model or the second text processing model.

[0079] In practical applications, when performing joint alternating training on the first text processing model and the second text processing model, since the second text processing model is connected to the output of the text feature extraction module to assist in the training of the text feature extraction module, the order of joint alternating training can be: perform one training on the first text processing model based on the first training data set, determine the value of the first loss function based on the first training data set and the text processing result of the first training data set obtained through the first text processing model. If the first loss function does not converge, adjust the model parameters of the first text processing model; perform one training on the second text processing model based on the second training data set, determine the value of the second loss function based on the second training data set and the text processing result of the second training data set obtained through the second text processing model; if the second loss function does not converge, adjust the model parameters of the second text processing model, and repeat the above training steps until the first loss function converges. At this time, even if the second loss function does not converge, training can stop, and the model training ends. Wherein, the first loss function can be a cross-entropy loss function. Additionally, if the second loss function converges first and the first loss function does not converge, the above joint alternating training process can continue until the first loss function converges, and the model training is completed.

[0080] In the embodiments of this application, the first training data set and the second training data set belong to different fields. Through the joint alternating training of the first text processing model and the second text processing model, the second text processing model assists in the training of the text feature extraction module in the first text processing model, which can improve the encoding and modeling ability of the finally trained text processing model in the field to which the second training data set belongs, thereby realizing the expansion of the model application field.

[0081] In a possible implementation, the second text processing model includes a masked language model and a named entity recognition model respectively connected to the output of the text feature extraction module, and the second training dataset includes a third training dataset corresponding to the masked language model and a fourth training dataset corresponding to the named entity recognition model;

[0082] Based on the first training dataset and the second training dataset, jointly and alternately train the first text processing model and the second text processing model until the loss function corresponding to the first text processing model converges, including:

[0083] Based on the first training dataset, the third training dataset, and the fourth training dataset, jointly and alternately train the first text processing model, the masked language model, and the named entity recognition model until the loss function corresponding to the first text processing model converges.

[0084] In practical applications, if the second text processing model includes a masked language model and a named entity recognition model, the model branches are any one of the first text processing model, the masked language model, or the named entity recognition model. The first text processing model, the masked language model, and the named entity recognition model can be jointly and alternately trained through the following process: Train the first text processing model once based on the first training dataset to determine the value of the first loss function; Train the masked language model once based on the third training dataset to determine the value of the third loss function; Train the named entity recognition model once based on the fourth training dataset to determine the value of the fourth loss function; After multiple trainings, until the first loss function converges, at this time, even if the third loss function and the fourth loss function do not converge, training can stop and the model training is completed. Additionally, if the third loss function and the fourth loss function converge first and the first loss function does not converge, the above joint alternating training process can continue until the first loss function converges and the model training is completed.

[0085] In a possible implementation, for the masked language model or the named entity recognition model, for each training, based on the training dataset corresponding to the model branch of this training and the text processing results of the training dataset obtained through the model branch, determine the value of the loss function corresponding to the model branch, including:

[0086] Input each training sample of the training dataset corresponding to the model branch into the text feature extraction module to obtain the text features of each sample;

[0087] Input the text features of each sample into the model branch to obtain the text processing results of each sample;

[0088] Based on the sample labels of each sample and the text processing results of each sample, determine the value of the loss function corresponding to the model branch.

[0089] In practical applications, when training the model branches, for the masked language model or the named entity recognition model, the corresponding training datasets respectively include the training samples corresponding to each model branch and the sample labels of each sample. The masked language model and the named entity recognition model are respectively connected to the output of the text feature extraction module. The training samples corresponding to the model branches are input into the text feature extraction module to obtain the text features of each sample. The text features of each sample are input into the model branches to obtain the text processing results of each sample. Based on the sample labels of each sample and the text processing results of each sample, the value of the loss function corresponding to the model branch can be calculated, that is, the value of the third loss function corresponding to the masked language model and the value of the fourth loss function corresponding to the named entity recognition model.

[0090] In a possible implementation, the second text processing model includes a masked language model. Obtaining the second training dataset corresponding to the second text processing model includes:

[0091] Obtaining each first text belonging to the first domain;

[0092] For each first text, masking at least one character in the first text to obtain a second text;

[0093] Taking each first text and the second text corresponding to the first text as a training sample corresponding to the masked language model in the second training dataset.

[0094] In practical applications, the first domain can be the domain of the text to be processed by the trained text processing model, that is, the extended domain of the model, which is a domain different from the first training dataset. By obtaining the training dataset in this domain, the encoding and modeling ability of the trained text processing model in this domain can be enhanced. Each training sample can be the first text and the second text obtained by masking the characters in the first text. Among them, the first text can be a sentence in the first domain.

[0095] In an example, the first text is sentence A1. 15% of the characters in sentence A1 are masked, and the masked sentence A1 is used as the second text. The unmasked sentence A1 and the masked sentence A1 are used as a training sample of the masked language model.

[0096] In a possible implementation, the second text processing model includes a named entity recognition model. Obtaining the second training dataset corresponding to the second text processing model includes:

[0097] Obtaining third texts belonging to the second domain;

[0098] For each third text, annotating at least one entity in the third text to obtain a fourth text;

[0099] Use each third text and the fourth text corresponding to the third text as a training sample of the second training dataset.

[0100] In practical applications, in order to enhance the encoding and modeling ability of the named entities in the text of the extended domain by the trained text processing model, training samples of the named entity model can be obtained in the extended domain. Specifically, obtain the third text. The third text can be the text in the second domain different from the first domain or the text in the first domain. The third text can specifically be a sentence. Label the entities in the sentence to obtain the labeled sentence as the fourth text, and use the third text and the fourth text as a training sample of the named entity recognition model. Optionally, when labeling the entities in the third text, the entities in the sentence can be labeled, and the labeled entities can include nouns in the sentence, proper nouns in the first domain, and three other categories.

[0101] The training process of the second text processing model in the technical solution of the present application will be described in detail below through a specific embodiment. This embodiment is only one embodiment of the technical solution of the present application and does not represent all implementation manners of the technical solution of the present application.

[0102] As Figure 2 shown, in this embodiment, the second text processing model includes a masked language model or a named entity recognition model, and the text feature extraction module is a BERT model. The process of inputting samples into the second text processing module and training will be introduced below:

[0103] The BERT model includes an input layer, an encoding layer, and an output layer. The training sample of the second text processing model is input into the input layer of the BERT model. The training sample can be a text in the extended field of the text processing model. This embodiment takes the training of the masked language model as an example, wherein each training sample is a sentence, such as "an example of display" as shown in the figure, wherein CLS and SEP are flags in the sentence, which are used to segment each sentence, and the "individual" in the sentence is masked, and the MASK flag indicates the position of the masked text in the sentence. The input layer performs initialization feature extraction on the input training samples. Specifically, the initialization features of the text are respectively obtained through the token embedding layer, segment embedding layer, and position embedding layer in the input layer to obtain word features, segment features, and position features, and the features of these three dimensions are spliced to obtain an initialization feature vector, which is input into the encoding layer of the BERT model (BERT as shown in the figure) to obtain the sample features after encoding, and the encoded sample features are output to the masked language model for processing, and the loss function corresponding to the masked language model is calculated. The language model is used to assist in the training of the BERT model, which can improve the encoding modeling ability of the text processing model in the extended field, thereby improving the data processing effect of the text processing model in the extended field.

[0104] The following is a detailed description of the process of joint alternating training of the first text processing model and the second text processing model in the technical solution of the present application through a specific embodiment. This embodiment is only an embodiment of the technical solution of the present application, and does not represent all implementation methods of the technical solution of the present application.

[0105] like Figure 3 As shown, in this embodiment, the second text processing model includes a masked language model and a named entity recognition model, the text feature extraction module is a BERT model, and the result prediction module is a similarity prediction module (cosine sim as shown in the figure). For the convenience of explanation, when training the masked language model, the "sentence A" shown in the figure is used as a sample in the training set of the masked language model, including sentence A and masked sentence A obtained by masking the characters in sentence A; when training the named entity recognition model, the "sentence A" shown in the figure is used as a sample in the training set of the named entity recognition model, including sentence A and annotated sentence A obtained by annotating the named entities in sentence A. When training the first text processing model, sentence A, sentence B, and the similarity labels of sentence A and sentence B are used as training samples. The specific process of joint alternating training of the first text processing model and the second text processing model is as follows:

[0106] First, train the masked language model once with the training samples corresponding to the masked language model. Determine the masked loss function according to sentence A and the prediction results of the masked sentence A output by the masked language model. Then, train the named entity recognition model once with the training samples corresponding to the named entity recognition model. Determine the named entity loss function according to sentence A and the prediction results of the named entities in sentence A output by the named entity recognition model. Finally, train the BERT model and the similarity prediction module once based on the training samples corresponding to the first text processing model. Input the training sample sentences A and B into the BERT model for feature extraction respectively. Process sentence A to obtain features in multiple dimensions corresponding to sentence A, perform pooling on the features in multiple dimensions to obtain the feature vector U corresponding to sentence A. In the same processing manner, process sentence B to obtain features in multiple dimensions corresponding to sentence B, perform pooling on the features in multiple dimensions to obtain the feature vector V corresponding to sentence B. Predict the similarity between vectors U and V through the similarity prediction module, calculate the similarity loss function according to the similarity labels of sentences A and B and the predicted similarity. Repeat the above training process to jointly and alternately train the masked language model, the named entity recognition model, the BERT model, and the similarity prediction module until the similarity loss function converges, and the training of the text processing model is completed.

[0107] In an optional embodiment, the relevant parameters of the software and hardware environment of the training method of the text processing model provided by the embodiment of the present application are shown in Table 1:

[0108] Operating System Memory Language Environment Linux >16G Python / c++

[0109] Table 1

[0110] The electronic device for executing the training method of the text processing model provided by the embodiment of the present application can be a server, and its relevant parameters of the software and hardware environment are shown in Table 1. The operating system can be the Linux system, the memory > 16G, and the language environment can be Python / c++.

[0111] In the embodiment of the present application, through the joint and alternate training of the first text processing model and the second text processing model, the second text processing model assists the text feature extraction module in the first text processing model for training, which can improve the encoding and modeling ability of the finally trained text processing model in the extended field, thereby realizing the expansion of the model application field.

[0112] The training method of the text processing model provided by the embodiments of the present application uses the first training dataset and the second training dataset in different fields to jointly train the first text processing model and the second text processing model until the preset training end condition is met, so as to obtain a text processing model that meets the requirements. Through joint training, the auxiliary training of the first text processing model can be realized, which helps to improve the accuracy of the trained text processing model in processing data in different fields. Moreover, since the second text processing model is at least one of a masked language model or a named entity recognition model, the training samples do not require sentence pairs in the extended field and corresponding labels, which will not increase the training cost of the model, and can achieve field extension without increasing the cost, thus solving the problems of difficult field migration and low data processing accuracy when the model processes data in the extended field, and improving the usage effect of the text processing model in the extended field.

[0113] The embodiments of the present application provide a text processing method. The execution subject of this method can be any electronic device. For example, this method can be executed by a server, such as Figure 4 As shown, this method may include:

[0114] Step S201, obtain the text to be processed;

[0115] Among them, the source of the text to be processed is not limited in the embodiments of the present application. For example, it can be any text in each text pre-stored in the database, or any text in multiple texts received from each user device.

[0116] It can be understood that for different application scenarios, the above-mentioned text to be processed may be different.

[0117] In a possible implementation manner, the text to be processed includes the text in the field to which the second training dataset belongs.

[0118] In practical applications, the field to which the second training dataset belongs can be the extended field of the text processing model. The text to be processed can be the text in the extended field of the model. It can be understood that the text to be processed can also be the text in the original processing field of the model, that is, the field to which the first training dataset belongs. When the text processing model is trained, training the training samples in different fields of the first text processing model and the second text processing model is beneficial to improving the encoding and modeling ability of the text processing model for the extended field.

[0119] Step S202, input the text to be processed into the text processing model to obtain a processing result;

[0120] Specifically, the text processing model is the first text processing model at the end of training provided in the embodiments of the present application, including a text feature extraction module and a result prediction module. The text to be processed is input into the text processing model, and after the text processing model extracts text features and predicts results, the processing result output by the text processing model is obtained.

[0121] Step S203, perform corresponding processing based on the processing result;

[0122] The text processing model can be any one of a text matching model, a text classification model, and a text generation model. Corresponding output results are obtained based on the specific type of the text processing model, and corresponding processing such as text matching, text classification, and text generation is performed.

[0123] In a possible implementation, the text processing model is a text matching model, and the text to be processed includes the query text of the requester and several candidate query results corresponding to the query text;

[0124] Inputting the text to be processed into the text processing model includes:

[0125] Input the query text and each candidate query result into the text matching model to obtain the first matching degree between the query text and each candidate query result. The processing result includes the first matching degree;

[0126] Performing corresponding processing based on the processing result includes:

[0127] Based on each first matching degree, determine the target query result from each candidate query result and provide the target query result to the requester.

[0128] In practical applications, the text processing model can be a text matching model. The text to be processed includes the query text of the requester and several candidate query results corresponding to the query text. The matching degree between the query text query and several candidate query results can be calculated based on the text matching model to determine the target query result. Optionally, the candidate query results with the matching degree within a preset range can be used as the target query result, or the query result with the highest matching degree can be provided to the requester as the target query result.

[0129] In an example, such as Figure 5As shown, in the application scenario of provident fund query, the user terminal receives the query text "provident fund" entered by the user through the search box in the provident fund query interface, obtains several candidate query results corresponding to the query text "provident fund" in the provident fund query database, inputs the query text "provident fund" and the corresponding several candidate query results into the text matching model, determines the matching degree between "provident fund" and the corresponding several candidate query results through the text matching model, determines the target query result from each candidate query result according to each matching degree, and provides the target query result to the requester. As shown in the figure, the finally obtained target query results are texts corresponding to various categories such as "provident fund query", "provident fund service", and "provident fund - article", and are provided to the requester through the display interface of the user terminal.

[0130] In a possible implementation manner, the text processing model includes a question - answering model, and the text to be processed includes the question text of the questioner and several candidate answers corresponding to the question text;

[0131] Inputting the text to be processed into the text processing model includes:

[0132] Inputting the question text and each candidate answer into the question - answering model to obtain the second matching degree between the question text and each candidate answer, and the processing result includes the second matching degree;

[0133] Based on the processing result, corresponding processing is performed, including:

[0134] Based on each second matching degree, determine the target answer from each candidate answer and provide the target answer to the questioner.

[0135] In practical applications, the text processing model can be a question - answering model. The text to be processed includes the question text of the questioner and several candidate answers corresponding to the question text. The matching degree between the question text and several candidate answers can be calculated based on the question - answering model to determine the target answer. Optionally, the candidate answers with the matching degree within a preset range can be used as the target answers, or the candidate answer with the highest matching degree can be provided to the questioner as the target answer.

[0136] In an example, such as Figure 6As shown, the question text input by the user through the user terminal is received, and several candidate answers corresponding to the question text are retrieved from the retrieval library. Then, the candidate answers are input into the Q&A model to calculate the matching degree between the question text and each candidate answer (such as the text matching calculation shown in the figure). The candidate answers are sorted according to each matching degree (such as the matching result sorting shown in the figure), and the candidate answers ranked in the top preset positions are provided to the questioner. Optionally, the target answer can be text in the form of a question, and according to the selection instruction input by the user for this text, the corresponding answer is determined. Optionally, the target answer can also be text in the form of the answer corresponding to the question text, that is, directly provide the answer to the question to the user.

[0137] In another example, as Figure 7 shown, the question text input by the user through the user terminal is received, and the content of the question text is "Why am I not popular?". Several candidate answers corresponding to "Why am I not popular?" are retrieved from the retrieval library. Then, the candidate answers are input into the Q&A model to calculate the matching degree between the question text and each candidate answer. The candidate answers are sorted according to each matching degree, and the top 5 candidate answers are used as the target answers. The target answers are "Why can't my friends see the 30-second short video I synced to the Moments?", "How to cancel the follow?", "Can the Weishi account be deleted?", "How to turn off the watermark?", "Why does my playback fail?" as shown in the figure. Receive the selection instructions given by the user for each of the above target answers, determine the answers to the questions corresponding to the selection instructions, and provide them to the questioner. Among them, "Do you want to know the following questions?" and "None of the above" shown in the figure are default texts pre-configured in the Q&A model, which can be directly provided to the user without participating in the similarity calculation.

[0138] The text processing method provided by the embodiments of the present application, the text processing model is obtained by jointly training the first text processing model and the second text processing model by using the first training data set and the second training data set in different fields. Through joint training, the auxiliary training of the first text processing model can be realized, which helps to improve the accuracy of the trained text processing model in processing data in different fields. Moreover, since the second text processing model is at least one of a masked language model or a named entity recognition model, the training samples do not require sentence pairs and corresponding labels in the extended field, and the training cost of the model will not increase, and domain expansion without increasing costs can be achieved, thus solving the problems of difficult domain migration and low data processing accuracy when the model processes data in the extended field, and improving the use effect of the text processing model in the extended field.

[0139] Same as the principle of the method shown in Figure 1 , an embodiment of the present disclosure also provides a training device 30 for a text processing model, as Figure 8 shown, the training device 30 for the text processing model includes:

[0140] A model construction module 31 for constructing an initial neural network model, where the initial neural network model includes a first text processing model and a second text processing model. The first text processing model includes a cascaded text feature extraction module and a result prediction module, and the second text processing model is connected to the output of the text feature extraction module. The second text processing model includes at least one of a masked language model or a named entity recognition model;

[0141] A data acquisition module 32 for acquiring a first training data set corresponding to the first text processing model and a second training data set corresponding to the second text processing model, where the first training data set and the second training data set belong to different fields;

[0142] A model training module 33 for training the initial neural network model based on the first training data set and the second training data set until a preset training end condition is met, and taking the first text processing model at the end of the training as the final text processing model.

[0143] In a possible implementation, the second text processing model includes a masked language model and a named entity recognition model respectively connected to the output of the text feature extraction module, and the second training data set includes a third training data set corresponding to the masked language model and a fourth training data set corresponding to the named entity recognition model.

[0144] In a possible implementation, the model training module 33 is specifically configured to:

[0145] Jointly and alternately train the first text processing model and the second text processing model based on the first training data set and the second training data set until the loss function corresponding to the first text processing model converges. The training end condition includes the convergence of the loss function corresponding to the first text processing model;

[0146] Wherein, for each training, determine the value of the loss function corresponding to the model branch based on the training data set corresponding to the model branch of the current training and the text processing result of the training data set obtained through the model branch, and adjust the model parameters of the model branch based on the value of the loss function corresponding to the model branch. The model branch is the first text processing model or the second text processing model.

[0147] In a possible implementation, the second text processing model includes a masked language model and a named entity recognition model respectively connected to the output of the text feature extraction module, and the second training data set includes a third training data set corresponding to the masked language model and a fourth training data set corresponding to the named entity recognition model;

[0148] The model training module 33 is used to jointly and alternately train the first text processing model and the second text processing model based on the first training dataset and the second training dataset until the loss function corresponding to the first text processing model converges, and then is used for:

[0149] Jointly and alternately train the first text processing model, the masked language model, and the named entity recognition model based on the first training dataset, the third training dataset, and the fourth training dataset until the loss function corresponding to the first text processing model converges;

[0150] Wherein, the model branch is any one of the first text processing model, the masked language model, or the named entity recognition model.

[0151] In a possible implementation, for the masked language model or the named entity recognition model, for each training, when the model training module 33 determines the value of the loss function corresponding to the model branch based on the training dataset corresponding to the model branch of this training and the text processing results of the training dataset obtained through the model branch, it is used for:

[0152] Input each training sample in the training dataset corresponding to the model branch into the text feature extraction module to obtain the text features of each sample;

[0153] Input the text features of each sample into the model branch to obtain the text processing results of each sample;

[0154] Based on the sample labels of each sample and the text processing results of each sample, determine the value of the loss function corresponding to the model branch.

[0155] In a possible implementation, the second text processing model includes a masked language model. When the data acquisition module 32 acquires the second training dataset corresponding to the second text processing model, it is used for:

[0156] Acquire each first text belonging to the first domain;

[0157] For each first text, mask at least one character in the first text to obtain a second text;

[0158] Use each first text and the second text corresponding to the first text as a training sample in the second training dataset corresponding to the masked language model.

[0159] In a possible implementation, the second text processing model includes a named entity recognition model. When the data acquisition module 32 acquires the second training dataset corresponding to the second text processing model, it is used for:

[0160] Acquire the third text belonging to the second domain;

[0161] For each third text, annotate at least one entity in the third text to obtain a fourth text;

[0162] Each third text and the fourth text corresponding to the third text are used as a training sample of the second training data set.

[0163] The training device of the text processing model of the embodiment of the present disclosure can execute the text processing model provided by the embodiment of the present disclosure. Figure 1 The implementation principle of the corresponding training method of the text processing model is similar. The actions performed by each module in the training device of the text processing model in the embodiment of the present disclosure correspond to the steps in the training method of the text processing model in the embodiment of the present disclosure. For the detailed functional description of each module in the training device of the text processing model, please refer to the description in the corresponding training method of the text processing model shown in the previous text, which will not be repeated here.

[0164] The present application provides a training device for a text processing model, which uses a first training data set and a second training data set in different fields to jointly train the first text processing model and the second text processing model until the preset training end condition is met, thereby obtaining a text processing model that meets the requirements. Through joint training, auxiliary training of the first text processing model can be achieved, which helps to improve the accuracy of the trained text processing model in processing data from different fields. Moreover, since the second text processing model is at least one of a masked language model or a named entity recognition model, the training samples do not require sentence pairs and corresponding labels in the expanded field, and the training cost of the model will not increase, and the field expansion without increasing the cost can be achieved, thereby solving the problem of difficult field migration and low data processing accuracy when the model processes data in the expanded field, and improving the use effect of the text processing model in the expanded field.

[0165] and Figure 4 Based on the same principle as the method shown in , the embodiment of the present disclosure also provides a text processing device 40, such as Figure 9 As shown, the text processing device 40 includes:

[0166] A text acquisition module 41, used to acquire the text to be processed;

[0167] The model processing module 42 is used to input the text to be processed into the text processing model to obtain the processing result;

[0168] A result processing module 43, used for performing corresponding processing based on the processing result;

[0169] Among them, the text processing model is based on this application Figure 1 The method in the corresponding embodiment is trained.

[0170] In a possible implementation, the text to be processed includes text in the field to which the second training data set belongs.

[0171] In a possible implementation, the text processing model is a text matching model, and the text to be processed includes the query text of the requester and several candidate query results corresponding to the query text;

[0172] The model processing module 42 is specifically configured to:

[0173] Input the query text and each candidate query result into the text matching model to obtain the first matching degree between the query text and each candidate query result, and the processing result includes the first matching degree;

[0174] The result processing module 43 is specifically configured to:

[0175] Based on each first matching degree, determine the target query result from each candidate query result and provide the target query result to the requester.

[0176] In a possible implementation, the text processing model includes a question-answering model, and the text to be processed includes the question text of the questioner and several candidate answers corresponding to the question text;

[0177] The model processing module 42 is specifically configured to:

[0178] Input the question text and each candidate answer into the question-answering model to obtain the second matching degree between the question text and each candidate answer, and the processing result includes the second matching degree;

[0179] The result processing module 43 is specifically configured to:

[0180] Based on each second matching degree, determine the target answer from each candidate answer and provide the target answer to the questioner.

[0181] The text processing device of the embodiments of the present disclosure can execute the text processing method provided by the embodiments of the present disclosure, and the implementation principles are similar. The actions performed by each module in the text processing device in the embodiments of the present disclosure correspond to the steps in the text processing method in the embodiments of the present disclosure. For the detailed function descriptions of each module of the text processing device, reference can specifically be made to the descriptions in the corresponding text processing method shown above, and details are not described herein again. Figure 1

[0182] ​The text processing device provided by the embodiment of the present application obtains the text processing model through joint training of a first text processing model and a second text processing model by using a first training data set and a second training data set in different fields. Through joint training, auxiliary training of the first text processing model can be achieved, which helps to improve the accuracy of the trained text processing model in processing data in different fields. Moreover, since the second text processing model is at least one of a masked language model or a named entity recognition model, the training samples do not require sentence pairs and corresponding labels in the extended field, and the training cost of the model will not increase. Domain expansion without increasing costs can be achieved, thereby solving the problems of difficult domain migration and low data processing accuracy when the model processes data in the extended field, and improving the usage effect of the text processing model in the extended field.

[0183] Among them, the training device or text processing device of the text processing model can be a computer program (including program code) running on a computer device. For example, the training device or text processing device of the text processing model is an application software; this device can be used to execute the corresponding steps in the training method or text processing method of the text processing model provided by the embodiment of the present application.

[0184] In some embodiments, the training device or text processing device of the text processing model provided by the embodiment of the present invention can be implemented in a combination of software and hardware. As an example, the training device or text processing device of the text processing model provided by the embodiment of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the training method or text processing method of the text processing model provided by the embodiment of the present invention. For example, the processor in the form of a hardware decoding processor can use one or more application-specific integrated circuits (ASICs, Application Specific Integrated Circuits), DSPs, programmable logic devices (PLDs, Programmable Logic Devices), complex programmable logic devices (CPLDs, Complex Programmable Logic Devices), field-programmable gate arrays (FPGAs, Field-Programmable Gate Arrays) or other electronic components.

[0185] In other embodiments, the training device or text processing device of the text processing model provided by the embodiment of the present invention can be implemented in software. Figure 8 、 Figure 9The training device and text processing device of the text processing model stored in the memory are respectively shown. They can be software in the form of programs and plugins, etc., and include a series of modules. The training device 30 of the text processing model includes a model construction module 31, a data acquisition module 32, and a model training module 33, which are used to implement the training method of the text processing model provided by the embodiments of the present invention. The text processing device 40 includes a text acquisition module 41, a model processing module 42, and a result processing module 43, which are used to implement the text processing method provided by the embodiments of the present invention.

[0186] The above embodiments introduce the training device and text processing device of the text model from the perspective of virtual modules. The following introduces an electronic device from the perspective of physical modules, which is specifically as follows:

[0187] An embodiment of the present application provides an electronic device, such as Figure 10 shown Figure 10 The electronic device 8000 shown in the figure includes: a processor 8001 and a memory 8003. Among them, the processor 8001 and the memory 8003 are connected, such as through a bus 8002. Optionally, the electronic device 8000 may further include a transceiver 8004. It should be noted that in actual applications, the transceiver 8004 is not limited to one, and the structure of the electronic device 8000 does not constitute a limitation to the embodiments of the present application.

[0188] The processor 8001 can be a CPU, a general-purpose processor, a GPU, a DSP, an ASIC, an FPGA or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can implement or execute various exemplary logic blocks, modules and circuits described in combination with the disclosure of the present application. The processor 8001 can also be a combination that realizes computing functions, such as a combination including one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0189] The bus 8002 may include a path for transmitting information between the above components. The bus 8002 can be a PCI bus or an EISA bus, etc. The bus 8002 can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, Figure 10 only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus.

[0190] The memory 8003 can be a ROM or other type of static storage device that can store static information and instructions, a RAM or other type of dynamic storage device that can store information and instructions, or an EEPROM, CD-ROM, or other optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0191] The memory 8003 is used to store the application program code for executing the solution of this application and is controlled by the processor 8001 for execution. The processor 8001 is used to execute the application program code stored in the memory 8003 to implement the content shown in any of the foregoing method embodiments.

[0192] An embodiment of this application provides an electronic device. The electronic device in the embodiment of this application includes: one or more processors; a memory; one or more computer programs, where one or more computer programs are stored in the memory and are configured to be executed by one or more processors. When the one or more programs are executed by the processor, an initial neural network model is constructed. The initial neural network model includes a first text processing model and a second text processing model. Among them, the first text processing model includes a cascaded text feature extraction module and a result prediction module. The second text processing model is connected to the output of the text feature extraction module. The second text processing model includes at least one of a masked language model or a named entity recognition model; obtain a first training data set corresponding to the first text processing model and a second training data set corresponding to the second text processing model. Among them, the first training data set and the second training data set belong to different fields; based on the first training data set and the second training data set, train the initial neural network model until a preset training end condition is met, and use the first text processing model at the end of training as the final text processing model.

[0193] An embodiment of this application provides a computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program runs on a processor, the processor can execute the corresponding content in the foregoing method embodiments.

[0194] According to one aspect of this application, a computer program product or a computer program is provided. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in various alternative implementations of the above text image recognition method.

[0195] It should be understood that although each step in the flowchart of the accompanying drawings is shown sequentially as indicated by the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless there is a clear indication in this document, there is no strict order restriction for the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or sub-steps or stages of other steps.

[0196] The above are only some embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A training method for a text processing model, characterized in that The method includes: Constructing an initial neural network model, the initial neural network model includes a first text processing model and a second text processing model, wherein, the first text processing model includes a cascaded text feature extraction module and a result prediction module, the second text processing model is connected to the output of the text feature extraction module, and the second text processing model includes at least one of a masked language model or a named entity recognition model; Obtaining a first training dataset corresponding to the first text processing model and a second training dataset corresponding to the second text processing model, wherein, the first training dataset and the second training dataset belong to different domains; Based on the first training dataset and the second training dataset, jointly and alternately training the first text processing model and the second text processing model until a preset training end condition is satisfied, and taking the first text processing model at the end of training as the trained text processing model; Wherein, the training end condition includes the convergence of a first loss function, and the joint and alternate training includes multiple training operations, and each training operation includes: Performing one training on the first text processing model based on the first training dataset, determining the value of the first loss function based on the first training dataset and the text processing result of the first training dataset obtained through the first text processing model, and if the first loss function does not converge, adjusting the model parameters of the first text processing model; Performing one training on the second text processing model and the text feature extraction module of the first text processing model based on the second training dataset, determining the value of the second loss function based on the second training dataset and the text processing result of the second training dataset obtained through the text feature extraction module and the second text processing model; if the second loss function does not converge, adjusting the model parameters of the text feature extraction module and the second text processing model.

2. The method according to claim 1, wherein The second text processing model includes a masked language model and a named entity recognition model respectively connected to the output of the text feature extraction module, and the second training dataset includes a third training dataset corresponding to the masked language model and a fourth training dataset corresponding to the named entity recognition model.

3. The method according to claim 1 or 2, characterized in that, The performing one training on the second text processing model and the text feature extraction module of the first text processing model based on the second training dataset includes: Inputting each training sample of the second training dataset into the text feature extraction module of the first text processing model to obtain the text features of each training sample; Inputting the text features of each training sample into the second text processing model to obtain the text processing results of each training sample.

4. The method according to claim 1, characterized in that, The second text processing model includes a masked language model and a named entity recognition model respectively connected to the output of the text feature extraction module, and the second training dataset includes a third training dataset corresponding to the masked language model and a fourth training dataset corresponding to the named entity recognition model; Performing one training on the text feature extraction modules of the second text processing model and the first text processing model based on the second training dataset, and determining the value of the second loss function based on the text processing result of the second training dataset obtained through the text feature extraction module and the second text processing model based on the second training dataset, includes: Performing one training on the masked language model and the text feature extraction module based on the third training dataset, and determining the value of the third loss function based on the text processing result of the third training dataset obtained through the text feature extraction module and the masked language model based on the third training dataset; Performing one training on the named entity recognition model and the text feature extraction module based on the fourth training dataset, and determining the value of the fourth loss function based on the text processing result of the fourth training dataset obtained through the text feature extraction module and the named entity recognition model based on the fourth training dataset.

5. The method according to claim 1, wherein The second text processing model includes a masked language model. Obtaining the second training dataset corresponding to the second text processing model includes: Obtaining each first text belonging to the first domain; For each of the first texts, masking at least one character in the first text to obtain a second text; Taking each of the first texts and the second text corresponding to the first text as a training sample in the second training dataset corresponding to the masked language model.

6. The method according to claim 1, characterized in that The second text processing model includes a named entity recognition model. Obtaining the second training dataset corresponding to the second text processing model includes: Obtaining the third text belonging to the second domain; For each of the third texts, annotating at least one entity in the third text to obtain a fourth text; Taking each of the third texts and the fourth text corresponding to the third text as a training sample in the second training dataset corresponding to the named entity recognition model.

7. A training device for a text processing model, characterized in that, The apparatus includes: A model construction module for constructing an initial neural network model, the initial neural network model including a first text processing model and a second text processing model, wherein the first text processing model includes a cascaded text feature extraction module and a result prediction module, the second text processing model is connected to the output of the text feature extraction module, and the second text processing model includes at least one of a masked language model or a named entity recognition model; A data acquisition module for acquiring a first training dataset corresponding to the first text processing model and a second training dataset corresponding to the second text processing model, wherein the first training dataset and the second training dataset belong to different domains; A model training module for jointly and alternately training the first text processing model and the second text processing model based on the first training dataset and the second training dataset until a preset training end condition is satisfied, and taking the first text processing model at the end of training as the trained text processing model; Among them, the training end condition includes the convergence of the first loss function, and the joint alternating training includes multiple training operations. Each training operation includes: Performing one training on the first text processing model based on the first training dataset, determining the value of the first loss function based on the first training dataset and the text processing result of the first training dataset obtained through the first text processing model. If the first loss function does not converge, adjusting the model parameters of the first text processing model; Performing one training on the text feature extraction module of the second text processing model and the first text processing model based on the second training dataset, determining the value of the second loss function based on the second training dataset and the text processing result of the second training dataset obtained through the text feature extraction module and the second text processing model. If the second loss function does not converge, adjusting the model parameters of the text feature extraction module and the second text processing model.

8. The device according to claim 7, wherein The second text processing model includes a masked language model and a named entity recognition model respectively connected to the output of the text feature extraction module. The second training dataset includes a third training dataset corresponding to the masked language model and a fourth training dataset corresponding to the named entity recognition model.

9. The device according to claim 7 or 8, characterized in that, The model training module is used for: Inputting each training sample of the second training dataset into the text feature extraction module of the first text processing model to obtain the text features of each training sample; Inputting the text features of each training sample into the second text processing model to obtain the text processing results of each training sample.

10. The device according to claim 9, wherein The second text processing model includes a masked language model and a named entity recognition model respectively connected to the output of the text feature extraction module. The second training dataset includes a third training dataset corresponding to the masked language model and a fourth training dataset corresponding to the named entity recognition model; The model training module is used for: Performing one training on the masked language model and the text feature extraction module based on the third training dataset, determining the value of the third loss function based on the third training dataset and the text processing result of the third training dataset obtained through the text feature extraction module and the masked language model; Performing one training on the named entity recognition model and the text feature extraction module based on the fourth training dataset, determining the value of the fourth loss function based on the fourth training dataset and the text processing result of the fourth training dataset obtained through the text feature extraction module and the named entity recognition model.

11. The device according to claim 7, characterized in that, The second text processing model includes a masked language model. The data acquisition module is used for: Acquiring each first text belonging to the first domain; For each of the first texts, masking at least one character in the first text to obtain a second text; Taking each of the first texts and the second text corresponding to the first text as a training sample in the second training dataset corresponding to the masked language model.

12. The device according to claim 7, characterized in that, The second text processing model includes a named entity recognition model, and the data acquisition module is configured to: Obtain a third text belonging to a second domain; For each of the third texts, label at least one entity in the third text to obtain a fourth text; Use each of the third texts and the corresponding fourth text of the third text as a training sample corresponding to the named entity recognition model in the second training dataset.

13. An electronic device, characterized in that, The electronic device includes: One or more processors; A memory; One or more computer programs, wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors, and the one or more computer programs are configured to execute the method according to any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program, and when the computer program runs on a processor, the processor can execute the method according to any one of claims 1 to 6.

Images