Text processing method and device, and method and device for training text processing model

Abstract

Disclosed in embodiments of the present disclosure are a text processing method and device, and a method and device for training a text processing model. The text processing method comprises: acquiring an input text; inputting a text sequence comprising the input text and a prompt text into a text processing model, and acquiring target content, which is outputted by the text processing model on the basis of information indicating the type of a text processing task and corresponds to the input text, wherein the prompt text comprises the information indicating the type of the text processing task; using a feature extraction network to perform feature extraction on the inputted text sequence to obtain a feature representation of a first sub-text sequence comprising the input text; and a pointer network using the feature representation of the first sub-text sequence to predict the target content corresponding to the input text. According to the present disclosure, the pointer network is introduced into the text processing model, and by constructing the text sequence of "the input text and the prompt text", a natural language understanding task is unified into an extraction paradigm, thereby improving the information extraction effect of a general text processing model on information extraction tasks.

Description

Text processing method, method and device for training text processing model

[0001] Cross-reference

[0002] This disclosure claims priority to the Chinese patent application filed with the China Patent Office on July 4, 2023, with application number 202310814286.6 and invention name “Text processing method, method and device for training text processing model”, the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0003] The present disclosure relates to the technical field of natural language processing, and in particular to a text processing method, a method for training a text processing model, and a device. Background Art

[0004] Natural Language Processing (NLP) is a key area of ​​research in computer science and artificial intelligence. It uses computer technology to analyze, understand, and process natural human language. NLP technology enables tasks such as text classification and information extraction, and is applicable to a wide range of scenarios.

[0005] Taking information extraction as an example, information extraction refers to extracting specific types of targets from text, thereby automatically classifying, extracting and reconstructing massive amounts of content, such as entity extraction, entity relationship extraction, event extraction, attribute sentiment extraction, text domain classification and many other types. However, when faced with specific types of tasks, most existing models need to be trained separately for specific types of tasks, and the transferability is poor. To solve this problem, a general text processing model has been proposed, that is, using a unified text processing model to solve various types of text understanding tasks. However, the current general text processing model works well for non-information extraction tasks such as text classification, but the effect is very poor for information extraction tasks.

[0006] Summary of the Invention

[0007] In view of this, the present disclosure provides a text processing method, a method and an apparatus for training a text processing model, so as to improve the information extraction effect for information extraction tasks.

[0008] The present disclosure provides the following solutions:

[0009] In a first aspect, a text processing method is provided, the method comprising:

[0010] Get input text;

[0011] Inputting a text sequence including the input text and the prompt text into a text processing model, and obtaining target content corresponding to the input text output by the text processing model based on the information indicating the type of text processing task, wherein the prompt text includes the information indicating the type of text processing task, and the text processing model includes a feature extraction network and a pointer network;

[0012] Using the feature extraction network, extracting features from an input text sequence to obtain a feature representation of a first subtext sequence including the input text;

[0013] The pointer network is used to predict target content corresponding to the input text using the feature representation of the first subtext sequence.

[0014] According to an achievable method in an embodiment of the present disclosure, the feature extraction network includes: a first extraction network, a second extraction network and a feature fusion network;

[0015] Using the first extraction network, feature extraction is performed on a first subtext sequence containing the input text in the input text sequence to obtain a first feature representation of each element token in the first subtext sequence;

[0016] Using the second extraction network, feature extraction is performed on a second subtext sequence containing the prompt text in the input text sequence to obtain a second feature representation of each token in the second subtext sequence;

[0017] Using the feature fusion network, the first feature representation and the second feature representation are spliced ​​to obtain a spliced ​​sequence, the spliced ​​sequence is self-attention processed to obtain a feature representation of each token in the spliced ​​sequence, and the third feature representation of each token in the first sub-text sequence is obtained from the feature representation of each token in the spliced ​​sequence to constitute the feature representation of the first sub-text sequence.

[0018] According to an achievable manner in an embodiment of the present disclosure, obtaining the input text includes: obtaining the input text and information indicating the type of the text processing task from a text processing instruction;

[0019] In response to the information indicating the type of text processing task indicating a round of text processing task, the information indicating the type of text processing task is used to construct a prompt text, and the step of inputting the sequence including the input text and the prompt text into the text processing model is performed.

[0020] According to an implementable manner of an embodiment of the present disclosure, in response to the information indicating the type of text processing task indicating multiple rounds of text processing tasks, in a first round of text processing tasks, a first round of prompt text is constructed using the information indicating the type of the first round of text processing task, and a text sequence including the input text and the first round of prompt text is input into a text processing model;

[0021] In the other rounds of text processing tasks among the multi-round text processing tasks except the first round of text processing task, the prompt text of the previous round of the multi-round text processing task, the target content output by the text processing model in the previous round, and the information indicating the type of the current round of text processing task of the multi-round text processing task are used to construct the prompt text of the current round of the multi-round text processing task, and the text sequence including the input text and the prompt text of the current round is input into the text processing model.

[0022] In a second aspect, an information extraction method is provided, the method comprising:

[0023] Get input text;

[0024] Inputting a text sequence including an input text and a prompt text into a text processing model, and obtaining target content corresponding to the target type extracted by the text processing model from the input text, wherein the prompt text includes information indicating the target type to be extracted, and the text processing model includes a feature extraction network and a pointer network;

[0025] Using the feature extraction network, extracting features from an input text sequence to obtain a feature representation of a first subtext sequence including the input text;

[0026] The target content corresponding to the target type in the input text is predicted by using the feature representation of the first subtext sequence through the pointer network.

[0027] According to an achievable method in an embodiment of the present disclosure, the target type is an entity type, and the target content is an entity word; or,

[0028] The target type is an entity relationship type, and the target content is an entity pair corresponding to the entity relationship type in the input text; or

[0029] The target type is an event type, and the target content is an event trigger word and / or parameter word corresponding to the event type in the input text; or

[0030] The target type is an attribute type and an emotion type, and the target content is an attribute word corresponding to the attribute type and an emotion word corresponding to the emotion type in the input text; or

[0031] The target type is the answer corresponding to the question, and the target content is the answer segment corresponding to the question in the input text.

[0032] In a third aspect, a text classification method is provided, the method comprising:

[0033] Get input text;

[0034] Inputting a text sequence including the input text, candidate category labels, and prompt text into a text processing model, and obtaining a category label to which the input text belongs, determined by the text processing model from the candidate category labels based on the prompt text, wherein the prompt text includes information indicating a classification task type, and the text processing model includes a feature extraction network and a pointer network;

[0035] Using the feature extraction network, extracting features from an input text sequence to obtain a feature representation of a first subtext sequence including the input text and the candidate category labels;

[0036] The category label to which the input text belongs is predicted by using the feature representation of the first subtext sequence through the pointer network.

[0037] According to an implementable manner in an embodiment of the present disclosure, the classification task type is text field classification, and the candidate category label is a text field label; or,

[0038] The classification task type is sentiment classification, and the candidate category label is a sentiment category label; or

[0039] The classification task type is text similarity classification, the input text is a text pair, and the candidate category label is a text similarity label; or,

[0040] The classification task type is text relationship classification, the input text is a text pair, and the candidate category label is a text relationship label; or

[0041] The classification task type is a multiple-choice question-answering task, and the candidate category labels are answer option labels.

[0042] In a fourth aspect, a text processing method is provided, which is executed by a cloud server, and the method includes:

[0043] Get input text from the user's device;

[0044] Inputting a text sequence including the input text and a prompt text into a text processing model, and obtaining target content corresponding to the input text output by the text processing model based on the prompt text, wherein the prompt text includes information indicating a type of text processing task, and the text processing model includes a feature extraction network and a pointer network;

[0045] sending the target content or service content corresponding to the target content to the user device;

[0046] Using the feature extraction network, extracting features from an input text sequence to obtain a feature representation of a first subtext sequence including the input text;

[0047] The pointer network is used to predict target content corresponding to the input text using the feature representation of the first subtext sequence.

[0048] In a fifth aspect, a method for training a text processing model is provided, the method comprising:

[0049] Acquire training data comprising a plurality of training samples, wherein the training samples comprise text samples and target samples annotated for the text samples;

[0050] A text processing model is trained by using a text sequence including a text sample and a prompt text as input, and using a target sample labeled for the text sample as a target output of the text processing model, wherein the prompt text includes information indicating a type of text processing task corresponding to the target sample, and the text processing model includes a feature extraction network and a pointer network;

[0051] Using the feature extraction network, extracting features from an input text sequence to obtain a feature representation of a first subtext sequence including the text sample;

[0052] The pointer network is used to predict target content corresponding to the text sample using the feature representation of the first sub-text sequence.

[0053] In a sixth aspect, a text processing device is provided, the device comprising:

[0054] A text acquisition unit configured to acquire input text;

[0055] a text processing unit configured to input a text sequence including the input text and a prompt text into a text processing model, and obtain target content corresponding to the input text output by the text processing model based on the prompt text, wherein the prompt text includes information indicating a type of text processing task, and the text processing model includes a feature extraction network and a pointer network;

[0056] Using the feature extraction network, extracting features from an input text sequence to obtain a feature representation of a first subtext sequence including the input text;

[0057] The pointer network is used to predict target content corresponding to the input text using the feature representation of the first subtext sequence.

[0058] In a seventh aspect, a device for training a text processing model is provided, the device comprising:

[0059] A sample acquisition unit is configured to acquire training data including a plurality of training samples, wherein the training samples include text samples and target samples annotated for the text samples;

[0060] a model training unit configured to take a text sequence including a text sample and a prompt text as an input of a text processing model, and take a target sample annotated for the text sample as a target output of the text processing model, to train the text processing model, wherein the prompt text includes information indicating a type of text processing task corresponding to the target sample, and the text processing model includes a feature extraction network and a pointer network;

[0061] Using the feature extraction network, extracting features from an input text sequence to obtain a feature representation of a first subtext sequence including the text sample;

[0062] The pointer network is used to predict target content corresponding to the text sample using the feature representation of the first sub-text sequence.

[0063] According to an eighth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the steps of the method described in any one of the first to fifth aspects.

[0064] According to the ninth aspect, a computer program product is provided, comprising a non-volatile computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of the first to fifth aspects.

[0065] According to a tenth aspect, a computer program is provided, which, when executed by a processor, implements the steps of the method described in any one of the first to fifth aspects.

[0066] According to the eleventh aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the steps of the method described in any one of the first to fifth aspects are implemented.

[0067] According to a twelfth aspect, there is provided an electronic device, comprising:

[0068] one or more processors; and

[0069] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the first to fifth aspects above.

[0070] According to the specific embodiments provided by the present disclosure, the present disclosure discloses the following technical effects:

[0071] 1) The present disclosure introduces a pointer network that excels at extracting information from input text sequences into a text processing model. By constructing a text sequence of "input text + prompt text", the text processing tasks based on natural language understanding are unified into an extraction paradigm, thereby improving the information extraction effect of the general text processing model on information extraction tasks.

[0072] 2) The present disclosure adopts a twin network in the feature extraction network, that is, a first extraction network and a second extraction network are used to extract features from a first subtext sequence containing the input text and a second subtext sequence containing the prompt text, respectively, and then perform feature fusion. This approach allows the first feature representation of each token containing the first subtext sequence of the text sample or input text to be reused during model training or text processing, for the same text sample or input text, thereby saving computing resources and improving efficiency.

[0073] 3) For complex text processing tasks, by configuring information indicating the types of multiple rounds of text processing tasks in the text processing instructions, the prompt text for each round is constructed based on the information of the previous round of text processing task types and the obtained target content, so as to utilize the text processing model to extract multiple rounds of target content, thereby realizing complex text processing tasks.

[0074] 4) The text processing model provided by the present invention can be used not only as a general model for information extraction, but also as a general model for text classification. Different prompt texts can be configured according to different application scenarios to achieve various information extraction tasks such as entity extraction, entity relationship extraction, event extraction, attribute sentiment extraction, extraction-based question and answer tasks, as well as various text classification tasks such as text domain classification, sentiment classification, text matching, inference of text relations, and selective question and answer tasks, all of which can achieve good results.

[0075] Of course, any product implementing the present disclosure does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0077] FIG1 is a diagram of a system architecture applicable to an embodiment of the present disclosure;

[0078] FIG2 is a flowchart of a text processing method provided by an embodiment of the present disclosure;

[0079] FIG3 is a schematic diagram of the structure of a text processing model provided by an embodiment of the present disclosure;

[0080] FIG4 is a flow chart of an information extraction method according to an embodiment of the present disclosure;

[0081] FIG5 is a flow chart of a text classification method provided by an embodiment of the present disclosure;

[0082] FIG6 is a flow chart of a method for training a text processing model according to an embodiment of the present disclosure;

[0083] FIG7 is a schematic block diagram of a text processing device provided by an embodiment of the present disclosure;

[0084] FIG8 is a schematic block diagram of a training text processing model provided in an embodiment of the present disclosure;

[0085] FIG9 is a schematic block diagram of an electronic device provided by an embodiment of the present disclosure;

[0086] FIG10 is a schematic diagram showing the structure of a processor according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0087] The following will be combined with the accompanying drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the embodiments described are only part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of the present disclosure.

[0088] The terms used in the embodiments of the present disclosure are for the purpose of describing specific embodiments only and are not intended to limit the present disclosure. The singular forms "a," "an," "the," and "the" used in the embodiments of the present disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0089] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0090] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0091] To facilitate understanding of the present disclosure, we first briefly describe the system architecture to which the present disclosure is applicable. Figure 1 illustrates an exemplary system architecture to which embodiments of the present disclosure can be applied. As shown in Figure 1, the system architecture includes a model training device for establishing a text processing model in an offline manner, and a text processing device for performing online text processing on input text.

[0092] Among them, after obtaining the training data, the model training device can use the method provided in the embodiment of the present disclosure to perform model training to obtain a text processing model.

[0093] The text processing device uses the established text processing model to process the input text and obtain the target content corresponding to the input text. For example, it extracts target content of a specified target type from the input text, or determines the category label to which the input text belongs. This will be described in detail in subsequent embodiments.

[0094] The model training device and text processing device can be installed on separate servers, on the same server or server group, or on separate or shared cloud servers. A cloud server, also known as a cloud computing server or cloud host, is a host product within the cloud computing service ecosystem. It addresses the management difficulties and poor scalability of traditional physical hosts and virtual private servers (VPS). The model training device and text processing device can also be installed on a computer terminal with strong computing power.

[0095] It should be noted that, in addition to performing text processing online, the above-mentioned text processing device can also perform text processing offline, for example, performing text processing on batches of input texts separately and storing them in a database for subsequent call or query.

[0096] It should be understood that the number of model training devices, text processing devices, and text processing models in Figure 1 is merely illustrative. Any number of model training devices, text processing devices, and text processing models may be provided as needed.

[0097] FIG2 is a flow chart of a text processing method provided by an embodiment of the present disclosure, which can be executed by the text processing device in the system architecture shown in FIG1. ​​As shown in FIG2, the method may include the following steps:

[0098] Step 202: Get input text.

[0099] Step 204: Input a text sequence including an input text and a prompt text into a text processing model, and obtain a target content corresponding to the input text output by the text processing model based on information indicating the type of text processing task, wherein the prompt text includes information indicating the type of text processing task, and the text processing model includes a feature extraction network and a pointer network; utilize the feature extraction network to perform feature extraction on the input text sequence to obtain a feature representation of a first subtext sequence including the input text; and utilize the pointer network to predict the target content corresponding to the input text using the feature representation of the first subtext sequence.

[0100] It can be seen from the above process that the present disclosure introduces a pointer network that is good at extracting information from input text sequences into the text processing model. By constructing a text sequence of "input text + prompt text", the text processing tasks are unified into an extraction paradigm, thereby improving the information extraction effect of the general text processing model for information extraction tasks.

[0101] It should be noted that the terms "first" and "second" in this disclosure do not restrict size, order, or quantity, but are merely used to distinguish between them. For example, "first subtext sequence" and "second subtext sequence" are used to distinguish between two subtext sequences. Another example is "first feature representation" and "second feature representation" are used to distinguish between two feature representations.

[0102] The following describes each step in the above method flow. First, the above step 202, ie, "obtaining input text," is described in detail in conjunction with an embodiment.

[0103] The input text involved in the embodiments of the present disclosure may be text obtained from a user device. For example, when a user wishes to perform a specific text processing task on the text, such as an information extraction task or a text classification task, the user device may upload the text to a text processing device on a server side. The text processing device then uses the text as input text and determines the target content corresponding to the input text using the methods provided in the embodiments of the present disclosure. For example, target content of a specific target type may be extracted from the input text, or a category label corresponding to the input text may be determined.

[0104] The input text involved in the embodiments of the present disclosure may also be text obtained from a database. For example, if a specific text processing task needs to be performed on a batch of texts in the database, each text obtained from the database can be used as the input text, and the target content corresponding to the input text can be determined using the method provided in the embodiments of the present disclosure.

[0105] The input text may be a sentence, a paragraph or document composed of sentences, or a text unit of smaller granularity, such as a phrase.

[0106] The above step 204, namely "inputting a text sequence including input text and prompt text into a text processing model, and obtaining target content corresponding to the input text output by the text processing model based on the prompt text" is described in detail below in conjunction with an embodiment.

[0107] In the disclosed embodiment, the prompt text mainly includes information indicating the type of text processing task, which is used to help the text processing model understand the processing of the input text. Different prompt texts can be used according to different text processing task types. For example, in the information extraction task scenario, the prompt text includes information indicating the target type to be extracted. For another example, in the text classification task scenario, the prompt text includes information indicating the classification task type. Furthermore, in the text classification task scenario, the above-mentioned text sequence may further include candidate type labels, and when performing text classification, one of the candidate type labels is selected as the target category label. This will be described in detail in subsequent embodiments.

[0108] The text processing model in the embodiment of the present disclosure can output the target content corresponding to the input text according to the information indicating the type of text processing task in the prompt text when a text sequence including input text and prompt text is input. The prompt text can adopt a preset prompt text, or can obtain the information including the type of text processing task from the text processing instruction to construct the prompt text. For example, for a fixed text processing task, the corresponding prompt text can be pre-set for the text processing task. For another example, if the user wants to implement a flexible text processing task, the text processing instruction can be sent, and the text processing instruction follows the preset grammatical rules. According to the preset grammatical rules, the input text and the information indicating the type of text processing task can be parsed and obtained, and the information indicating the type of text processing task can be used to construct the prompt text, which will be described in detail with examples in the subsequent embodiments.

[0109] The text processing model can include a feature extraction network and a pointer network.

[0110] The feature extraction network is used to extract features from the input text sequence to obtain a feature representation of the first subtext sequence including the input text.

[0111] Through the pointer network, the feature representation of the first subtext sequence is used to predict the target content corresponding to the input text.

[0112] As one of the feasible ways, the feature extraction network can use a traditional pre-trained language model to perform feature encoding on each token (element) in the input text sequence, obtain the feature representation of each token, and then obtain the feature representation of the first sub-text sequence containing the input text.

[0113] As another possible implementation, the present disclosure provides a preferred feature extraction network. This feature extraction network adopts a Siamese Network architecture, namely, a "Siamese Network + Feature Fusion Network" format. As shown in Figure 3, the feature extraction network may include a first extraction network, a second extraction network, and a feature fusion network.

[0114] The first extraction network is used to perform feature extraction on the first subtext sequence of the input text in the input text sequence to obtain a first feature representation of each token in the first subtext sequence.

[0115] The second extraction network is used to perform feature extraction on the second subtext sequence including the prompt text in the input text sequence to obtain a second feature representation of each token in the second subtext sequence.

[0116] Using the feature fusion network, the first feature representation and the second feature representation are spliced ​​to obtain a spliced ​​sequence. The spliced ​​sequence is processed with self-attention to obtain the feature representation of each token in the spliced ​​sequence. Then, from the feature representation of each token in the spliced ​​sequence, the third feature representation of each token in the first sub-text sequence is obtained to form the feature representation of the first sub-text sequence.

[0117] For example, the feature extraction network uses a pre-trained language model to initialize its parameters. The first k layers of the pre-trained language model serve as two sub-branches, each of which contains the initialization parameters of the first k layers of the pre-trained language model. One branch serves as the first extraction network to primarily extract features from the input text, and the other branch serves as the second extraction network to primarily extract features from the prompt text. The last Kk layers of the pre-trained language model serve as a feature fusion network to fuse the feature representations obtained from the two sub-branches. K is the total number of layers of the pre-trained language model, which is a preset positive integer. k is a positive integer less than K.

[0118] The first subtext sequence mentioned above mainly contains the input text. For example, for the text processing instruction:

[0119] {"text":"Xiaofang is so pretty","schema":{"name":null}}

[0120] Among them, "text" is information indicating the input text, "schema" is information indicating the prompt text, and "name" is information indicating the target type to be extracted, that is, extracting the name of a person.

[0121] In this example, the first subtext sequence may be: "Xiaofang is really pretty", and the second subtext sequence may be: "name of a person".

[0122] The first extraction network and the second extraction network may first perform embedding processing on each token in the first subtext sequence and the second subtext sequence, respectively. The embedding processing may include at least: word embedding and position embedding. Word embedding encodes each token into a word vector to obtain a word vector representation. Position embedding encodes the position of each token in the text sequence to be predicted to obtain a position representation.

[0123] The above-mentioned information extraction model in the embodiment of the present disclosure can be obtained by training based on a pre-trained language model, using pre-trained language models such as Bidirectional Encoder Representation from Transformers (BERT) and an autoregressive model (XLNet) that implements bidirectional contextual information by arranging language models as a model framework, and further training is performed on this basis. Among them, BERT is a bidirectional pre-trained language model that uses a transformer encoder as a model structure. BERT can make good use of contextual information for feature learning. XLNet is a model similar to BERT and is a more generalized autoregressive pre-trained model.

[0124] Each token in a text sequence (the first subtext sequence or the second subtext sequence) refers to the elements that make up the text sequence. For a text sequence, if the text sequence is segmented into character or word sequences, then the characters or words, as well as the start character and separator in the text sequence, are all tokens.

[0125] Specifically, a feature fusion network is used to concatenate the first and second feature representations of each token in the first subtext sequence to obtain a concatenated sequence. The concatenated sequence is then input into the Transformer layer for self-attention processing to obtain the feature representation of each token in the concatenated sequence. Since the Transformer layer and the self-attention mechanism are both currently available technologies, they will not be described in detail here. After obtaining the feature representation of each token in the concatenated sequence, the third feature representation of each token corresponding to the first subtext sequence is obtained to form the feature representation of the first subtext sequence, which is then provided to the pointer network for prediction.

[0126] The above-mentioned text processing model involved in the embodiments of the present disclosure actually adopts an encoder-decoder architecture, where the above-mentioned feature extraction network corresponds to the encoder and the pointer network corresponds to the decoder.

[0127] In traditional sequence-to-sequence (Seq2Seq) models, the number of target contents output by the decoder is fixed, usually equal to the size of the dictionary, that is, each token in the output sequence is taken from a preset dictionary. However, for extraction tasks, the output sequence is not taken from a preset dictionary, but from the input text, that is, it is a continuous segment of the input text. For example, entity extraction is to extract entity words of a specified entity type from the input text, and event extraction is to extract emotion objects and / or emotion words corresponding to a specified emotion type from the input text, and so on. The Pointer Network uses an attention mechanism, which uses attention as a pointer to a single token in the input text, so that the output sequence can be obtained directly from the input text, avoiding the use of an output vocabulary. In other words, the tokens of the output sequence all come from the input text and are a subset of the tokens contained in the input text (that is, the output sequence is a segment of the input text).

[0128] Assume that the feature representation of the first subtext sequence output by the feature extraction network is (e1, e2,…, e n ), where e j The pointer network uses the feature representation of the first subtext sequence to perform decoding, and the latent vector of the i-th Token in the first subtext sequence is d i , after the following attention processing: p(C i |C1,…,C i-1 ,Ρ)=softmax(u i ) (2)

[0129] in, To e j and d i The vector representation obtained after attention processing. T represents the transpose of v. tanh() is the hyperbolic tangent function. v, W1, and W2 are all model parameters learned by the text processing model during training. u i for Ρ represents the first subtext sequence, p(C i |C1,…,C i-1 ,Ρ) represents the ith Token in the first subtext sequence, namely C i The corresponding output probability.

[0130] Then, the output token is determined as the target content based on the output probability corresponding to each token in the first subtext sequence. For example, consecutive tokens with an output probability greater than or equal to a preset probability threshold are output as the target content. Another example is to output the top N consecutive tokens with output probabilities as the target content, where N is a preset positive integer, and so on. It can be seen that the target content ultimately output is the token in the first subtext sequence.

[0131] In addition, for some complex text processing tasks, a single round of text processing by the text processing model is not sufficient, and multiple rounds of text processing tasks can be used to achieve this. Specifically, after obtaining the input text and the type of knowledge text processing task from the text processing instruction, in response to the information indicating the text processing task type indicating a single round of text processing, prompt text is constructed using the information indicating the text processing task type, and the aforementioned step of inputting the sequence including the input text and prompt text into the text processing model is executed. The text processing model then outputs the target content corresponding to the input text based on the information indicating the text processing task type, and the process ends.

[0132] In response to the information indicating the type of text processing task indicating multiple rounds of text processing tasks, in the first round of text processing tasks, the information indicating the type of the first round of text processing task is used to construct the prompt text of the first round, and the text sequence including the input text and the prompt text of the first round is input into the text processing model, and the target content output by the text processing model in the first round of text processing is obtained.

[0133] In the text processing tasks of other rounds except the first round in the multi-round text processing task, the prompt text of the previous round of the multi-round text processing task, the target content output by the text processing model in the previous round, and the information indicating the type of the text processing task of the current round of the multi-round text processing task are used to construct the prompt text of the current round of the multi-round text processing task. For example, the prompt text of the previous round, the target content output by the text processing model in the previous round, and the information indicating the type of the text processing task of the current round are spliced ​​to obtain the prompt text of the current round. Then, the text sequence including the input text and the prompt text of the current round is input into the text processing model to obtain the target content output by the text processing model in the current round of text processing. The process of this multi-round processing will be described by example in the subsequent embodiments.

[0134] The above-mentioned text processing method provided by the embodiment of the present disclosure can be applied to various application scenarios. The text processing method provided by the present disclosure is described in detail below, taking information extraction and text classification as examples.

[0135] FIG4 is a flow chart of an information extraction method provided by an embodiment of the present disclosure. As shown in FIG4 , the method may include the following steps:

[0136] Step 402: Get input text.

[0137] In information extraction scenarios, if you need to extract target content of a specified target type from text, you use that text as input. The input text can be sentences, paragraphs composed of sentences, documents, or smaller text units such as phrases.

[0138] Step 404: Input a text sequence including an input text and a prompt text into a text processing model, and obtain target content corresponding to a target type extracted from the input text by the text processing model, wherein the prompt text includes information indicating a target type to be extracted, and the text processing model includes a feature extraction network and a pointer network; utilize the feature extraction network to perform feature extraction on the input text sequence to obtain a feature representation of a first subtext sequence including the input text; and utilize the pointer network to predict the target content corresponding to the target type in the input text using the feature representation of the first subtext sequence.

[0139] In the information extraction scenario, the prompt text contains information indicating the target type to be extracted. The following specific applications are used as examples to describe each:

[0140] Application scenario 1: entity extraction.

[0141] In this application scenario, the input text and the prompt text containing information indicating the entity type to be extracted can be concatenated to obtain a text sequence input text processing model. In addition to the input text and the prompt information, the text sequence can also contain information indicating the input text and information indicating the prompt text.

[0142] For example, the text processing instructions can be as follows:

[0143] {"text":"Xiaofang is so pretty","schema":{"name":null}}

[0144] Among them, "text" is information indicating the input text, "schema" is information indicating the prompt text, and "name" is information indicating the target type to be extracted, that is, extracting the name of a person.

[0145] In this example, the first subtext sequence may be: "Xiaofang is really pretty", and the second subtext sequence may be: "name of a person".

[0146] The first subtext sequence is input into the first extraction network shown in Figure 3 to obtain the first feature representation of each token in the first subtext sequence. The second subtext sequence is input into the second extraction network shown in Figure 3 to obtain the second feature representation of each token in the second subtext sequence. The feature fusion network concatenates the first feature representation of each token in the first subtext sequence with the second feature representation of each token in the second subtext sequence to obtain a concatenated sequence. The concatenated sequence is subjected to self-attention processing to obtain the feature representation of each token in the concatenated sequence. The third feature representation of each token in the first subtext sequence is obtained from the feature representation of each token in the concatenated sequence to form the feature representation of the first subtext sequence.

[0147] The pointer network uses the feature representation of the first subtext sequence to predict the output probability of each token in the first subtext sequence. Based on this output probability, the entity word to be extracted from the first subtext sequence is actually the entity word corresponding to the target type in the input text. For example, from the input text "Xiao Fang is really beautiful" in the above example, the entity word "Xiao Fang" corresponding to the name of a person is extracted.

[0148] Application scenario 2: entity relationship extraction.

[0149] In this application scenario, the input text and the prompt text containing information indicating the entity relationship type to be extracted can be spliced ​​to obtain a text sequence input text processing model. In addition to the input text and the prompt information, the text sequence can also contain information indicating the input text and information indicating the prompt text.

[0150] When performing entity relationship extraction in this application scenario, it is usually necessary to limit the type of one of the entities. This requires multiple rounds of information extraction, that is, combining entity extraction and entity relationship extraction. For example, the text processing instructions can be as follows:

[0151] {"text":"Although Xiao Ming's home is not far away, he has lived in the AAA University dormitory for a long time","schema":{"Person"{"Residence(Location)":null}}}

[0152] Among them, "text" indicates the input text, and "schema" indicates the prompt text. Multiple rounds of text processing types are parsed from the text processing instructions. First, the first-round prompt text is constructed using the target type information indicated in the first round. That is, the first-round prompt text is constructed using "person". "Person" indicates the entity type that the model first wants to extract. The text sequence containing the input text and the first-round prompt text is input into the text processing model. The target content output by the text processing model in the first round is the entity word of the entity type "person": "Xiao Ming".

[0153] In the second round of text processing, the prompt text from the first round, the target content obtained from the first round, and the target type information indicated in the second round are used to construct the prompt text for the second round: "Person: Xiao Ming, Residence." The text sequence containing the input text and the second round of prompt text is input into the text processing model. The target content output by the text processing model in the second round is the "Place" entity type that has a "Residence" relationship with "Person: Xiao Ming," that is, "AAA University Dormitory."

[0154] In each round of text processing, the first subtext sequence is input into the first extraction network shown in Figure 3 to obtain the first feature representation of each token in the first subtext sequence. The second subtext sequence is input into the second extraction network shown in Figure 3 to obtain the second feature representation of each token in the second subtext sequence. Using the feature fusion network, the first feature representation of each token in the first subtext sequence and the second feature representation of each token in the second subtext sequence are concatenated to obtain a concatenated sequence. The concatenated sequence is subjected to self-attention processing to obtain the feature representation of each token in the concatenated sequence. From the feature representation of each token in the concatenated sequence, the third feature representation of each token in the first subtext sequence is obtained to form the feature representation of the first subtext sequence.

[0155] The pointer network uses the feature representation of the first subtext sequence to predict the output probability of each token in the first subtext sequence and, based on this output probability, determines the entity words to be extracted from the first subtext sequence. Through two rounds of entity word extraction, an entity pair is obtained. This entity pair is actually the two entity words in the input text that match the aforementioned entity relationship type. For example, from the input text "Xiao Ming's home is not far away, but he has lived in the AAA University dormitory for a long time" in the above example, the entity pair corresponding to the entity relationship "residence" is: "Xiao Ming" and "AAA University dormitory."

[0156] Application scenario 3: event extraction.

[0157] In this application scenario, the input text and the prompt text containing information indicating the event type to be extracted can be spliced ​​to obtain a text sequence input text processing model. In addition to the input text and the prompt information, the text sequence can also contain information indicating the input text and information indicating the prompt text.

[0158] For example, the text processing instructions can be as follows:

[0159] {“text”: “Mr. Li fell down this morning”, “schema”: {“action (event trigger word)”: {“subject”: null}}}

[0160] Among them, "text" is information indicating the input text, and "schema" is information indicating the prompt text. Multiple rounds of text processing types are parsed from the text processing instructions. First, the first round of prompt text is constructed using the target type information to be extracted in the first round. That is, the first round of prompt text is constructed using "action behavior (event trigger word)". "Action behavior (event trigger word)" indicates the type of event trigger word that the model should first extract. The text sequence containing the input text and the first round of prompt text is input into the text processing model. The target content output by the text processing model in the first round is the event trigger word that matches the action behavior: "fall down".

[0161] In the second round of text processing, the prompt text from the first round, the target content obtained in the first round, and the target type information indicated in the second round are used to construct the second round of prompt text: "Action (event trigger word): fell, subject." The text sequence containing the input text and the second round of prompt text is input into the text processing model. The target content output by the text processing model in the second round is the "subject" who performed the "action": Lao Li.

[0162] In each round of text processing, the first subtext sequence is input into the first extraction network shown in Figure 3 to obtain the first feature representation of each token in the first subtext sequence. The second subtext sequence is input into the second extraction network shown in Figure 3 to obtain the second feature representation of each token in the second subtext sequence. Using the feature fusion network, the first feature representation of each token in the first subtext sequence and the second feature representation of each token in the second subtext sequence are concatenated to obtain a concatenated sequence. The concatenated sequence is subjected to self-attention processing to obtain the feature representation of each token in the concatenated sequence. From the feature representation of each token in the concatenated sequence, the third feature representation of each token in the first subtext sequence is obtained to form the feature representation of the first subtext sequence.

[0163] The pointer network uses the feature representation of the first subtext sequence to predict the output probability of each token in the first subtext sequence, and determines the event trigger word and / or parameter word extracted from the first subtext sequence based on the output probability. Parameter words can include the event subject, event object, time of occurrence, location of occurrence, etc. For example, from the input text "Lao Li fell down this morning" in the above example, the trigger word corresponding to the event trigger word "action behavior" is: "fell down", and the event subject is "Lao Li".

[0164] Application scenario 4: attribute sentiment extraction.

[0165] In this application scenario, the input text and the prompt text containing information indicating the attribute type and sentiment type to be extracted can be concatenated to obtain a text sequence input text processing model. In addition to the input text and prompt information, the text sequence can also contain information indicating the input text and information indicating the prompt text.

[0166] In this application scenario, it is usually necessary to extract attribute words first, and then extract sentiment words of a specific sentiment type, so multiple rounds of information extraction can be performed. For example, the text processing instructions can be as follows:

[0167] {“text”: “Although this dress is a little small, the style is really nice”, “schema”: {“appearance style (attribute word)”: {“positive emotion (emotional word)”: null}}}

[0168] Among them, "text" is the information indicating the input text, and "schema" is the information indicating the prompt text. Multiple rounds of text processing types are parsed from the text processing instructions. First, the first round of prompt text is constructed using the target type information to be extracted in the first round, that is, the first round of prompt text is constructed using "appearance style (attribute word)". "Appearance style (attribute word)" is information indicating the attribute type that the model should first extract. The text sequence containing the input text and the first round of prompt text is input into the text processing model. The target content output by the text processing model in the first round is the attribute word with the attribute type of appearance style: "template".

[0169] In the second round of text processing, the prompt text from the first round, the target content obtained in the first round, and the target type information indicated in the second round are used to construct the second round of prompt text: "Appearance style (attribute word): layout, positive sentiment (emotional word)." The text sequence containing the input text and the second round of prompt text is input into the text processing model. When the target content output in the second round is "appearance style" and the attribute word is "layout", the text processing model will generate a sentiment word expressing the "positive" sentiment type.

[0170] The first subtext sequence is input into the first extraction network shown in Figure 3 to obtain the first feature representation of each token in the first subtext sequence. The second subtext sequence is input into the second extraction network shown in Figure 3 to obtain the second feature representation of each token in the second subtext sequence. Using the feature fusion network, the first feature representation of each token in the first subtext sequence and the second feature representation of each token in the second subtext sequence are spliced ​​to obtain a spliced ​​sequence. The spliced ​​sequence is subjected to self-attention processing to obtain the feature representation of each token in the spliced ​​sequence. From the feature representation of each token in the spliced ​​sequence, the third feature representation of each token in the first subtext sequence is obtained to form the feature representation of the first subtext sequence.

[0171] The pointer network uses the feature representation of the first subtext sequence to predict the output probability of each token in the first subtext sequence. Based on this output probability, it determines the attribute words and sentiment words to be extracted from the first subtext sequence. For example, from the input text "This dress is a little small, but the cut is really nice," the attribute word corresponding to "appearance and style" extracted is "cut," and the positive sentiment word is "nice."

[0172] Application scenario 5: Extractive question answering task.

[0173] In this application scenario, the input text and the prompt text containing the answer to the question to be extracted can be concatenated to obtain a text sequence input text processing model. In addition to the input text and the prompt information, the text sequence can also contain information indicating the input text and information indicating the prompt text.

[0174] For example, the text processing instructions can be as follows:

[0175] {"text":"Although Xiao Ming's home is nearby, he lives in the AAA University dormitory","schema":{"Where does Xiao Ming live":null}}

[0176] Among them, "text" is information indicating the input text, and "schema" is information indicating the prompt text, which may include a question, indicating that the content to be extracted is the answer corresponding to the question.

[0177] In this example, the first subtext sequence may be: "Although Xiao Ming's home is very close, he lives in the dormitory of AAA University", and the second subtext sequence may be: "Where does Xiao Ming live?"

[0178] The first subtext sequence is input into the first extraction network shown in Figure 3 to obtain the first feature representation of each token in the first subtext sequence. The second subtext sequence is input into the second extraction network shown in Figure 3 to obtain the second feature representation of each token in the second subtext sequence. Using the feature fusion network, the first feature representation of each token in the first subtext sequence and the second feature representation of each token in the second subtext sequence are spliced ​​to obtain a spliced ​​sequence. The spliced ​​sequence is subjected to self-attention processing to obtain the feature representation of each token in the spliced ​​sequence. From the feature representation of each token in the spliced ​​sequence, the third feature representation of each token in the first subtext sequence is obtained to form the feature representation of the first subtext sequence.

[0179] The pointer network uses the feature representation of the first subtext sequence to predict the output probability of each token in the first subtext sequence. Based on this output probability, it determines the answer to the question extracted from the first subtext sequence. For example, from the input text "Xiao Ming's home is close, but he lives in the AAA University dormitory" in the above example, the answer to the question "Where does Xiao Ming live?" is "AAA University dormitory."

[0180] In addition to the above application scenarios, it can also be applied to other information extraction scenarios, which are not listed here one by one.

[0181] FIG5 is a flow chart of a text classification method provided by an embodiment of the present disclosure. As shown in FIG5 , the method may include the following steps:

[0182] Step 502: Get input text.

[0183] In text classification scenarios, if a text needs to be classified into a specific category, the text is used as input text. The input text can be a sentence, a paragraph composed of sentences, a document, or a smaller text unit such as a phrase.

[0184] Step 504: Input a text sequence including input text, candidate category labels and prompt text into a text processing model, and obtain the category label to which the input text belongs, which is determined by the text processing model from the candidate category labels based on the prompt text, wherein the prompt text includes information indicating the type of classification task, and the text processing model includes a feature extraction network and a pointer network; utilize the feature extraction network to perform feature extraction on the input text sequence to obtain a feature representation of a first sub-text sequence including the input text and the candidate category labels; and utilize the pointer network to predict the category label to which the input text belongs, utilizing the feature representation of the first sub-text sequence.

[0185] In text classification scenarios, candidate category labels can be concatenated with the input text to form the first subtext sequence. The significance of the candidate category labels lies in the fact that the classification of the input text must be selected from the candidate category labels. The prompt text contains information indicating the type of classification task, that is, how the text needs to be classified. The following specific applications are used as examples to describe each:

[0186] Application scenario 1: text classification.

[0187] In this application scenario, the input text, candidate text domain labels and prompt text containing information indicating the text classification task can be spliced ​​together to obtain a text sequence input text processing model.

[0188] For example, the text processing instructions can be as follows:

[0189] {"text":"Entertainment, Education, Sports, Technology | When will this year's college entrance examination results be announced?","schema":{"text field classification":null}}

[0190] "text" indicates the input text, with candidate text domain labels "entertainment," "education," "sports," and "technology" concatenated before the input text. "schema" indicates the prompt text. "Text domain classification" indicates that the input text needs to be classified into a specific domain.

[0191] In this example, the first subtext sequence may be: "Entertainment, Education, Sports, Science and Technology | When will this year's college entrance examination results be announced?", and the second subtext sequence may be: "Text field classification."

[0192] The first subtext sequence is input into the first extraction network shown in Figure 3 to obtain the first feature representation of each token in the first subtext sequence. The second subtext sequence is input into the second extraction network shown in Figure 3 to obtain the second feature representation of each token in the second subtext sequence. Using the feature fusion network, the first feature representation of each token in the first subtext sequence and the second feature representation of each token in the second subtext sequence are concatenated to obtain a concatenated sequence. The concatenated sequence is subjected to self-attention processing to obtain the third feature representation of each token in the concatenated sequence. From the feature representation of each token in the concatenated sequence, the third feature representation of each token in the first subtext sequence is obtained to form the feature representation of the first subtext sequence.

[0193] The pointer network uses the feature representation of the first subtext sequence to predict the output probability of each token in the first subtext sequence, and determines the text domain label to be extracted from the first subtext sequence based on the output probability. For example, from the first subtext sequence "Entertainment, Education, Sports, Technology | When will the college entrance examination results be announced this year" in the above example, the text domain label corresponding to the input text "When will the college entrance examination results be announced this year" is "Education". It can be seen that the present disclosure transforms the text classification task into information extraction from the first subtext sequence containing candidate text domain labels.

[0194] Application scenario 2: sentiment classification.

[0195] In this application scenario, the input text, candidate sentiment category labels, and prompt text containing information indicating the sentiment classification task can be concatenated to obtain a text sequence input text processing model.

[0196] For example, the text processing instructions can be as follows:

[0197] {"text":"positive, neutral, negative | This dish is delicious","schema":{"sentiment classification":null}}

[0198] "text" indicates the input text, with candidate sentiment category labels "positive," "neutral," and "negative" concatenated before the input text. "schema" indicates the prompt text. "Sentiment classification" indicates that sentiment classification is required for the input text.

[0199] In this example, the first subtext sequence may be: "positive, neutral, negative | This dish is delicious", and the second subtext sequence may be: "emotion classification".

[0200] The first subtext sequence is input into the first extraction network shown in Figure 3 to obtain the first feature representation of each token in the first subtext sequence. The second subtext sequence is input into the second extraction network shown in Figure 3 to obtain the second feature representation of each token in the second subtext sequence. Using the feature fusion network, the first feature representation of each token in the first subtext sequence and the second feature representation of each token in the second subtext sequence are spliced ​​to obtain a spliced ​​sequence. The spliced ​​sequence is subjected to self-attention processing to obtain the feature representation of each token in the spliced ​​sequence. From the feature representation of each token in the spliced ​​sequence, the third feature representation of each token in the first subtext sequence is obtained to form the feature representation of the first subtext sequence.

[0201] The pointer network uses the feature representation of the first subtext sequence to predict the output probability of each token in the first subtext sequence. Based on this output probability, the sentiment category label extracted from the first subtext sequence is determined. For example, from the first subtext sequence "positive, neutral, negative | This dish is really delicious" in the above example, the sentiment category label corresponding to the input text "This dish is really delicious" is extracted as "positive." This shows that the present disclosure transforms the sentiment classification task into information extraction from the first subtext sequence containing candidate sentiment category labels.

[0202] Application scenario 3: text matching.

[0203] In this application scenario, the input text, candidate text similarity labels, and prompt text containing instructions for determining text similarity can be spliced ​​together to obtain a text sequence input text processing model. The input text is a text pair, that is, two texts, and the purpose is to determine whether the two texts are similar.

[0204] For example, the text processing instructions can be as follows:

[0205] {"text":"Similar, Dissimilar|This dish is really delicious & This is a really delicious dish","schema":{"SimilarityClassification":null}}

[0206] "text" indicates the input text, which contains the text pairs "This dish is really delicious" and "This is a really delicious dish." The candidate text similarity labels "similar" and "dissimilar" are concatenated before the input text. "schema" indicates the prompt text. "Similarity classification" indicates that similarity needs to be determined for the text pairs in the input text.

[0207] In this example, the first subtext sequence may be: “similar, dissimilar | This dish is really delicious & This is really a delicious dish”, and the second subtext sequence may be: “similarity classification”.

[0208] The first subtext sequence is input into the first extraction network shown in Figure 3 to obtain the first feature representation of each token in the first subtext sequence. The second subtext sequence is input into the second extraction network shown in Figure 3 to obtain the second feature representation of each token in the second subtext sequence. Using the feature fusion network, the first feature representation of each token in the first subtext sequence and the second feature representation of each token in the second subtext sequence are spliced ​​to obtain a spliced ​​sequence. The spliced ​​sequence is subjected to self-attention processing to obtain the feature representation of each token in the spliced ​​sequence. From the feature representation of each token in the spliced ​​sequence, the third feature representation of each token in the first subtext sequence is obtained to form the feature representation of the first subtext sequence.

[0209] The pointer network uses the feature representation of the first subtext sequence to predict the output probability of each token in the first subtext sequence, and determines the text similarity label extracted from the first subtext sequence based on the output probability. For example, from the first subtext sequence "Similar, Dissimilar | This dish is really delicious & This is really delicious dish" in the above example, the similarity category label corresponding to the text pair "This dish is really delicious" and "This is really delicious dish" is "similar". It can be seen that the present disclosure transforms the text matching task into information extraction from the first subtext sequence containing candidate text similarity labels.

[0210] Application scenario 4: Reasoning text relationships.

[0211] In this application scenario, the input text, candidate text relationship labels, and prompt text containing the task of classifying the text relationship can be spliced ​​together to obtain a text sequence input text processing model. The input text is a text pair, that is, two texts, and the purpose is to determine the relationship between the two texts.

[0212] For example, the text processing instructions can be as follows:

[0213] {"text":"Implied, contradictory, neutral | Xiaoming ate a fruit & Xiaoming ate an apple","schema":{"text relation reasoning":null}}

[0214] "text" indicates the input text, which contains the text pairs "Xiao Ming ate a fruit" and "Xiao Ming ate an apple." The candidate text relationship labels "implied," "neutral," and "contradictory" are concatenated before the input text. "schema" indicates the prompt text. "Text relationship reasoning" indicates that text relationships need to be determined for the text pairs in the input text.

[0215] In this example, the first subtext sequence may be: “Implying, contradicting, neutral | Xiaoming ate a fruit & Xiaoming ate an apple”, and the second subtext sequence may be: “Textual relationship reasoning”.

[0216] The first subtext sequence is input into the first extraction network shown in Figure 3 to obtain the first feature representation of each token in the first subtext sequence. The second subtext sequence is input into the second extraction network shown in Figure 3 to obtain the second feature representation of each token in the second subtext sequence. Using the feature fusion network, the first feature representation of each token in the first subtext sequence and the second feature representation of each token in the second subtext sequence are spliced ​​to obtain a spliced ​​sequence. The spliced ​​sequence is subjected to self-attention processing to obtain the feature representation of each token in the spliced ​​sequence. From the feature representation of each token in the spliced ​​sequence, the third feature representation of each token in the first subtext sequence is obtained to form the feature representation of the first subtext sequence.

[0217] The pointer network uses the feature representation of the first subtext sequence to predict the output probability of each token in the first subtext sequence, and determines the text relationship label to be extracted from the first subtext sequence based on the output probability. For example, from the first subtext sequence "Implication, Contradiction, Neutrality | Xiao Ming ate a fruit & Xiao Ming ate an apple" in the above example, the text relationship label corresponding to the text pair "Xiao Ming ate a fruit" and "Xiao Ming ate an apple" is "Implication". It can be seen that the present disclosure transforms the task of inferring text relationships into information extraction from the first subtext sequence containing candidate text relationship labels.

[0218] Application scenario 5: selective question answering task.

[0219] In this application scenario, the input text, answer options, and prompt text containing instructions for the question-answering task can be spliced ​​together to form a text sequence input text processing model. The prompt text is the question, and the goal of the task is to select the answer to the question from the answer options based on the content expressed in the input text.

[0220] For example, the text processing instructions can be as follows:

[0221] {"text":"1980, 1981, 1982 | Xiaoming's mother and father got married in 1981, and Xiaoming was born the following year","schema":{"Year of Xiaoming's birth":null}}

[0222] "text" indicates the input text, which is "Xiaoming's mother and father got married in 1981, and Xiaoming was born the following year." The question is "In which year was Xiaoming born?" The answer options "1980," "1981," and "1982" are prepended to the input text. "schema" indicates the prompt text. "In which year was Xiaoming born?" indicates that the answer should be selected from the answer options.

[0223] In this example, the first subtext sequence may be: "1980, 1981, 1982 | Xiao Ming's mother and father got married in 1981, and Xiao Ming was born the following year", and the second subtext sequence may be: "In which year was Xiao Ming born".

[0224] The first subtext sequence is input into the first extraction network shown in Figure 3 to obtain the first feature representation of each token in the first subtext sequence. The second subtext sequence is input into the second extraction network shown in Figure 3 to obtain the second feature representation of each token in the second subtext sequence. Using the feature fusion network, the first feature representation of each token in the first subtext sequence and the second feature representation of each token in the second subtext sequence are concatenated to obtain a concatenated sequence. The concatenated sequence is subjected to self-attention processing to obtain the feature representation of each token in the concatenated sequence. From the feature representation of each token in the concatenated sequence, the third feature representation of each token in the first subtext sequence is obtained to form the feature representation of the first subtext sequence.

[0225] The pointer network uses the feature representation of the first subtext sequence to predict the output probability of each token in the first subtext sequence and determines the answer to be extracted from the first subtext sequence based on the output probability. For example, from the first subtext sequence in the above example, "1980, 1981, 1982 | Xiao Ming's mother and father got married in 1981, and Xiao Ming was born the following year," the answer to the question "In which year was Xiao Ming born?" is "1982." This shows that the present disclosure transforms the multiple-choice question-answering task into information extraction from the first subtext sequence containing answer options.

[0226] The above is a description of the text processing method provided by the embodiment of the present disclosure. The following describes in detail the method for training a text processing model provided by the embodiment of the present disclosure in conjunction with the embodiment. FIG6 is a flow chart of the method for training a text processing model provided by the embodiment of the present disclosure. As shown in FIG6 , the method may include the following steps:

[0227] Step 602: Acquire training data including a plurality of training samples, where the training samples include text samples and target samples annotated for the text samples.

[0228] The training samples obtained in this step can be manually labeled, that is, manually identifying target samples corresponding to the text processing task in the text samples. The text processing task can be an information extraction task such as entity extraction, entity relationship extraction, event extraction, sentiment extraction, or extractive question answering, or a text classification task such as text domain classification, sentiment classification, text matching, inferring text relationships, or selective question answering.

[0229] Step 604: Using a text sequence including a text sample and a prompt text as input to a text processing model, using a target sample annotated for the text sample as a target output of the text processing model, and training the text processing model, wherein the prompt text includes information indicating the type of text processing task corresponding to the target sample, and the text processing model includes a feature extraction network and a pointer network; utilizing the feature extraction network to perform feature extraction on the input text sequence to obtain a feature representation of a first sub-text sequence including the text sample; and using the pointer network to predict the target content corresponding to the text sample using the feature representation of the first sub-text sequence.

[0230] In the disclosed embodiments, the prompt text primarily includes information indicating the type of text processing task, to help the text processing model understand the processing to be performed on the input text. Different prompt texts can be used depending on the type of text processing task. For example, in an entity extraction task, the prompt text includes information indicating the entity type to be extracted. In an entity relationship extraction task, the prompt text includes information indicating the entity relationship type to be extracted. In an event extraction task, the prompt text includes information indicating the event type to be extracted. In an attribute and sentiment extraction task, the prompt text includes information indicating the attribute type and sentiment type to be extracted. In an extractive question-answering task, the prompt text includes information indicating the answer to the question to be extracted. In a text domain classification task, the prompt text includes information indicating the text domain classification task. In a sentiment classification task, the prompt text includes information indicating the sentiment classification task. In a text matching task, the prompt text includes information indicating the text similarity determination task. In a text relationship inference task, the prompt text includes information indicating the text relationship classification task. In a selective question-answering task, the prompt text includes information indicating the question-answering task.

[0231] As one possible implementation method, the feature extraction network may include a first extraction network, a second extraction network, and a feature fusion network.

[0232] The first extraction network is used to perform feature extraction on a first subtext sequence including a text sample in an input text sequence to obtain a first feature representation of each token in the first subtext sequence.

[0233] The second extraction network is used to perform feature extraction on the second subtext sequence including the prompt text in the input text sequence to obtain a second feature representation of each token in the second subtext sequence.

[0234] Using the feature fusion network, the first feature representation and the second feature representation are spliced ​​to obtain a spliced ​​sequence. The spliced ​​sequence is processed with self-attention to obtain the feature representation of each token in the spliced ​​sequence. From the feature representation of each token in the spliced ​​sequence, the third feature representation of each token in the first sub-text sequence is obtained to constitute the feature representation of the first sub-text sequence.

[0235] In this implementation, the first feature representation of each token in the first subtext sequence containing the text sample can be reused. For example, if a text sample is used for multiple types of tasks, such as entity extraction, entity relationship extraction, etc. In this case, the first subtext sequence containing the text sample is the same, but the prompt text is different. Therefore, the first feature representation of each token in the first subtext sequence containing the text sample can be cached for use in multiple types of tasks, thereby saving computing resources.

[0236] The details and structure of the text processing model can be found in the relevant description of FIG3 in the above embodiment, and will not be described in detail here.

[0237] During the model training process, the target samples annotated for the text samples are used as the target output of the text processing model. That is, the training objectives include minimizing the difference between the target content output by the text processing model and the corresponding target samples.

[0238] As one possible implementation, the text processing model can use the aforementioned training objective to construct a loss function, using the parameters of the pre-trained language model as the initial parameters of the feature extraction network. Based on this, the loss function is used in each iteration to update the model parameters of the text processing model using methods such as gradient descent until a preset training termination condition is met. The training termination condition may include, for example, the loss function value being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold.

[0239] As one of the feasible ways, the loss function can adopt circle loss (circle boundary loss function), for example, the loss function L is:

[0240] Among them, L i Represents the loss function corresponding to the i-th first subtext sequence. and They are the labels of the jth and kth tokens in the i-th first subtext sequence output by the text processing model, respectively. A label of 0 indicates that the token does not belong to the target content, and a label of 1 indicates that the token belongs to the target content. and They are the labels of the j-th and k-th Token annotated for the text samples in the i-th first subtext sequence, respectively. A label of 0 indicates that the Token does not belong to the target content, and a label of 1 indicates that the Token belongs to the target content.

[0241] In addition to circle loss, other types of loss functions can also be used, such as cross entropy loss function, etc., which are not listed here one by one.

[0242] As one possible implementation, the text processing method and text processing model training method provided in the embodiments of the present disclosure can be executed by a cloud server. A cloud server, also known as a cloud computing server or cloud host, is a host product within a cloud computing service ecosystem. It addresses the management difficulties and limited scalability of traditional physical hosts and VPS services.

[0243] After receiving input text sent by a user through a terminal device, the cloud server performs text processing using the method described above. Specifically, the cloud server inputs a text sequence containing the input text and a preset prompt text into a text processing model. The prompt text includes information indicating the type of text processing task. The cloud server then obtains the target content corresponding to the input text, which is output by the text processing model based on the information indicating the type of text processing task. The target content, or the service content corresponding to the target content, is then sent to the user device.

[0244] The service content corresponding to the target content can be flexibly configured based on the specific application scenario. For example, the user's input text in the intelligent customer service system can be analyzed for sentiment. Based on the sentiment category obtained from the sentiment analysis, the user can be provided with corresponding service content. For example, if the user's sentiment is identified as negative, the call can be transferred to a human customer service representative for processing. If the user's sentiment is identified as positive, a promotional message can be sent to the user, and so on.

[0245] The above-mentioned text processing model includes a feature extraction network and a pointer network; the feature extraction network extracts features from the input text sequence to obtain a feature representation of the first subtext sequence containing the input text; the pointer network uses the feature representation of the first subtext sequence to predict the target content corresponding to the input text.

[0246] The aforementioned terminal devices may include, but are not limited to, smart mobile terminals, smart home devices, wearable devices, and personal computers (PCs). Smart mobile devices may include mobile phones, tablets, laptops, personal digital assistants (PDAs), and internet-connected cars. Smart home devices may include smart TVs, smart speakers, and smart refrigerators. Wearable devices may include smart watches, smart glasses, virtual reality devices, augmented reality devices, and mixed reality devices (i.e., devices that support both virtual reality and augmented reality).

[0247] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0248] According to another embodiment, a text processing device is provided. FIG7 shows a schematic block diagram of a text processing device according to one embodiment. As shown in FIG7 , the device 700 may include: a text acquisition unit 701 and a text processing unit 702. The main functions of each component unit are as follows:

[0249] The text acquisition unit 701 is configured to acquire input text.

[0250] The text processing unit 702 is configured to input a text sequence including an input text and a prompt text into a text processing model, and obtain the target content corresponding to the input text output by the text processing model based on information indicating the type of text processing task, wherein the prompt text includes information indicating the type of text processing task, and the text processing model includes a feature extraction network and a pointer network; using the feature extraction network, feature extraction is performed on the input text sequence to obtain a feature representation of a first sub-text sequence including the input text; through the pointer network, the feature representation of the first sub-text sequence is used to predict the target content corresponding to the input text.

[0251] As one of the possible implementation methods, the feature extraction network includes: a first extraction network, a second extraction network and a feature fusion network.

[0252] Using the first extraction network, feature extraction is performed on a first subtext sequence of the input text in the input text sequence to obtain a first feature representation of each element Token in the first subtext sequence.

[0253] The second extraction network is used to perform feature extraction on the second subtext sequence including the prompt text in the input text sequence to obtain a second feature representation of each token in the second subtext sequence.

[0254] Using the feature fusion network, the first feature representation and the second feature representation are spliced ​​to obtain a spliced ​​sequence. The spliced ​​sequence is processed with self-attention to obtain the feature representation of each token in the spliced ​​sequence. From the feature representation of each token in the spliced ​​sequence, the third feature representation of each token in the first sub-text sequence is obtained to constitute the feature representation of the first sub-text sequence.

[0255] As one possible implementation method, the text acquisition unit 701 may acquire the input text and information indicating the type of the text processing task from the text processing instruction.

[0256] In response to the information indicating the text processing task type indicating a round of text processing tasks, the text processing unit 702 constructs prompt text using the information indicating the text processing task type and executes the step of inputting the sequence including the input text and the prompt text into the text processing model.

[0257] In response to the information indicating the type of text processing task indicating multiple rounds of text processing tasks, the text processing unit 702 constructs the prompt text for the first round in the first round of text processing tasks using the information indicating the type of the first round of text processing tasks, and inputs the text sequence including the input text and the prompt text for the first round into the text processing model.

[0258] In the text processing of other rounds of the multi-round text processing task except the first round of the text processing task in the multi-round text processing task, the text processing unit 702 uses the prompt text of the previous round of the multi-round text processing task, the target content output by the text processing model in the previous round, and the information indicating the type of the text processing task of the current round of the multi-round text processing task to construct the prompt text of the current round of the multi-round text processing task, and inputs the text sequence including the input text and the prompt text of the current round into the text processing model.

[0259] As one of the possible implementation methods, it can be applied to information extraction tasks, where the prompt text includes information indicating the type of target to be extracted.

[0260] If applied to entity extraction, the target type is the entity type and the target content is the entity word; or,

[0261] If applied to entity relationship extraction, the target type is the entity relationship type, and the target content is the entity pair corresponding to the entity relationship type in the input text; or,

[0262] If applied to event extraction, the target type is the event type, and the target content is the event trigger word and / or parameter word corresponding to the event type in the input text; or

[0263] If applied to attribute sentiment extraction, the target type is the attribute type and the sentiment type, and the target content is the attribute word corresponding to the attribute type and the sentiment word corresponding to the sentiment type in the input text; or

[0264] If applied to extractive question answering tasks, the target type is the answer corresponding to the question, and the target content is the answer fragment corresponding to the question in the input text.

[0265] As another feasible approach, it can be applied to a text classification task, where the text sequence also includes candidate category labels, and the prompt text includes information indicating the type of classification task.

[0266] If applied to text field classification, the classification task type is text field classification, and the candidate category label is the text field label; or,

[0267] If applied to sentiment classification, the classification task type is sentiment classification, and the candidate category label is the sentiment type label; or,

[0268] If applied to text matching, the classification task type is text similarity classification, the input text is a text pair, and the candidate category label is a text similarity label; or,

[0269] If applied to reasoning about text relations, the classification task type is text relation classification, the input text is a text pair, and the candidate category label is a text relation label; or,

[0270] If applied to a selective question answering task, the classification task type is a selective question answering task, and the candidate category labels are answer option labels.

[0271] According to another embodiment, a device for training a text processing model is provided. FIG8 shows a schematic block diagram of a text processing model training method according to one embodiment, corresponding to the model training device in the system shown in FIG1 . As shown in FIG8 , the device 800 may include: a sample acquisition unit 801 and a model training unit 802. The main functions of each component unit are as follows:

[0272] The sample acquisition unit 801 is configured to acquire training data including a plurality of training samples, wherein the training samples include text samples and target samples annotated for the text samples.

[0273] Model training unit 802 is configured to train a text processing model using a text sequence comprising a text sample and a prompt text as input, and a target sample labeled for the text sample as the target output of the text processing model. The prompt text includes information indicating the type of text processing task corresponding to the target sample, and the text processing model includes a feature extraction network and a pointer network. The feature extraction network extracts features from the input text sequence to obtain a feature representation of a first subtext sequence comprising the text sample. The pointer network then uses the feature representation of the first subtext sequence to predict the target content corresponding to the text sample.

[0274] As one possible implementation method, the feature extraction network may include a first extraction network, a second extraction network, and a feature fusion network.

[0275] The first extraction network is used to perform feature extraction on a first subtext sequence including a text sample in the input text sequence to obtain a first feature representation of each token in the first subtext sequence.

[0276] The second extraction network is used to perform feature extraction on the second subtext sequence including the prompt text in the input text sequence to obtain a second feature representation of each token in the second subtext sequence.

[0277] The feature fusion network is used to splice the first feature representation and the second feature representation to obtain a spliced ​​sequence, and the spliced ​​sequence is processed with self-attention to obtain the feature representation of each token in the spliced ​​sequence. From the feature representation of each token in the spliced ​​sequence, the third feature representation of each token in the first sub-text sequence is obtained to constitute the feature representation of the first sub-text sequence.

[0278] As one possible implementation, the text processing model can use the above training objectives to construct a loss function, use the parameters of the pre-trained language model as the initial parameters of the feature extraction network, and on this basis, use the value of the loss function in each round of iteration to update the model parameters of the text processing model using methods such as gradient descent until the preset training end conditions are met. The training end conditions may include, for example, the value of the loss function is less than or equal to a preset loss function threshold, the number of iterations reaches a preset number threshold, etc. As one possible implementation, the loss function can use circle loss. For specific formulas, see Formula (3) and Formula (4) in the method embodiment.

[0279] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For the relevant parts, refer to the partial description of the method embodiment. The device embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0280] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0281] In addition, an embodiment of the present disclosure further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0282] And an electronic device comprising:

[0283] one or more processors; and

[0284] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

[0285] The present disclosure also provides a computer program product, comprising a computer program, which implements the steps of any one of the aforementioned method embodiments when executed by a processor.

[0286] 9 exemplarily illustrates the architecture of an electronic device, which may include a processor 910, a video display adapter 911, a disk drive 912, an input / output interface 913, a network interface 914, and a memory 920. The processor 910, the video display adapter 911, the disk drive 912, the input / output interface 913, the network interface 914, and the memory 920 may be communicatively connected via a communication bus 930.

[0287] The processor 910 may be implemented as a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and may be used to execute relevant programs to implement the technical solutions provided by the present disclosure.

[0288] The memory 920 can be implemented in the form of a read-only memory (ROM), a random access memory RAM (RAM), a static storage device, a dynamic storage device, etc. The memory 920 can store an operating system 921 for controlling the operation of the electronic device 900, and a basic input and output system (BIOS) 922 for controlling the low-level operations of the electronic device 900. In addition, a web browser 923, a data storage management system 924, and a text processing device / model training device 925, etc. can also be stored. The above-mentioned text processing device / model training device 925 can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present disclosure. In short, when the technical solution provided by the present disclosure is implemented by software or firmware, the relevant program code is stored in the memory 920 and is called and executed by the processor 910.

[0289] The input / output interface 913 is used to connect input / output modules to implement information input and output. The input / output modules can be configured as components in the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0290] The network interface 914 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0291] The bus 930 comprises a pathway for transmitting information between the various components of the device (eg, the processor 910 , the video display adapter 911 , the disk drive 912 , the input / output interface 913 , the network interface 914 , and the memory 920 ).

[0292] It should be noted that although the above device only shows the processor 910, video display adapter 911, disk drive 912, input / output interface 913, network interface 914, memory 920, bus 930, etc., in a specific implementation, the device may also include other components necessary for normal operation. In addition, those skilled in the art will understand that the above device may also include only the components necessary to implement the disclosed solution, and does not necessarily include all the components shown in the figure.

[0293] The present disclosure also provides a processor. Figure 10 shows a schematic diagram of the structure of a processor for implementing the present disclosure. As shown in Figure 10, the processor 1000 is configured to run a program, wherein the program, when executed by the processor, executes the method in the above embodiment.

[0294] In the embodiment of the present disclosure, the processor 1000 may execute the operating program of the method in the embodiment.

[0295] Optionally, the processor 1000 can be configured to perform the following steps: obtaining input text; inputting a text sequence including the input text and prompt text into a text processing model, and obtaining the target content corresponding to the input text output by the text processing model based on the information indicating the type of text processing task, wherein the prompt text includes information indicating the type of text processing task, and the text processing model includes a feature extraction network and a pointer network; using the feature extraction network, performing feature extraction on the input text sequence to obtain a feature representation of a first sub-text sequence including the input text; and using the feature representation of the first sub-text sequence through the pointer network, predicting the target content corresponding to the input text.

[0296] Optionally, the processor 1000 can be configured to perform the following steps: obtaining input text; inputting a text sequence including the input text and the prompt text into a text processing model, and obtaining target content corresponding to the target type extracted by the text processing model from the input text, wherein the prompt text includes information indicating the target type to be extracted, and the text processing model includes a feature extraction network and a pointer network; using the feature extraction network, performing feature extraction on the input text sequence to obtain a feature representation of a first sub-text sequence including the input text; and using the feature representation of the first sub-text sequence through the pointer network, predicting the target content corresponding to the target type in the input text.

[0297] Optionally, the processor 1000 can be configured to perform the following steps: obtaining input text; inputting a text sequence including the input text, candidate category labels and prompt text into a text processing model, and obtaining the category label to which the input text belongs, determined by the text processing model from the candidate category labels based on the prompt text, wherein the prompt text includes information indicating the type of classification task, and the text processing model includes a feature extraction network and a pointer network; using the feature extraction network, performing feature extraction on the input text sequence to obtain a feature representation of a first sub-text sequence including the input text and the candidate category labels; and using the feature representation of the first sub-text sequence through the pointer network, predicting the category label to which the input text belongs.

[0298] Optionally, the processor 1000 can be configured to perform the following steps: obtaining input text from a user device; inputting a text sequence including the input text and a prompt text into a text processing model, and obtaining target content corresponding to the input text output by the text processing model based on the prompt text, wherein the prompt text includes information indicating the type of text processing task, and the text processing model includes a feature extraction network and a pointer network; sending the target content or the service content corresponding to the target content to the user device; using the feature extraction network to extract features from the input text sequence to obtain a feature representation of a first sub-text sequence including the input text; and using the feature representation of the first sub-text sequence through the pointer network to predict the target content corresponding to the input text.

[0299] Optionally, the processor 1000 can be configured to perform the following steps: obtaining training data including multiple training samples, wherein the training samples include text samples and target samples annotated for the text samples; using a text sequence including the text sample and the prompt text as the input of a text processing model, and using the target sample annotated for the text sample as the target output of the text processing model, to train the text processing model, wherein the prompt text includes information indicating the type of text processing task corresponding to the target sample, and the text processing model includes a feature extraction network and a pointer network; using the feature extraction network to extract features from the input text sequence to obtain a feature representation of a first sub-text sequence including the text sample; and using the feature representation of the first sub-text sequence through the pointer network to predict the target content corresponding to the text sample.

[0300] Computer-readable storage media may also be referred to as computer storage media. They may include data signals transmitted in baseband or as part of a carrier wave, carrying readable program code. Such transmitted data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable storage media may transmit, propagate, or transfer programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0301] The program code contained in the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, radio frequency, etc., or any suitable combination of the foregoing.

[0302] The embodiments of the present disclosure further provide a computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method provided in the embodiments of the present disclosure is implemented.

[0303] The embodiments of the present disclosure further provide a computer program, which, when executed by a processor, implements the method provided in the embodiments of the present disclosure.

[0304] Optionally, the above-mentioned computer program implements program code of the following steps when executed by a processor: obtaining input text; inputting a text sequence including the input text and prompt text into a text processing model, and obtaining target content corresponding to the input text output by the text processing model based on the information indicating the type of text processing task, wherein the prompt text includes information indicating the type of text processing task, and the text processing model includes a feature extraction network and a pointer network; using the feature extraction network, extracting features from the input text sequence to obtain a feature representation of a first sub-text sequence including the input text; and using the feature representation of the first sub-text sequence through the pointer network, predicting the target content corresponding to the input text.

[0305] Optionally, the above-mentioned computer program implements program code of the following steps when executed by a processor: obtaining input text; inputting a text sequence including the input text and the prompt text into a text processing model, and obtaining target content corresponding to the target type extracted from the input text by the text processing model, wherein the prompt text includes information indicating the target type to be extracted, and the text processing model includes a feature extraction network and a pointer network; using the feature extraction network, extracting features from the input text sequence to obtain a feature representation of a first sub-text sequence including the input text; and using the feature representation of the first sub-text sequence through the pointer network, predicting the target content corresponding to the target type in the input text.

[0306] Optionally, the above-mentioned computer program implements the program code of the following steps when executed by the processor: obtaining input text; inputting a text sequence including the input text, candidate category labels and prompt text into a text processing model, and obtaining the category label to which the input text belongs, which is determined by the text processing model from the candidate category labels based on the prompt text, wherein the prompt text includes information indicating the type of classification task, and the text processing model includes a feature extraction network and a pointer network; using the feature extraction network, extracting features from the input text sequence to obtain a feature representation of a first sub-text sequence including the input text and the candidate category labels; and using the feature representation of the first sub-text sequence through the pointer network, predicting the category label to which the input text belongs.

[0307] Optionally, the above-mentioned computer program implements program code of the following steps when executed by a processor: obtaining input text from a user device; inputting a text sequence including the input text and a prompt text into a text processing model, and obtaining target content corresponding to the input text output by the text processing model based on the prompt text, wherein the prompt text includes information indicating the type of text processing task, and the text processing model includes a feature extraction network and a pointer network; sending the target content or the service content corresponding to the target content to the user device; using the feature extraction network to extract features from the input text sequence to obtain a feature representation of a first sub-text sequence including the input text; and using the feature representation of the first sub-text sequence through the pointer network to predict the target content corresponding to the input text.

[0308] Optionally, the computer program implements the following program code when executed by a processor: obtaining training data including a plurality of training samples, wherein the training samples include text samples and target samples annotated for the text samples; using a text sequence including text samples and prompt texts as input of a text processing model, and using the target samples annotated for the text samples as target output of the text processing model, to train the text processing model, wherein the prompt text includes information indicating the type of text processing task corresponding to the target sample, and the text processing model includes a feature extraction network and a pointer network; using the feature extraction network to extract features from the input text sequence to obtain a feature representation of a first sub-text sequence including the text sample; and using the feature representation of the first sub-text sequence through the pointer network to predict the target content corresponding to the text sample.

[0309] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that the present disclosure can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, can be embodied in the form of a computer program product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present disclosure.

[0310] The technical solutions provided by the present disclosure are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present disclosure. The descriptions of the above embodiments are intended only to help understand the methods and core concepts of the present disclosure. Furthermore, those skilled in the art will appreciate that variations in the specific implementation methods and scope of application may occur based on the concepts of the present disclosure. In summary, the contents of this specification should not be construed as limiting the present disclosure. Industrial Applicability

[0311] The solution provided by the embodiment of the present disclosure can be applied to the text processing process to obtain input text; input a text sequence including the input text and a prompt text into a text processing model, and obtain the target content corresponding to the input text output by the text processing model based on the information indicating the type of text processing task, wherein the prompt text includes information indicating the type of text processing task, and the text processing model includes a feature extraction network and a pointer network; utilize the feature extraction network to perform feature extraction on the input text sequence to obtain a feature representation of a first sub-text sequence including the input text; through the pointer network, utilize the feature representation of the first sub-text sequence to predict the target content corresponding to the input text, thereby solving the technical problem that the general text processing model has poor information extraction effect on information extraction tasks.

Claims

1. A text processing method, the method comprising: Get input text; Inputting a text sequence including the input text and the prompt text into a text processing model, and obtaining a target content corresponding to the input text output by the text processing model according to the prompt text, wherein the prompt text includes information indicating a type of text processing task, and the text processing model includes a feature extraction network and a pointer network; Using the feature extraction network, extracting features from the input text sequence to obtain a feature representation of a first subtext sequence including the input text; The target content corresponding to the input text is predicted by using the feature representation of the first subtext sequence through the pointer network.

2. The method according to claim 1, wherein: The feature extraction network includes: a first extraction network, a second extraction network and a feature fusion network; Using the first extraction network, feature extraction is performed on a first subtext sequence of the input text sequence, to obtain a first feature representation of each element Token in the first subtext sequence; Using the second extraction network, feature extraction is performed on a second subtext sequence including the prompt text in the input text sequence to obtain a second feature representation of each Token in the second subtext sequence; The first feature representation and the second feature representation are concatenated using the feature fusion network to obtain a concatenated sequence, the concatenated sequence is subjected to self-attention processing to obtain a feature representation of each Token in the concatenated sequence, and a third feature representation of each Token in the first sub-text sequence is obtained from the feature representation of each Token in the concatenated sequence to constitute the feature representation of the first sub-text sequence.

3. The method according to claim 1 or 2, wherein: The acquiring of the input text comprises: acquiring the input text and the information indicating the type of text processing task from a text processing instruction; In response to the information indicating the type of text processing task indicating a round of text processing task, the prompt text is constructed using the information indicating the type of text processing task, and the step of inputting the text sequence including the input text and the prompt text into the text processing model is performed.

4. The method according to claim 3, wherein: In response to the information indicating the type of text processing task indicating multiple rounds of text processing tasks, in a first round of text processing tasks, using the information indicating the type of the first round of text processing tasks to construct a first round of prompt text, and inputting a text sequence including the input text and the first round of prompt text into the text processing model; In the other rounds of text processing tasks in the multi-round text processing tasks except the first round of text processing tasks, the prompt text of the previous round of the multi-round text processing tasks, the target content output by the text processing model in the previous round, and the information indicating the type of the current round of text processing tasks of the multi-round text processing tasks are used to construct the prompt text of the current round of the multi-round text processing tasks, including the input text and the current The text sequence of the prompt text of the round is input into the text processing model.

5. An information extraction method, the method comprising: Get input text; Inputting a text sequence including the input text and the prompt text into a text processing model, and obtaining target content corresponding to a target type extracted from the input text by the text processing model, wherein the prompt text includes information indicating the target type to be extracted, and the text processing model includes a feature extraction network and a pointer network; Using the feature extraction network, extracting features from the input text sequence to obtain a feature representation of a first subtext sequence including the input text; The target content corresponding to the target type in the input text is predicted by the pointer network using the feature representation of the first subtext sequence.

6. The method according to claim 5, wherein: The target type is an entity type, and the target content is an entity word; or The target type is an entity relationship type, and the target content is an entity pair corresponding to the entity relationship type in the input text; or The target type is an event type, and the target content is an event trigger word and / or parameter word corresponding to the event type in the input text; or, The target type is an attribute type and a sentiment type, and the target content is an attribute word corresponding to the attribute type and a sentiment word corresponding to the sentiment type in the input text; or The target type is the answer corresponding to the question, and the target content is the answer segment corresponding to the question in the input text.

7. A text classification method, the method comprising: Get input text; Inputting a text sequence including the input text, candidate category labels and prompt text into a text processing model, and obtaining a category label to which the input text belongs determined by the text processing model from the candidate category labels according to the prompt text, wherein the prompt text includes information indicating a classification task type, and the text processing model includes a feature extraction network and a pointer network; Using the feature extraction network, extracting features from the input text sequence to obtain a feature representation of a first subtext sequence including the input text and the candidate category labels; The category label to which the input text belongs is predicted by using the feature representation of the first sub-text sequence through the pointer network.

8. The method according to claim 7, wherein: The feature extraction network includes: a first extraction network, a second extraction network and a feature fusion network; Using the first extraction network, feature extraction is performed on a first subtext sequence of the input text sequence, to obtain a first feature representation of each element Token in the first subtext sequence; Using the second extraction network, feature extraction is performed on a second subtext sequence including the prompt text in the input text sequence to obtain a second feature representation of each Token in the second subtext sequence; The first feature representation and the second feature representation are concatenated using the feature fusion network to obtain a concatenated sequence, the concatenated sequence is subjected to self-attention processing to obtain a feature representation of each Token in the concatenated sequence, and a third feature representation of each Token in the first sub-text sequence is obtained from the feature representation of each Token in the concatenated sequence to constitute the feature representation of the first sub-text sequence.

9. The method according to claim 7 or 8, wherein: The acquiring of the input text comprises: acquiring the input text and the information indicating the type of text processing task from a text processing instruction; If the information indicating the type of text processing task indicates a round of text processing task, the prompt text is constructed using the information indicating the type of text processing task, and the step of inputting the text sequence including the input text and the prompt text into the text processing model is performed.

10. The method according to claim 7, wherein: The classification task type is text field classification, and the candidate category label is a text field label; or, The classification task type is sentiment classification, and the candidate category label is a sentiment category label; or, The classification task type is text similarity classification, the input text is a text pair, and the candidate category label is a text similarity label; or, The classification task type is text relationship classification, the input text is the text pair, and the candidate category label is a text relationship label; or, The classification task type is a multiple-choice question-answering task, and the candidate category labels are answer option labels.

11. A text processing method, executed by a cloud server, wherein: The method comprises: Get input text from the user's device; Inputting a text sequence including the input text and the prompt text into a text processing model, and obtaining a target content corresponding to the input text output by the text processing model according to the prompt text, wherein the prompt text includes information indicating a type of text processing task, and the text processing model includes a feature extraction network and a pointer network; Sending the target content or service content corresponding to the target content to the user equipment; Using the feature extraction network, extracting features from the input text sequence to obtain a feature representation of a first subtext sequence including the input text; The target content corresponding to the input text is predicted by using the feature representation of the first subtext sequence through the pointer network.

12. The method according to claim 11, wherein: The feature extraction network includes: a first extraction network, a second extraction network and a feature fusion network; Using the first extraction network, feature extraction is performed on a first subtext sequence of the input text sequence, to obtain a first feature representation of each element Token in the first subtext sequence; Using the second extraction network, the second subtext of the input text sequence including the prompt text is extracted. Perform feature extraction on this sequence to obtain a second feature representation of each Token in the second subtext sequence; The first feature representation and the second feature representation are concatenated using the feature fusion network to obtain a concatenated sequence, the concatenated sequence is subjected to self-attention processing to obtain a feature representation of each Token in the concatenated sequence, and a third feature representation of each Token in the first sub-text sequence is obtained from the feature representation of each Token in the concatenated sequence to constitute the feature representation of the first sub-text sequence.

13. A method for training a text processing model, the method comprising: Acquire training data including a plurality of training samples, wherein the training samples include text samples and target samples annotated for the text samples; Taking a text sequence including the text sample and the prompt text as an input of a text processing model, taking a target sample annotated for the text sample as a target output of the text processing model, and training the text processing model, wherein the prompt text includes information indicating a type of text processing task corresponding to the target sample, and the text processing model includes a feature extraction network and a pointer network; Using the feature extraction network, extracting features from an input text sequence to obtain a feature representation of a first subtext sequence including the text sample; The target content corresponding to the text sample is predicted by using the feature representation of the first sub-text sequence through the pointer network.

14. A text processing device, comprising: A text acquisition unit, configured to acquire input text; A text processing unit is configured to input a text sequence including the input text and the prompt text into a text processing model, and obtain a target content corresponding to the input text output by the text processing model based on the prompt text, wherein the prompt text includes information indicating the type of text processing task, and the text processing model includes a feature extraction network and a pointer network; using the feature extraction network, feature extraction is performed on the input text sequence to obtain a feature representation of a first sub-text sequence including the input text; through the pointer network, the feature representation of the first sub-text sequence is used to predict the target content corresponding to the input text.

15. A device for training a text processing model, the device comprising: A sample acquisition unit is configured to acquire training data including a plurality of training samples, wherein the training samples include text samples and target samples annotated for the text samples; The model training unit is configured to use a text sequence including the text sample and the prompt text as an input of a text processing model, and use a target sample annotated for the text sample as a target output of the text processing model to train the text processing model, wherein the prompt text includes information indicating the type of text processing task corresponding to the target sample, and the text processing model includes a feature extraction network and a pointer network; using the feature extraction network, feature extraction is performed on the input text sequence to obtain a feature representation of a first sub-text sequence including the text sample; and through the pointer network, the feature representation of the first sub-text sequence is used to predict the target content corresponding to the text sample.

16. A computer program product, wherein: The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 13.

17. A computer program product, wherein: The invention comprises a non-volatile computer-readable storage medium storing a computer program, wherein the computer program implements the method according to any one of claims 1 to 13 when executed by a processor.

18. A computer program, wherein When the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

19. A computer-readable storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

20. An electronic device, wherein: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of claims 1 to 13.

Images