Text filtering model training method and device and text filtering method and device

Abstract

The invention provides a text filtering model training method and a text filtering method and device.The training method comprises the steps that a training text sequence is input into a to-be-trained text filtering model, an obtained target text sequence is input into a target text processing model, and a target predicted value output by the target text processing model is obtained; obtaining target text loss based on the target predicted value and the training text sequence label; and training a text filtering model based on the target text loss to obtain a target text filtering model. Text filtering is carried out by training the text filtering model, more long text filtering scenes can be handled while automatic text filtering is achieved, the labor cost is reduced, the text filtering application range and efficiency are improved, the fitting performance of the target text processing model is guaranteed, the text filtering model is added in front of the target text processing model for text filtering, and the text filtering efficiency is improved. The text filtering task can be carried out without changing the structure of the target text processing model, and the fitting ability and the processing efficiency of the target text processing model are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a text filtering model training method, a text filtering method and a device. Background Art

[0002] Natural language processing technology has been widely used in various industries. For example, in risk control and news content processing in the financial field, it is usually necessary to use natural language processing models based on massive text to judge user overdue risks, news content tendencies, etc.

[0003] There is usually a lot of redundant content in massive texts, which is poorly related to the target tasks performed by the natural language processing model. Due to the need to process these redundant contents, the natural language processing model will consume additional computer storage and computing resources. At the same time, these redundant contents may also bring large noise to the model, thereby causing the model to overfit, which greatly affects the accuracy of the natural language processing model.

[0004] Therefore, how to effectively filter out redundant content in massive texts is a major challenge in currently using natural language processing models to process long text content. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a text filtering model training method, a text filtering method and a device to effectively filter redundant content in a text.

[0006] According to one aspect of the present invention, a text filtering model training method is provided, the method comprising:

[0007] For each training text sequence in the training set, the training text sequence is input into the text filtering model to be trained, so that the text filtering model filters the training text sequence to obtain a target text sequence;

[0008] Inputting the target text sequence into a target text processing model so that the target text processing model outputs a target prediction value for the target text sequence;

[0009] Obtaining a target text loss based on the target prediction value and the label of the training text sequence;

[0010] The text filtering model is trained based on the target text loss to obtain a target text filtering model.

[0011] In a possible embodiment, the text filtering model includes a first text filter and a second text filter, and the first text filter and the second text filter are connected in parallel;

[0012] The text filtering model filters the training text sequence to obtain a target text sequence, including:

[0013] The first text filter and the second text filter filter the training text sequence respectively to obtain a first target text sequence and a second target text sequence;

[0014] The step of inputting the target text sequence into a target text processing model so that the target text processing model outputs a target prediction value for the target text sequence includes:

[0015] Inputting the first target text sequence and the second target text sequence into the target text processing model respectively, so that the target text processing model outputs a first prediction value for the first target text sequence and outputs a second prediction value for the second target text sequence;

[0016] The step of obtaining a target text loss based on the target prediction value and the label of the training text sequence includes:

[0017] Obtaining a first text loss based on the first prediction value and the label of the training text sequence;

[0018] Obtaining a second text loss based on the second prediction value and the label of the training text sequence;

[0019] The training of the text filtering model based on the target text loss to obtain the target text filtering model includes:

[0020] Based on the first text loss and the second text loss, the first text filter and the second text filter are trained to obtain a target text filtering model.

[0021] In a possible embodiment, the first text filter and the second text filter respectively filter the training text sequence to obtain a first target text sequence and a second target text sequence, including:

[0022] The first text filter and the second text filter respectively score each text contained in the training text sequence;

[0023] The first text filter and the second text filter obtain the first target text sequence and the second text sequence based on the scores and retention lengths of the texts. The retention length is a parameter in the first text filter and the second text filter, and an initial value of the retention length is a preset value.

[0024] In a possible embodiment, the first target text sequence includes a common text and a first difference text, wherein the common text is the same text in the first target text sequence and the second target text sequence, the first difference text is the text that exists in the first target text sequence but does not exist in the second target text sequence, and the second target text sequence includes the common text and the second difference text, and the second difference text is the text that exists in the second target text sequence but does not exist in the first target text sequence;

[0025] The method further comprises:

[0026] Based on the first text loss, the second text loss, the score of the first difference text, and the score of the second difference text, a filter loss is obtained, wherein the filter loss is positively correlated with the difference between the first text loss and the second text loss, and is positively correlated with the difference between the score of the first difference text and the score of the second difference text;

[0027] The first text filter and the second text filter are trained based on the filter loss to obtain a target text filtering model.

[0028] In a possible embodiment, the training of the first text filter and the second text filter based on the filter loss to obtain a target text filtering model includes:

[0029] The filter loss is minimized by a gradient descent algorithm to obtain a target text filtering model.

[0030] In a possible embodiment, the target text processing model is a pre-trained model, and the method further includes:

[0031] The first text filter, the second text filter and the target text processing model are trained based on the first text loss, the second text loss and the filter loss to obtain a target text filtering model and a trained target text training model.

[0032] In a possible embodiment, the training set is predetermined by the following steps:

[0033] Based on the function of the target text processing model, determining a candidate text sequence whose label corresponds to the function;

[0034] Based on the proportion of each of the labels in the candidate text sequence, the candidate text sequence is sampled so that the difference in the number of each of the labels in the training set is less than a preset difference threshold.

[0035] In a possible embodiment, the method further includes:

[0036] Performing word segmentation on the training text sequence to obtain a word segmentation result;

[0037] Based on a preset vocabulary, tokenize the word segmentation result to obtain a token sequence corresponding to the training text sequence;

[0038] The step of inputting the training text sequence into a text filtering model to be trained so that the text filtering model filters the training text sequence to obtain a target text sequence includes:

[0039] The token sequence is input into a text filtering model to be trained so that the text filtering model embeds the token sequence to obtain a vector sequence corresponding to the training text sequence, and the training text sequence is filtered based on the vector sequence to obtain a target text sequence.

[0040] According to another aspect of the present invention, a text filtering method is provided, the method comprising:

[0041] Get the target text to be processed;

[0042] Inputting the target text into a pre-trained target text filtering model, so that the target text filtering model filters the target text and inputs it into a pre-trained target text processing model, wherein the target text filtering model is pre-trained by any of the above-mentioned text filtering model training methods;

[0043] Obtain the processing result output by the target text processing model.

[0044] In a possible embodiment, the target text filtering model includes a first text filter and a second text filter; and inputting the target text into the pre-trained target text filtering model includes:

[0045] The target text is input into the first text filter or the second text filter.

[0046] According to another aspect of the present invention, a text filtering model training device is provided, the device comprising:

[0047] A first input module is used for inputting each training text sequence in the training set into the text filtering model to be trained, so that the text filtering model filters the training text sequence to obtain a target text sequence;

[0048] A first output module, used for inputting the target text sequence into a target text processing model, so that the target text processing model outputs a target prediction value for the target text sequence;

[0049] A loss acquisition module, used to obtain a target text loss based on the target prediction value and the label of the training text sequence;

[0050] The training module is used to train the text filtering model based on the target text loss to obtain a target text filtering model.

[0051] According to another aspect of the present invention, a text filtering device is provided, the device comprising:

[0052] An acquisition module is used to acquire the target text to be processed;

[0053] A second input module is used to input the target text into a pre-trained target text filtering model, so that the target text filtering model filters the target text and inputs it into the pre-trained target text processing model, wherein the target text filtering model is pre-trained by any of the above-mentioned text filtering model training methods;

[0054] The second output module is used to obtain the processing result output by the target text processing model.

[0055] According to another aspect of the present invention, there is provided an electronic device, comprising:

[0056] Processor; and

[0057] Memory for storing programs,

[0058] Wherein, the program includes instructions, and when the instructions are executed by the processor, the processor executes any of the above-mentioned text filtering model training methods or text filtering methods.

[0059] According to another aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute any of the above-mentioned text filtering model training methods or text filtering methods.

[0060] One or more technical solutions provided in the embodiments of the present invention perform text filtering by training a text filtering model, and while realizing automatic text filtering, can cope with more long text filtering scenarios, reduce labor costs, improve the scope of application of text filtering and text filtering efficiency, and ensure the fit of the target text processing model. In addition, by adding a text filtering model before the target text processing model to filter the input text, there is no need to change the structure of the target text processing model to perform the text filtering task, thereby ensuring the fit ability and processing efficiency of the target text processing model. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Further details, features and advantages of the invention are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0062] Figure 1 A flowchart of a text filtering model training method provided by an embodiment of the present invention;

[0063] Figure 2 A further flow chart of a text filtering model training method provided by an embodiment of the present invention;

[0064] Figure 3 A flowchart of a text filtering method provided by an embodiment of the present invention;

[0065] Figure 4 A structural schematic diagram of a text filtering model training device provided by an embodiment of the present invention;

[0066] Figure 5 A schematic diagram of a flow chart of a text filtering device provided by an embodiment of the present invention;

[0067] Figure 6 A block diagram of an exemplary electronic device that can be used to implement an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0068] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0069] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0070] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0071] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0072] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

[0073] In order to filter massive amounts of text and thus reduce the difficulty of text processing by the natural language processing model, the text input to the natural language processing model is usually pre-processed in the following two ways in related technologies:

[0074] 1. Method based on rule system: First, through manual statistics and manual analysis, word packages or phrase libraries that meet the target task are obtained. The target task can be overdue risk prediction, news tendency prediction, etc. Then, the input text is processed based on the word package or phrase library through manually designed filtering rules, such as filtering the same semantic words after word segmentation of the input text, filtering based on similarity, etc., and the filtered content is input into the downstream classification or regression task for training, so as to achieve the purpose of reducing the input length of the model.

[0075] However, this method is highly dependent on manual experience, and requires people who have been doing the corresponding tasks for a long time to have sufficient experience to conduct statistical analysis and design rules. In addition, the relevant personnel need to conduct a large amount of text statistical analysis and rule design, which is very costly and inefficient. In addition, in actual scenarios, the larger the filtering volume, the greater the statistical, analytical and rule design work. It is difficult for a rule to be applicable in all scenarios, which can easily lead to limited filtering content, thus affecting the fitting effect of the natural language processing model.

[0076] 2. Model-based method: When the target task processing model performs the target task based on text, the input sequence is clustered and pooled to shorten the processing length of the next layer of the model as much as possible without significantly weakening the model fitting ability, so as to reduce the model processing content layer by layer and save storage and computing resources. For any common training task, in order to ensure the model fitting effect, the complexity of the target task model needs to be guaranteed. For example, if the Attention-based model architecture such as Transformer is used, sufficient layers and hidden layer dimensions are required.

[0077] This method improves the structure of the target task processing model so that it can process shorter texts layer by layer, which often results in a significant loss in fitting ability. In addition, in order to balance fitting ability and processing efficiency, the proportion of model processing length reduction is often limited. Therefore, it cannot meet the expected requirements in terms of processing efficiency and fitting ability.

[0078] In order to solve the above problems, the embodiments of the present invention provide a text filtering model training method, a text filtering method, an apparatus, an electronic device and a storage medium. The text filtering model training method provided by the embodiments of the present invention can be applied to any electronic device with a text filtering model training function, such as a computer, a server or a mobile terminal.

[0079] like Figure 1 As shown, Figure 1 A flowchart of a text filtering model training method provided by an embodiment of the present invention may include the following steps:

[0080] S101, for each training text sequence in a training set, inputting the training text sequence into a text filtering model to be trained, so that the text filtering model filters the training text sequence to obtain a target text sequence;

[0081] S102, inputting the target text sequence into a target text processing model, so that the target text processing model outputs a target prediction value for the target text sequence;

[0082] S103, obtaining a target text loss based on the target prediction value and the label of the training text sequence;

[0083] S104: Training the text filtering model based on the target text loss to obtain a target text filtering model.

[0084] By using the embodiment of the present invention, text filtering is performed by training a text filtering model, and while realizing automatic text filtering, more long text filtering scenarios can be handled, thereby reducing labor costs, improving the scope of application of text filtering and text filtering efficiency, and ensuring the fit of the target text processing model. In addition, by adding a text filtering model before the target text processing model to filter the input text, the text filtering task can be performed without changing the structure of the target text processing model, thereby ensuring the fit ability and processing efficiency of the target text processing model.

[0085] The above steps S101-S104 are exemplarily described below:

[0086] The above training set includes a large amount of text data, and the specific type of the training text sequence included in the training set can be determined based on the specific function of the target text processing model. Exemplarily, the function of the target text processing model can be risk prediction, news content tendency judgment, etc. Correspondingly, for the risk prediction model, the training text sequence in the training set can be user information in text form, including user credit records, credit records, user search terms, etc., and the label of the training text sequence can be user overdue or not overdue; for the news content tendency judgment model, the training text sequence in the training set can be a news document, and the label of the training text sequence can be positive, neutral or negative.

[0087] The training text sequence can be data from within the company or from an open source dataset. In a possible embodiment, in order to ensure that the labels of the training text sequences in the training set are evenly distributed, and thus ensure the fitting ability of the text filtering model to text sequences with different labels, the training set can be obtained by following the steps below:

[0088] Based on the function of the target text processing model, determining a candidate text sequence whose label corresponds to the function;

[0089] Based on the proportion of each of the labels in the candidate text sequence, the candidate text sequence is sampled so that the difference in the number of each of the labels in the training set is less than a preset difference threshold.

[0090] In a possible embodiment, a candidate text sequence can be obtained from source data based on the label of the text sequence. The source data can be all the data within the company or an open source data set. The correspondence between the above label and function means that the label belongs to the judgment result that the target text processing model may output. For example, for the overdue risk prediction model, the label that may be output is overdue or not overdue, so the data labeled as overdue or not overdue can be used as a candidate text sequence. Since overdue or not overdue is usually set for users, the candidate text sequence here can be a text sequence composed of user data generated by users in the financial platform.

[0091] In order to make the labels in the training set evenly distributed, the candidate text sequences can be sampled using a preset sampling rule. In a possible embodiment, the sampling rule can be to sample the candidate text sequences in the inverse ratio of the ratio of the number of labels in the candidate text sequences. Exemplarily, the labels of the candidate text sequences are label 0 or label 1, wherein there are 50 candidate text sequences corresponding to label 0 and 20 candidate text sequences corresponding to label 1, i.e., the ratio of the number of label 0 to label 1 is 5:2. Therefore, the candidate text sequences of label 0 and the candidate text sequences of label 1 can be sampled in a ratio of 2:5 to form a training set. For example, 10 candidate text sequences can be sampled in label 0 and 10 candidate text sequences can be sampled in label 1.

[0092] The sampling rule may also be to retain all the label data with a relatively small proportion in the candidate text sequence, and sample the candidate text sequences corresponding to the labels with a relatively large proportion, so that the difference between the number of candidate text sequences corresponding to the labels with a relatively small proportion in the training set and the number of candidate text sequences corresponding to the labels with a relatively large proportion is less than a preset difference threshold. The difference threshold may be preset according to actual needs, such as 10, 50, etc.

[0093] In other embodiments, the above sampling rule may also be other contents, which are not specifically limited in the embodiment of the present invention. It is only necessary to make the difference in the number of text sequences with different labels in the training set smaller than a preset difference threshold.

[0094] In a possible embodiment, before the training text sequence is input into the text filtering model to be trained, the training text sequence may be preprocessed, and the preprocessing may include deduplication, abnormal data removal, etc. Specifically, different preprocessing methods may be used for different target text processing models. Exemplarily, when the input of the target text processing model is a search text sequence of a user in a search scenario, the search text sequence may be deduplicated in the preprocessing stage, and when the input of the target text processing model is a news document, the document may be deduplicated in the preprocessing stage.

[0095] In a possible embodiment, the preprocessing may further include tokenizing the training text sequence. Token is the smallest unit in text processing, and tokenizing the training text sequence refers to splitting the training text sequence into tokens. In the embodiment of the present invention, the text data included in the training set may be tokenized by the following steps:

[0096] S11, performing word segmentation on the training text sequence to obtain a word segmentation result;

[0097] S12, tokenizing the word segmentation result based on a preset vocabulary to obtain a token sequence corresponding to the training text sequence;

[0098] In the embodiment of the present invention, the training text sequence can be segmented by any feasible segmentation tool, such as the jieba segmentation tool, to obtain the segmentation results. After that, each segmentation result can be converted according to the preset word list into a character form that can be understood by the machine, that is, the segmentation result is tokenized.

[0099] As a possible implementation, the preset vocabulary can be a vocabulary provided by the BERT (Bidirectional Encoder Representations from Transformers) open source model. As another possible implementation, the frequency of occurrence of each word can be counted based on the company's internal business data, and a vocabulary can be constructed based on words with a frequency higher than a preset frequency threshold. The vocabulary includes a correspondence between words and numbers, which can be set according to actual needs and can be either numbers or letters. The above vocabulary also includes a correspondence between each single word and number.

[0100] When it is necessary to tokenize the word segmentation results, you can query the preset vocabulary. Specifically, for the word segmentation results that exist in the vocabulary, you can directly tokenize them based on the vocabulary. For the word segmentation results that do not exist in the vocabulary, you can use the number corresponding to the text included in the word segmentation result to tokenize the word segmentation result.

[0101] For example, to tokenize the text sequence "ABCDEFG", it is first segmented to obtain the segmentation results "AB", "CD", and "EFG", where the corresponding numbers of the segmentation results "AB" and "CD" in the preset vocabulary are 001 and 005 respectively, and "EFG" cannot be found in the preset vocabulary. Therefore, the corresponding numbers of "E", "F", and "G" in the preset vocabulary can be used as tokens corresponding to the segmentation results.

[0102] In a possible embodiment, the training text sequence may also be subjected to position tokenization, category tokenization, etc., which may be specifically determined according to the function of the target text processing model.

[0103] The text filtering model to be trained can filter the input text sequence. The text filtering model to be trained can be a model with a relatively simple structure and low computational and storage complexity, and its dimension and number of layers can be relatively low, such as a linear Attention model with a relatively low number of layers, a CNN model (Convolutional Neural Networks), etc., and specifically can be a two-layer 64-dimensional CNN model.

[0104] As a possible embodiment, the text filtering model to be trained can score each text in the input text sequence, and retain the text with higher score as the target text sequence according to the retention number. The above retention number is a parameter in the text filtering model, and its initial value is a preset value. As a possible implementation method, each text can be sorted in order from high to low according to the score of each text, and the highest ranking retention number of texts are taken as the target text sequence. As another possible implementation method, the retention number of texts can be randomly selected from the texts with scores higher than the preset score threshold as the target text sequence.

[0105] In a possible embodiment, the token sequence obtained after tokenizing the training text sequence can be input into the text filtering model to be trained, so that the text filtering model embeds the token sequence to obtain a vector sequence corresponding to the training text sequence, and the training text sequence is filtered based on the vector sequence to obtain a target text sequence.

[0106] The text filtering model can embed the training text sequence, which means vectorizing the tokens to obtain a text embedding sequence. The text embedding sequence is an N*D matrix, where N is the length, that is, the number of tokens, and D is the vector dimension corresponding to the token, where the vector corresponding to the token is pre-set for each token.

[0107] As another possible implementation, the training text sequence may be input into a text filtering model, and the text filtering model tokenizes and embeds the training text sequence.

[0108] In a possible embodiment, the text filtering model can input the target text sequence into the target text processing model according to the order of each text in the target text sequence in the training text sequence. The target text processing model can be an NLP (Natural Language Processing) model, a model with a more complex structure, an LSTM model (Long Short-Term Memory), CNN, RNN (Recurrent Neural Network), bert, flash, etc. As a possible implementation method, the target text processing model can be a 12-layer 256-dimensional CNN model. The functions of the target text processing model can be classification, associative word output, etc., which are not specifically limited in the present invention. The above-mentioned target text processing model can include an embedding layer, a convolutional layer, a pooling layer, etc.

[0109] In a possible embodiment, the target text processing model can be pre-trained in advance. Specifically, each training text sequence can be input into the text processing model to be trained, and the MLM (Masked Language Modeling) task can be used for the sequence output by the text processing model to be trained for the training text sequence, so that all parameters of each layer in the text processing model to be trained can be fully trained. In a possible embodiment, the target text processing model can also be a trained target text processing model, which is not specifically limited in the present invention.

[0110] The target text processing model can process the target text sequence and output a target prediction value for the target text sequence. The target prediction value can be determined according to the function of the target text processing model, such as the probability of user overdue payment, the probability of positive news tendency, etc.

[0111] The target text loss can be obtained based on the target prediction value and the label of the training text sequence. The target text loss can be a cross entropy loss, a mean square error loss, etc., which can be determined according to the actual application scenario.

[0112] Then, the text filtering model and the target text processing model can be trained based on the target text loss. For example, the text filtering model and the target text processing model can be trained by minimizing the target text loss through the gradient descent method.

[0113] In a possible embodiment, what needs to be controlled in the text filtering model are hyperparameters and the number of model layers, dimensions, drop-rate, etc. The hyperparameters include N and K, where N is the maximum length of the text sequence that can be processed by the text filtering model, and K is the maximum length of the target text sequence, that is, the maximum retention length.

[0114] For the hyperparameter K, if a pre-trained target task model is used, in order to make the training smoother, the initial value of the maximum retention length actually used should not be directly set to K, because this will destroy the semantic integrity of the input of the pre-trained target task model, thereby affecting the final fitting effect of the model. Therefore, in actual training, the K value to be trained can be reduced from N, and gradually approach the target value K as the training progresses, so that the pre-trained target task model can gradually and adaptively learn the sequence of filter outputs during the filter training process.

[0115] Specifically, the K value to be trained can decrease linearly or exponentially with the number of training steps:

[0116] K_train=α+(1-α)*(rate^β)

[0117] α=K / N

[0118] rate = step / Total_step

[0119] Where K_train is the maximum retention length to be trained, β is the power hyperparameter, K is the maximum retention length, step is the number of training steps, and Total_step is the total number of training steps.

[0120] In a possible embodiment, the text filtering model can be obtained by self-supervised learning. Specifically, the text filtering model includes a first text filter and a second text filter, and the first text filter and the second text filter are connected in parallel.

[0121] The first text filter and the second text filter may have the same structure or different structures, and their internal initial parameters may be the same or different, which is not specifically limited in the present invention. In a possible embodiment, the first text filter and the second text filter may both be linear Attention models.

[0122] Accordingly, the training text sequence can be input into the first text filter and the second text filter, and the first text filter and the second text filter can filter the training text sequence respectively to obtain the first target text sequence and the second target text sequence.

[0123] Specifically, the first text filter and the second text filter can respectively score each text contained in the training text sequence; and based on the score and retention length of each text, the first target text sequence and the second text sequence are obtained. The retention length is a parameter in the first text filter and the second text filter, and its initial value is a preset value.

[0124] The first text filter and the second text filter can segment the training text sequence, score each segmentation result, and obtain the first target text sequence and the second target text sequence based on the retention length and the score of each segmentation result. The specific implementation method can be found in the above description, and only a brief description is given here, and no further details are given.

[0125] As described above, the training text sequence may be tokenized before being input into the text filtering model. In a possible embodiment, the first text filter and the second text filter may score each token in the token sequence obtained after tokenization, and obtain the first target text sequence and the second target text sequence according to the retention length and the score of each token.

[0126] In a possible embodiment, the first text filter and the second text filter may embed an input training text sequence, where the input training text may be the training text itself or a token sequence corresponding to the training text.

[0127] Specifically, the first text filter and the second text filter embed the token sequence to obtain a text embedding sequence, and then the first text filter and the second text filter respectively score the vectors corresponding to each token in the text embedding sequence, and arrange the tokens in descending order based on the scores, and select the topK tokens to form the target text sequence. Among them, the text sequence composed of the topK tokens selected by the first text filter is called the first target text sequence, and the text sequence composed of the topK tokens selected by the second text filter is called the second target text sequence.

[0128] In a possible embodiment, the first filter and the second filter may share an embedding layer, and after the embedding layer embeds the token to obtain a text embedding sequence, each text embedding sequence may be input into the first text filter and the second text filter.

[0129] Afterwards, the first target text sequence and the second target text sequence may be respectively input into the target text processing model, so that the target text processing model outputs a first prediction value for the first target text sequence and outputs a second prediction value for the second target text sequence.

[0130] Then, a first text loss can be obtained based on the first prediction value and the label of the training text sequence; a second text loss can be obtained based on the second prediction value and the label of the training text sequence; and based on the first text loss and the second text loss, the first text filter and the second text filter are trained to obtain a target text filtering model.

[0131] In a possible embodiment, the first target text sequence includes a common text and a first difference text, wherein the common text is the same text in the first target text sequence and the second target text sequence, the first difference text is the text that exists in the first target text sequence but does not exist in the second target text sequence, and the second target text sequence includes the common text and the second difference text, and the second difference text is the text that exists in the second target text sequence but does not exist in the first target text sequence;

[0132] Exemplarily, the training text sequence length N = 10, the first target text sequence selected by the first text filter is {1, 2, 3, 4, 5, 6, 7}, and the second target text sequence selected by the second text filter is {4, 5, 6, 7, 8, 9, 10}, then the common text = {4, 5, 6, 7}, the first difference text = {1, 2, 3}, and the second difference text = {8, 9, 10}.

[0133] In a possible embodiment, a filter loss may be obtained based on the first text loss, the second text loss, the score of the first difference text, and the score of the second difference text, and the first text filter and the second text filter may be trained based on the filter loss to obtain a target text filtering model, wherein the filter loss is positively correlated with the difference between the first text loss and the second text loss, and is positively correlated with the difference between the score of the first difference text and the score of the second difference text;

[0134] The filter loss is a loss obtained by taking into account the losses of the first text filter and the second text filter itself. In a possible embodiment, the filter loss may be:

[0135] Loss-filter=(loss A -loss B )*(Aleftscore -B leftcore )

[0136] where loss A is the first text loss of the first text filter, and loss B is the second text loss of the second text filter, A leftscore is the score of the first differential text, and B leftscore is the score of the second differential text. The score of the first differential text (or the second differential text) can be the average value, the minimum value, etc. of the scores given by the first text filter and the second text filter to the first differential text (or the second differential text).

[0137] In the embodiments of the present invention, the parameters of the first text filter and the second text filter can be adjusted based on the filter loss. In a possible embodiment, gradient descent algorithms such as SGD (Stochastic Gradient Descent) can be used to optimize the first text filter and the second text filter based on the filter loss.

[0138] The above optimization refers to minimizing the filter loss. In a possible embodiment, in the embodiments of the present invention, the gradient adjustment direction is determined based on the filter loss by the gradient descent method. When loss-A < loss-B, it indicates that the first target text sequence selected by the first text filter is better than the second target text sequence. Therefore, it is necessary to enhance the first target text sequence.

[0139] Since (loss A - loss B ) in the filter loss is negative, and the direction of gradient update by the gradient descent method is the negative direction. That is to say, for (A leftscore - B leftcore ), the gradient is positive. That is to say, for (A leftscore - B leftcore ), through the gradient descent algorithm, this part of the value will increase. Specifically, A leftscore increases and B leftcore decreases, thereby achieving the purpose of enhancing A-left and suppressing B-left, so that the first text filter and the second text filter increase the score of the first differential text and decrease the score of the second differential token, achieving the purpose of enhancing the first target text sequence and suppressing the second target text sequence. Furthermore, it enables both the first text filter and the second text filter to score higher for valid content, that is, enables more valid content to be retained, thereby ensuring the processing effect of the target text processing model.

[0140] In an embodiment of the present invention, the first text filter, the second text filter and the target text processing model can also be trained based on the first text loss, the second text loss and the filter loss to obtain a target text filtering model and a trained target text training model.

[0141] Specifically, Loss=lossA+lossB+loss_filter can be minimized to train the first text filter, the second text filter and the target text processing model.

[0142] By applying the embodiment of the present invention, two text filters are set, and the text filtering model is trained through the difference between the text sequences selected by the two text filters. There is no need to mark each word in a long text, and the text filtering model is trained based on the marking results, which greatly saves manpower, improves the efficiency of obtaining training text sequences, and further improves the efficiency of model training.

[0143] like Figure 2 As shown, Figure 2 A flowchart of a text filtering model training method provided by an embodiment of the present invention may include the following steps:

[0144] S201, input the text embedding sequence into filter A (filter-Model-A) and filter B (Filter-Model-B) respectively.

[0145] The text embedding sequence is a matrix obtained by tokenizing and vectorizing the training text sequence. The filter A and the filter B here are the first text filter and the second text filter mentioned above.

[0146] S202, filter A and filter B respectively filter the text embedding sequence to obtain a first target text sequence and a second target text sequence, wherein the first target text sequence includes a common text Common and a first difference text A-left, and the second target text sequence includes a common text Common and a second difference text B-left.

[0147] S203, input the first target text sequence and the second target text sequence into the target text processing model respectively. The target text processing model can output prediction values ​​for the first target text sequence and the second target text sequence. Specifically, the first prediction value ProbA is output for the first target text sequence, and the second prediction value ProbB is output for the second target text sequence.

[0148] S204, obtaining a first loss Loss-A based on the first prediction value ProbA and the label of the training text sequence, and obtaining a second loss Loss-B based on the second prediction value ProbB and the label of the training text sequence.

[0149] S205. Obtain filter loss based on ProbA, ProbB, A-left and B-left. Specifically, the filter loss is Loss-filter=(lossA-lossB)*(Aleftscore-B leftscore).

[0150] Finally, the target text processing model, the first text filter and the second text filter can be trained based on the filter loss Loss-filter, the first loss Loss-A and the second loss loss-B.

[0151] The embodiment of the present invention implements a filter model that does not require manual sequence annotation. The filter model is a model optimized for target classification or regression tasks, and is guided by the effect of the target task model to guide the filter model to perform sequence filtering optimization. A reasonable filter loss is designed by designing a pre-A / B filter and using the subsequent target text processing model fitting situation. During training, the filter loss can continuously optimize the content filtered by the A / B filter, so that content information that is effective for modeling the target text processing model is filtered out through the A / B filter. The problem that the filter itself is difficult to obtain effective sequence labels is solved, and at the same time, the model processing length is greatly reduced without reducing the model fitting effect in the processing of long texts with limited effective content.

[0152] In the embodiment of the present invention, the target text processing model is similar to a reward model, which suppresses or enhances the differences in the output of the A / B filters according to the quality of the fitting effect. This technical solution enables the model itself to automatically learn which content needs to be filtered out as the input of the model and which content should be suppressed and eliminated according to the target task during training.

[0153] By using the embodiments of the present invention, there is no need for manual annotation, and there is no need for expensive and low-quality manual annotation work for long text sequences. In addition, compared with the static filtering method based on rules, the method based on the filter model often filters different contents for different sequences. And because static filtering is suitable in some scenarios, but it is detrimental to the target task in some scenarios, and the dynamic method provided by the embodiments of the present invention is non-destructive to the target task, in the long text processing model scenario with a lot of noise, the filter model can automatically help reduce noise, so it is helpful to combat the overfitting of the target task model caused by noise.

[0154] Based on the same inventive concept, an embodiment of the present invention also provides a text filtering method, which can be applied to any electronic device with a text filtering function, such as a server, a computer, or a mobile terminal. Figure 3 As shown, the method may include:

[0155] S301, obtaining a target text to be processed;

[0156] S302, inputting the target text into a pre-trained target text filtering model, so that the target text filtering model filters the target text and inputs it into a pre-trained target text processing model, wherein the target text filtering model is pre-trained by any of the above-mentioned text filtering model training methods;

[0157] S303: Obtain the processing result output by the target text processing model.

[0158] In a possible embodiment, the target text filtering model includes a first text filter and a second text filter; and inputting the target text into the pre-trained target text filtering model includes:

[0159] The target text is input into the first text filter or the second text filter.

[0160] During the training phase, in order for the filter to obtain the target sequence for downstream tasks through comparison, the target text processing model that consumes more computing and storage resources needs to be used twice to obtain the label of which is better, A-left or B-left.

[0161] However, during the training process, the A / B filter will gradually converge, and the output sequence consistency rate of a fully trained A / B filter can reach more than 95%. Therefore, in the inference stage, it is only necessary to input the target text into the A filter or the B filter, apply the filter only once, and hand over the sequence output by the filter to the target text processing model for semantic calculation to obtain the final processing result.

[0162] Among them, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the present invention are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0163] According to another aspect of the present invention, a text filtering model training device is provided. Figure 4 As shown, the apparatus 400 may include:

[0164] The first input module 401 is used for inputting each training text sequence in the training set into the text filtering model to be trained, so that the text filtering model filters the training text sequence to obtain a target text sequence;

[0165] A first output module 402, configured to input the target text sequence into a target text processing model, so that the target text processing model outputs a target prediction value for the target text sequence;

[0166] A loss acquisition module 403 is used to obtain a target text loss based on the target prediction value and the label of the training text sequence;

[0167] The training module 404 is used to train the text filtering model based on the target text loss to obtain a target text filtering model.

[0168] According to another aspect of the present invention, a text filtering device is provided. Figure 5 As shown, the apparatus 500 may include:

[0169] An acquisition module 501 is used to acquire a target text to be processed;

[0170] A second input module 502 is used to input the target text into a pre-trained target text filtering model, so that the target text filtering model filters the target text and inputs it into the pre-trained target text processing model, wherein the target text filtering model is pre-trained by any of the above-mentioned text filtering model training methods;

[0171] The second output module 503 is used to obtain the processing result output by the target text processing model.

[0172] The exemplary embodiment of the present invention further provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform a method according to an embodiment of the present invention when executed by the at least one processor.

[0173] Exemplary embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present invention.

[0174] An exemplary embodiment of the present invention further provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform a method according to an embodiment of the present invention.

[0175] refer to Figure 6 , a block diagram of an electronic device 600 that can be used as a server or client of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0176] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0177] A plurality of components in the electronic device 600 are connected to the I / O interface 605, including: an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609. The input unit 606 may be any type of device capable of inputting information to the electronic device 600, and the input unit 606 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 607 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 608 may include, but is not limited to, a disk, an optical disk. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0178] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above. For example, in some embodiments, the above-mentioned text filtering model training method or text filtering method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via the ROM 602 and / or the communication unit 609. In some embodiments, the computing unit 601 may be configured to perform the above-mentioned text filtering model training method or text filtering method by any other appropriate means (e.g., by means of firmware).

[0179] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0180] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0181] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0182] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0183] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0184] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

Claims

1. A text filtering model training method, characterized in that: The method comprises: For each training text sequence in the training set, the training text sequence is input into the text filtering model to be trained, so that the text filtering model filters the training text sequence to obtain a target text sequence; Inputting the target text sequence into a target text processing model so that the target text processing model outputs a target prediction value for the target text sequence; Obtaining a target text loss based on the target prediction value and the label of the training text sequence; The text filtering model is trained based on the target text loss to obtain a target text filtering model.

2. The method according to claim 1, characterized in that The text filtering model includes a first text filter and a second text filter, wherein the first text filter and the second text filter are connected in parallel; The text filtering model filters the training text sequence to obtain a target text sequence, including: The first text filter and the second text filter filter the training text sequence respectively to obtain a first target text sequence and a second target text sequence; The step of inputting the target text sequence into a target text processing model so that the target text processing model outputs a target prediction value for the target text sequence includes: Inputting the first target text sequence and the second target text sequence into the target text processing model respectively, so that the target text processing model outputs a first prediction value for the first target text sequence and outputs a second prediction value for the second target text sequence; The step of obtaining a target text loss based on the target prediction value and the label of the training text sequence includes: Obtaining a first text loss based on the first prediction value and the label of the training text sequence; Obtaining a second text loss based on the second prediction value and the label of the training text sequence; The training of the text filtering model based on the target text loss to obtain the target text filtering model includes: Based on the first text loss and the second text loss, the first text filter and the second text filter are trained to obtain a target text filtering model.

3. The method according to claim 2, characterized in that The first text filter and the second text filter filter the training text sequence respectively to obtain a first target text sequence and a second target text sequence, including: The first text filter and the second text filter respectively score each text contained in the training text sequence; The first text filter and the second text filter obtain the first target text sequence and the second text sequence based on the scores and retention lengths of the texts. The retention length is a parameter in the first text filter and the second text filter, and an initial value of the retention length is a preset value.

4. The method according to claim 3, characterized in that The first target text sequence includes a common text and a first difference text, wherein the common text is the same text in the first target text sequence and the second target text sequence, the first difference text is the text that exists in the first target text sequence but does not exist in the second target text sequence, and the second target text sequence includes the common text and a second difference text, wherein the second difference text is the text that exists in the second target text sequence but does not exist in the first target text sequence; The method further comprises: Based on the first text loss, the second text loss, the score of the first difference text, and the score of the second difference text, a filter loss is obtained, wherein the filter loss is positively correlated with the difference between the first text loss and the second text loss, and is positively correlated with the difference between the score of the first difference text and the score of the second difference text; The first text filter and the second text filter are trained based on the filter loss to obtain a target text filtering model.

5. The method according to claim 4, characterized in that The step of training the first text filter and the second text filter based on the filter loss to obtain a target text filtering model includes: The filter loss is minimized by a gradient descent algorithm to obtain a target text filtering model.

6. The method according to claim 4, characterized in that The target text processing model is a pre-trained model, and the method further includes: The first text filter, the second text filter and the target text processing model are trained based on the first text loss, the second text loss and the filter loss to obtain a target text filtering model and a trained target text training model.

7. The method according to claim 1, characterized in that The training set is predetermined by the following steps: Based on the function of the target text processing model, determining a candidate text sequence whose label corresponds to the function; Based on the proportion of each of the labels in the candidate text sequence, the candidate text sequence is sampled so that the difference in the number of each of the labels in the training set is less than a preset difference threshold.

8. The method according to claim 1, characterized in that: The method further comprises: Performing word segmentation on the training text sequence to obtain a word segmentation result; Based on a preset vocabulary, tokenize the word segmentation result to obtain a token sequence corresponding to the training text sequence; The step of inputting the training text sequence into a text filtering model to be trained so that the text filtering model filters the training text sequence to obtain a target text sequence includes: The token sequence is input into a text filtering model to be trained so that the text filtering model embeds the token sequence to obtain a vector sequence corresponding to the training text sequence, and the training text sequence is filtered based on the vector sequence to obtain a target text sequence.

9. A text filtering method, characterized in that: The method comprises: Get the target text to be processed; Inputting the target text into a pre-trained target text filtering model, so that the target text filtering model filters the target text and inputs it into a pre-trained target text processing model, wherein the target text filtering model is pre-trained by the text filtering model training method according to any one of claims 1 to 8; Obtain the processing result output by the target text processing model.

10. The method according to claim 9, characterized in that The target text filtering model includes a first text filter and a second text filter; the step of inputting the target text into the pre-trained target text filtering model includes: The target text is input into the first text filter or the second text filter.

11. A text filtering model training device, characterized in that: The device comprises: A first input module is used for inputting each training text sequence in the training set into the text filtering model to be trained, so that the text filtering model filters the training text sequence to obtain a target text sequence; A first output module, used for inputting the target text sequence into a target text processing model, so that the target text processing model outputs a target prediction value for the target text sequence; A loss acquisition module, used to obtain a target text loss based on the target prediction value and the label of the training text sequence; The training module is used to train the text filtering model based on the target text loss to obtain a target text filtering model.

12. A text filtering device, characterized in that: The device comprises: An acquisition module is used to acquire the target text to be processed; A second input module is used to input the target text into a pre-trained target text filtering model, so that the target text filtering model filters the target text and inputs it into the pre-trained target text processing model, wherein the target text filtering model is pre-trained by the text filtering model training method according to any one of claims 1 to 8; The second output module is used to obtain the processing result output by the target text processing model.

13. An electronic device comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1-8 or 9-10.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1-8 or 9-10.

Images