An inquiry abnormality detection method based on semi-supervised learning

Abstract

The application discloses a kind of based on semi-supervised learning's inquiry exception detection method, 1) preparation is used for foreign trade inquiry detection training and test data, the data includes normal inquiry data, junk inquiry data;2) data preprocessing, step includes noise word morph transformation, sample expansion;3) loading Bert-Large model, the Embedding of text is extracted;4) setting generator and discriminator model architecture: define generator Generator, Discriminator model structure and loss function;5) loading Generator, Discriminator model, using normal inquiry vector data, train model;Just can test the accuracy of model.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of deep learning and natural language processing, and particularly relates to a query abnormality detection method based on semi-supervised learning. BACKGROUND

[0002] The query is very important for a foreign trade company, and is one of the communication tools necessary for the communication between buyers and sellers. With the increasing frequency of use of the query, junk queries also come along. The outbreak of junk queries is a problem that plagues every supplier. At present, there is no exact and effective method in the foreign trade industry to ensure that the overseas market trade process is not affected by junk queries. Although the number of junk queries is large, the proportion is still very small compared to normal queries. Therefore, the traditional supervised text classification method cannot solve the high class imbalance ratio classification. Therefore, the semi-supervised method is used to solve the problem of too high class imbalance ratio of junk queries.

[0003] Semi-supervised learning (SSL) is the full name of Semi-Supervised Learning, which belongs to machine learning (Machine Learning, ML). In the case of only a small number of labeled samples and most of the samples being unlabeled, semi-supervised learning method can be used to label the unlabeled samples according to the similarity between the labeled samples and the unlabeled samples and the potential distribution of the unlabeled samples. The two core ideas are to label the unlabeled samples. BERT is the full name of Bidirectional Encoder Representations from Transformers. As can be seen from the full name, BERT uses the Transformer model and the Encoder part of the Transformer.

[0004] The present application designs a scheme for semi-supervised classification problem, and learns from the GANomaly method: the GANomaly loss function is divided into two parts, the first part is the generator loss, and the second part is the discriminator loss. The GANomaly loss function is divided into two categories, one is the Encoder structure, and the other is the Decoder structure. The Encoder mainly has the function of feature dimension reduction, such as compressing a picture data into a potential vector; on the contrary, the Decoder has the function of dimension increase, such as reconstructing a potential vector into a picture. According to the structure described in the paper, it can be divided into three substructures, which are the generator network G, the encoder network E and the discriminator network D. The present application uses the Transformer and the GAN network to do text anomaly detection. Through semi-supervised deep learning, this method can overcome the problems of too few junk query samples and too high class imbalance ratio. At the same time, it can also overcome the noise interference such as sample synonymy and label error in deep learning. SUMMARY

[0005] The technical problem solved by the present application is to overcome the shortcomings of the prior art and provide an inquiry classification method based on semi-supervised learning.

[0006] To solve the above technical problems, the present application provides an inquiry classification method based on semi-supervised learning, comprising the following steps:

[0007] Step one: prepare data for foreign trade inquiry detection training and testing, the data including normal inquiry data, junk inquiry data;

[0008] Step two: data preprocessing, the steps including noise transformation, sample expansion;

[0009] Step three: Word Embedding (extraction), load Bert-Large model, extract the Embedding of the text; the essence of Embedding is to replace the original feature of higher dimension with a lower dimensional vector; wherein the Bert-Large model is a 24-layer Encoder Transformer structure;

[0010] Step four: set the generator and discriminator model architecture: define the generator (Generator), discriminator (Discriminator) model structure and loss function; wherein the generator model includes two feature encoders (Encoder), a feature decoder (Decoder), wherein the role of Encoder is to reduce the dimension of the feature, and the role of Decoder is to generate the feature; and the discriminator model is a multi-layer DNN binary classification model, the output dimension of the second last layer is 256, and the purpose is to calculate the mean square error of Sequence Embedding and Synthetic Embedding;

[0011] Step five: model training, load Generator, Discriminator model, use Sequence Embedding data of normal inquiry to train the model;

[0012] Step six: based on the trained Generator and Discriminator and the embedding of the test set, test the accuracy of the model.

[0013] The step one, from the existing inquiry data based on inquiry theme (Object), inquiry content (Content), and inquiry mark field (Check_it_Effect), the text label pair (Object+Content, Label) is processed, and the normal inquiry data and the garbage inquiry data set are distinguished.

[0014] The step two, the data preprocessing further includes the following steps:

[0015] The product name in the text label pair is transformed into a text with different word forms by using word form reduction, word form expansion, symbol conversion and other methods to expand the sample.

[0016] In the step three, the pre-trained language model Bert-Large model is used, the mapping of the sample obtained by the word segmenter is input into the model, and the latent feature vector (Sequence Embedding) of each sample is obtained.

[0017] In the step four, the generator is composed of Encode1->Decoder1->Encode2 structure, the purpose is to generate a sentence vector (Synthetic Embedding) as similar as possible to the original sentence feature vector; and the discriminator is a multi-layer DNN binary classification network, and the model establishment further includes the following steps:

[0018] Step 4-1: define two same Encoder structures, the network structure is multi-layer DNN;

[0019] Step 4-2: define a Decoder structure, the network structure is DNN->GPT2->DNN in turn.

[0020] Step 4-3: define the discriminator, the network structure is a multi-layer DNN binary classification network;

[0021] Step 4-4: define the loss function, the Generator loss function is divided into three parts: the mean square error of the hidden variables obtained by Encode1 and the hidden variables obtained by Encode2 (Lenc); the norm error of Synthetic Embedding obtained by Decoder1 and Sequence Embedding (Lcon); the mean square error of the output of Sequence Embedding in Discriminator and the output of Synthetic Embedding in Discriminator (Ladv). Add the three errors according to certain weight to get the total error of the generator. The discriminator loss function is the discriminator loss of GAN.

[0022] In the fifth step, based on the model defined in the fourth step, the SequenceEmbedding of the normal inquiry sample is loaded, and the model is trained;

[0023] In the sixth step, the category of the inquiry is predicted, the SequenceEmbedding of a sample is input into the optimal model trained in the fifth step, the loss size of the SequenceEmbedding and the Synthetic Embedding is returned, and based on the given loss threshold, the sample greater than the threshold is identified as a junk inquiry.

[0024] Beneficial effects: the inquiry abnormality detection method based on semi-supervised learning mainly detects junk inquiries of foreign trade products, (1) the inquiry content and the inquiry theme are input, and whether the inquirer is a junk inquiry can be accurately distinguished, (2) various deformations of junk inquiries can be accurately classified by the model, (3) the method is suitable for large-scale inquiry functions, can help enterprises reduce labor costs and has high accuracy, (4) is suitable for search engine retrieval, product listing, data cleaning, intelligent recommendation and other business scenarios. The method and system for automatically identifying junk inquiries in the foreign trade industry can detect junk inquiry data by using three Encoder encoding models and one Decoder model only by giving normal inquiry data. The whole process is end-to-end, and the junk inquiry can be detected without any intervention of the junk inquiry data set. The method can help operators free themselves from time-consuming and laborious auditing tasks, and can reduce the time cost of labeling the data set without junk inquiries. The system running the method has fast junk inquiry detection speed and high accuracy, and can usually complete the detection of a junk inquiry within 20ms, and has high applicability. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 The figure is a flowchart of the product classification method based on keywords in the exemplary embodiment of the application;

[0026] Figure 2 The figure is a structure diagram of the generator and the discriminator. DETAILED DESCRIPTION

[0027] The application will be further described below in combination with the drawings and exemplary embodiments:

[0028] As Figure 1 shown, the application discloses an inquiry classification method based on normal inquiry content, which comprises:

[0029] Step 11: Prepare the training set, extract the fields Object, Content and Check it effect content from the data warehouse containing the inquiry data, and after simple data processing, use it as a training sample. Each sample is a text label pair containing (Object+Content, Label).

[0030] Step 12: Data preprocessing, do word form transformation on the extracted inquiry data set to expand the data set. The word form transformation measures taken include singular-plural conversion, adjective-adverb conversion, symbol conversion. Use the word form reduction tool to convert plural, adjectives, and adverbs to their original form, while recording these converted words. Reverse the original word in the training set to expand it into a plural, adjective, or adverb. In addition, English phrases often use "-" and "_" symbols to connect two words. These symbols are converted to spaces or removed without changing the basic meaning of the two words. Therefore, symbol conversion is added for preprocessing. By the above method, the product name and keyword in the text label pair are transformed into text with different word forms to expand the sample, avoiding the problem of misclassification caused by the same meaning but different input forms of keywords during training and testing.

[0031] Step 13: Word Embedding, use the pre-trained language model Bert-Large model to perform Word Embedding on each sample to extract the latent feature vector representation of each sample. Before obtaining the sentence vector, first input the text content of each sample into the Bert tokenizer to obtain the word mapping (input_ids) and attention mask (attention_mask); Then input it into the Bert-Large model to obtain their 512-dimensional feature representation vector (SequenceEmbedding).

[0032] Step 14: Model definition, design generator and discriminator model architecture, build the training network. At the same time, design the loss function, build the code service. The specific structure of the model is as shown in Figure 2

[0033] Step 141: as Figure 2 ​As shown, for the Generator structure, after the Generator accepts the input, it first enters the Encoder-1 (Encoder-1) to obtain a 256-dimensional feature vector; then inputs the output of the Encode-1 into the Decoder-1 (Decoder-1) to obtain a 512-dimensional feature vector; finally, inputs the output of the Decoder-1 into the Encoder-2 (Encoder-2) to perform feature dimension reduction to obtain a 256-dimensional feature vector. For the Encoder-1, it is a multi-layer DNN network, and its purpose is to perform feature dimension reduction and coding on the Sequence Embedding, reduce the dimension to 256, and reduce the subsequent calculation cost of the network; for the Decoder-1 structure, after the data is input from the Encoder-1 to the Decoder-1, it is first input into the fully connected layer 1 (Dense-1) to perform feature dimension increase to 768; then inputs the 768-dimensional feature into the GPT2 model to perform feature decoding, and outputs a 768-dimensional feature vector after decoding; finally, inputs the output of the GPT2 into the fully connected layer 2 (Dense-2) to perform feature dimension reduction to 512 to generate the Synthetic Embedding. The Synthetic Embedding should be infinitely close to the Sequence Embedding to some extent.

[0034] Step 142: as shown Figure 2 For the Discriminator, it is a multi-layer DNN network, and its purpose is to perform binary classification on the Sequence Embedding and the Synthetic Embedding. For the Sequence Embedding and the Synthetic Embedding of the same sample, after passing through the Discriminator, the same output should be obtained.

[0035] Step 143: the loss function includes two parts, one part is the Generator Loss, and the other part is the Discriminator Loss.

[0036] For the Generator Loss, it is divided into three parts. First, the Adversari Loss loss function, see formula (1), which corresponds to Figure Two the Ladv function in the formula, which is an L2 loss. f(x) represents the output of a certain intermediate layer of the discriminator network. Its purpose is to make the Sequence Embedding and the Synthetic Embedding as similar as possible.

[0037] Ladv = Ex～px ||f(x)-E x～px f(G(x))||2 Equation (1)

[0038] For the Contextual Loss loss function, see Equation (2), which corresponds to Figure 2 the Lcon function, which is an LI loss. Again, the goal is to make the Sequence Embedding and the Synthetic Embedding as similar as possible.

[0039] Lcon = E x～px ||x-G(x)||1 Equation (2)

[0040] Finally, for the Encoder loss function, see Equation (3), which corresponds to Figure 2 the Lenc function. The main goal of this loss function is to make the model perform better during inference.

[0041] Lenc = E x～px ||G E (x)-E(G(x))||2 Equation (3)

[0042] And for the overall Generator Loss, this is the sum of the three loss functions with certain weights, see Equation (4).

[0043] L = wadv*Ladv + wcon*Lcon + wenc*Lenc Equation (4)

[0044] Where Wadv, Wcon, Wenc are set according to the default parameters in GANomaly, Wcon = 50, Wadv = Wenc = 1.

[0045] For the Discriminator Loss, see Equation (5), which is the same as the loss function of the GAN model.

[0046]

[0047] Step 15: Model training, according to the Sequence Embedding obtained in step 13, using the model and loss function defined in step 14, and the training steps defined in steps 141 and 142, to train the model.

[0048] Step 16: The category of the inquiry is predicted, the inference of the model is carried out based on the optimal model trained in step 15. A Sequence Embedding of an inquiry subject and inquiry content obtained in step 13 is input into the model, the L2 loss (in the range of [0, 1]) of the inquiry is calculated according to formula (6), and the category of the inquiry is judged according to the set threshold value (the default threshold value is set to 0.5, and greater than 0.5 is a junk inquiry), and the category is fed back to the user.

[0049] L(x) = ||G E (x) - E(G(x))||2 Formula (6)

[0050] The application is mainly used for providing an inquiry abnormality detection method based on normal inquiry subject and content. The method can efficiently and accurately complete the inquiry classification function. The classification model can accurately detect the junk inquiry after the model is trained without relying on the junk inquiry training. The inquiry can be quickly classified, and the enterprise can reduce the labor cost.

[0051] The above examples do not limit the application in any way, and other improvements and applications made by equivalent transformation to the above examples belong to the protection scope of the application.

Claims

1. A method for detecting inquiry abnormality based on semi-supervised learning, characterized in that, Comprising the following steps: Step one: preparing data for foreign trade inquiry detection training and testing, the data includes normal inquiry data, junk inquiry data; In step one, based on the inquiry subject Object and the inquiry content Content, the existing inquiry data is processed into the form of text label pair Object+Content, Label, and the normal inquiry data and junk inquiry data set are distinguished; Step two: data preprocessing, including morphological transformation, sample expansion; Step three: load Bert-Large model, extract Sequence Embedding of text; Wherein the Bert-large model is a 24-layer Encoder Transformer structure; Step four: set the generator and discriminator model architecture: define the generator Generator, discriminator Discriminator model structure and loss function; Wherein the generator model includes two feature encoders Encoder and one feature decoder Decoder, wherein the function of Encoder is to reduce the dimension of features, and the function of Decoder is to generate features; And the discriminator model is a multi-layer DNN binary classification model, the output dimension of the second last layer is 256, which is used to calculate the mean square error of the sentence feature vector Sequence Embedding extracted in step three and the sentence vector SyntheticEmbedding generated by the generator; Step five: load Generator, Discriminator model, use Sequence Embedding data of normal inquiry to train the model; Step six: input the sequence embedding of the test set into the trained Generator, Discriminator model to judge whether it is a junk inquiry.

2. The detection method of claim 1, wherein: In step two, the data preprocessing further comprises the following steps: using the morphological reduction, morphological expansion and symbol conversion method to transform the product name in the text label pair into a text with different word forms to expand the sample.

3. The detection method of claim 1, wherein: In step three, the pre-trained language model Bert-Large model is used, the sentence mapping obtained by the tokenizer is input into the model, and the sentence feature vector Sequence Embedding of each sample is obtained.

4. The detection method of claim 1, wherein: In step four, the generator is composed of Encode1->Decoder1->Encode2 structure, the purpose is to generate a sentence vector Synthetic Embedding similar to the original sentence feature vector; The discriminator is a multi-layer DNN binary classification network, and the model establishment further comprises the following steps: Step 4-1: define two same Encoder structures, the network structure is multi-layer DNN; Step 4-2: define a Decoder structure, the network structure is DNN->GPT2->DNN in turn; Step 4-3: Define the discriminator, the network structure is a multi-layer DNN binary classification network; Step 4-4: Define the loss function, the Generator loss function is divided into three parts: the mean square error of the hidden variable obtained by Encode1 and the hidden variable obtained by Encode2 Lenc; the norm error of Synthetic Embedding obtained by Decoder1 and SequenceEmbedding Lcon; the mean square error of the output of Sequence Embedding in Discriminator and the output of Synthetic Embedding in Discriminator Ladv; add the three errors according to certain weights to obtain the total error of the generator; the discriminator loss function is the discriminator loss of GAN.

5. The detection method of claim 1, wherein: In the sixth step, based on the optimal model trained in the fifth step, input a sample Sequence Embedding into the model, return the loss size of Sequence Embedding and Synthetic Embedding, and based on the given loss threshold, identify the samples greater than the threshold as junk inquiries.

Images