A classification method and device based on multi-model fusion

Abstract

The embodiment of the present invention discloses a classification method and device based on multi-model fusion, which is used to solve the problem of inaccurate classification in the prior art. The method includes inputting the text to be classified into N different domain models to obtain N scores, where N is an integer greater than 1; determining the weights of the N domain models; determining the total score of the classification result according to the N scores and the weights of the N domain models; and classifying the text to be classified according to the relationship between the total score of the classification result, a first threshold and a second threshold, wherein the first threshold is less than the second threshold, the first threshold is related to the missed detection rate, and the second threshold is related to the false alarm rate. By fusing the scores of the N domain models, the total score of the final classification result is obtained, which has high robustness and high classification accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a classification method and device based on multi-model fusion. Background Art

[0002] With the rapid development of information technology and the Internet, the amount of text data generated is increasing, making the research of text mining technology more and more important. How to extract valuable information from these large amounts of text data and classify them is an important goal of text mining. Text classification, or automatic text classification, refers to the process by which a computer maps a text containing information to a certain category or several categories of topics given in advance. As an efficient information retrieval and mining technology, it plays an important role in the management of text data and can largely solve the problem of complex and chaotic information. Text classification uses natural language processing, data mining and machine learning technologies to effectively automatically classify different types of text and discover patterns.

[0003] Text classification is an important part of text mining. Its main task is to divide a given text set into one or more known category sets. At present, text classification has been applied to many fields, such as sentiment analysis, topic classification, spam detection, etc. At present, there are many methods for text classification. On the one hand, there are traditional classification methods, such as hidden Markov model, TFIDF, naive Bayes, decision tree, SVM, topic model, etc. The advantages of this type of classification method are simplicity, speed, and the ability to handle text classification of hundreds of thousands or millions of lengths. The disadvantage is that the correlation between word order and semantics is not considered, and the effect is not ideal when the number of texts is insufficient. Another type is deep learning methods, such as CNN, RNN, LSTM, TextCNN, BERT, XLNet, etc. This type of classification method is usually end-to-end. The deep model can directly realize the embedding of text (word embedding) through the network structure, convert words into vectors, and then classify through subsequent multi-layer networks.

[0004] Traditional classification methods usually assume that training data and test data follow the same data distribution. However, in actual applications, due to differences in the time and path of data collection, training data and test data may satisfy different data distributions, so the use of traditional classification methods may lead to inaccurate classification problems. Summary of the invention

[0005] The embodiment of the present invention provides a classification method and device based on multi-model fusion, which are used to solve the problem of inaccurate classification in the prior art.

[0006] In a first aspect, an embodiment of the present invention provides a classification method based on multi-model fusion, the method comprising inputting a text to be classified into N different domain models to obtain N scores, where N is an integer greater than 1; determining the weights of the N domain models; determining a total score of the classification result based on the N scores and the weights of the N domain models; classifying the text to be classified based on the relationship between the total score of the classification result, a first threshold and a second threshold, wherein the first threshold is less than the second threshold, the first threshold is related to the missed detection rate, and the second threshold is related to the false alarm rate.

[0007] Based on the above scheme, by fusing the scores of N domain models, the total score of the final classification result is obtained with high robustness and high classification accuracy.

[0008] In a possible implementation, before inputting the text to be classified into N different domain models to obtain N scores, the method further includes: performing regular matching on the text to be classified with a keyword vocabulary to filter out some normal text or abnormal text.

[0009] In a possible implementation, determining the weights of the N domain models includes: determining the target domain to which the to-be-classified text belongs through a conditional random field (CRF) annotation sequence; and setting the weight of the domain model corresponding to the target domain to be greater than the weights of the remaining N-1 domain models.

[0010] By identifying the field tendency of the text to be classified and setting a larger weight for similar fields, it helps to improve the classification accuracy.

[0011] In one possible implementation, the N domain models include a naive Bayes model and a deep model; the text to be classified is input into N different domain models to obtain N scores, including: if the length of the text to be classified is not greater than a third threshold, the text to be classified is input into the naive Bayes model; if the length of the text to be classified is greater than the third threshold, the text to be classified is input into the deep learning model.

[0012] Through the combination of the Naive Bayes algorithm and the deep learning algorithm, different algorithms are assigned corresponding roles in the classification task. That is, if the length of the text to be classified is not greater than the third threshold, the Naive Bayes model is used; if it is greater than the third threshold, the deep model is used. This enables multiple algorithms to work together to produce better classification results.

[0013] In a possible implementation, the missed detection rate = the number of illegal texts misidentified as normal texts by the domain model / the total number of texts; the false alarm rate = the number of normal texts misidentified as illegal texts by the domain model / the total number of texts.

[0014] Furthermore, the missed detection rate is calculated using the true labels. When the missed detection rate reaches one in ten thousand for the first time, the score corresponding to the text is determined to be threshold one; when the false alarm rate reaches one in one thousand for the first time, the score corresponding to the text is determined to be threshold two.

[0015] In this way, the missed detection rate is less than one in ten thousand, and the false alarm rate is less than one in a thousand, which can save more than 10% of labor costs.

[0016] In a possible implementation, the text to be classified is classified according to the relationship between the total score of the classification result, the first threshold and the second threshold, including: determining the text whose total score of the classification result is less than the first threshold as normal text; and determining the text whose total score of the classification result is greater than the second threshold as abnormal text.

[0017] In a possible implementation, before inputting the text to be classified into N different domain models, it also includes: collecting sample data through different channels and labeling the sample data; wherein the data includes at least one of the following: personal privacy information; advertising information; politically sensitive information; pornographic information; gambling information; hazardous chemical information; violent and bloody information; and terrorist information.

[0018] Through sample data collected from different channels, we can successfully identify illegal text information such as personal privacy information, advertising information, politically sensitive information, pornographic information, gambling information, hazardous chemical information, violent and bloody information, and terrorist information. The number of texts opened correctly accounts for more than 85% of the total texts.

[0019] In a second aspect, an embodiment of the present invention provides a classification device, comprising: an input unit for inputting a text to be classified into N different domain models to obtain N scores, wherein N is an integer greater than 1; a determination unit for determining the weights of the N domain models; and determining a total score of the classification result based on the N scores and the weights of the N domain models; the classification unit for classifying the text to be classified based on the relationship between the total score of the classification result, a first threshold and a second threshold, wherein the first threshold is less than the second threshold, the first threshold is related to the missed detection rate, and the second threshold is related to the false alarm rate.

[0020] In a possible implementation, the classification device further includes a matching unit, which is used to perform regular matching on the text to be classified with a keyword vocabulary to filter out some normal text or abnormal text.

[0021] In a possible implementation, the determination unit is specifically configured to determine the target domain to which the to-be-classified text belongs through a conditional random field (CRF) annotation sequence; and set the weight of the domain model corresponding to the target domain to be greater than the weights of the remaining N-1 domain models.

[0022] In one possible implementation, the N domain models include a naive Bayes model and a deep model; the input unit is specifically used to input the text to be classified into the naive Bayes model if the length of the text to be classified is not greater than a third threshold; if the length of the text to be classified is greater than the third threshold, input the text to be classified into the deep learning model.

[0023] In a possible implementation, the missed detection rate = the number of illegal texts misidentified as normal texts by the domain model / the total number of texts; the false alarm rate = the number of normal texts misidentified as illegal texts by the domain model / the total number of texts.

[0024] In a possible implementation, the classification unit is specifically configured to: determine text whose total score of the classification result is less than the first threshold as normal text; and determine text whose total score of the classification result is greater than the second threshold as abnormal text.

[0025] In a possible implementation, the detection unit may further include a collection unit for collecting sample data through different channels and labeling the sample data; wherein the data includes at least one of the following: personal privacy information; advertising information; politically sensitive information; pornographic information; gambling information; hazardous chemical information; violent and bloody information; and terrorist information.

[0026] In a third aspect, the present application provides a computer-readable storage medium, in which a computer program or instructions are stored. When the computer program or instructions are executed by a classification device, the classification device executes the method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0027] In a fourth aspect, the present application provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a classification device, the classification device executes the method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0028] The technical effects that can be achieved in any of the second to fourth aspects mentioned above can refer to the description of the beneficial effects in the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0030] Figure 1 A schematic diagram of a classification method based on multi-model fusion provided in an embodiment of the present invention;

[0031] Figure 2 A schematic diagram of a multi-model training method flow chart provided in an embodiment of the present invention;

[0032] Figure 3 A schematic diagram of a flow chart of another classification method based on multi-model fusion provided in an embodiment of the present invention;

[0033] Figure 4 A schematic diagram of the structure of a classification device provided by an embodiment of the present invention;

[0034] Figure 5 A schematic diagram of the structure of a classification device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] The embodiment of the present invention adopts a cross-domain classification method to achieve accurate classification of text. The goal of cross-domain classification is to use the training data under the original data distribution (source domain) to train a classifier under the new data distribution (target domain) and perform classification prediction on the target domain data.

[0037] like Figure 1 As shown, a classification method based on multi-model fusion is provided in an embodiment of the present invention, and the method may include the following steps:

[0038] Step 101: input the text to be classified into N different domain models to obtain N scores, where N is an integer greater than 1.

[0039] For the training of N different domain models, please refer to the following Figure 2 The introduction of will not be repeated here.

[0040] Step 102: Determine the weights of the N domain models.

[0041] In one possible implementation, the weights of N domain models are calculated by a model selection algorithm. Specifically, the target domain to which the text to be classified belongs can be determined by the conditional random field CRF tagging sequence, and the weight of the domain model corresponding to the target domain is set to be greater than the weight of the remaining N-1 domain models. Among them, CRF is a discriminant probability model, a sequence labeling algorithm, which receives an input sequence such as X = {x1, x2...xn} and outputs a target sequence Y = {y1, y2...yn}, and can also be regarded as a seq2seq model. Here, uppercase X and Y are used to represent sequences. For example, in the part-of-speech tagging task, the input sequence is a string of words, and the output sequence is the corresponding part of speech. For another example, in the classification standard task, N domain models are input as labels, the domain model feature layer is used as the input sequence, and the output sequence is the weight corresponding to each domain model. By identifying the domain tendency to which the text to be classified belongs, setting a larger weight for similar domains can help improve the classification accuracy.

[0042] Step 103: Determine the total score of the classification result according to the N scores and the weights of the N domain models.

[0043] In a possible implementation, the method of fusing models may include but is not limited to averaging, weighted averaging, sigmoid averaging, bagging, blending, stacking, etc., wherein weighted averaging refers to summing the scores and weights of the outputs of each model, the weights are determined by cross-validation, and the sum of the weights is 1. For example, taking N=3 as an example, the N scores are N1, N2, and N3, and the weights of the N domain models are Q1, Q2, and Q3, respectively, then the total score = (N1×Q1+N2×Q2+N3×Q3) / 3.

[0044] The following takes stacking as an example. Stacking is a multi-level method. In the first layer model, the predicted label is generated by cross-validation as the input feature of each domain model in the second layer (there are several domain models in the first layer and several features in the second layer). The second layer uses the output of the first layer as input, and is learned by each domain model in the second layer, and outputs the predicted result. Generally, the greater the difference in domain models, the better the fusion effect. In practice, the predicted label features can also be added to the original features, and then the reused model is trained again. Furthermore, a custom algorithm is used in the second layer to obtain the weights of different models according to the text content, and then the weighted sum of the scores of each model is obtained to obtain the final total score.

[0045] Step 104, classifying the text to be classified according to the total score of the classification result, the relationship between the first threshold and the second threshold, wherein the first threshold is smaller than the second threshold, the first threshold is related to the missed detection rate, and the second threshold is related to the false alarm rate.

[0046] In a possible implementation, the missed detection rate = the number of illegal texts misidentified as normal texts by the domain model / the total number of texts; the false alarm rate = the number of normal texts misidentified as illegal texts by the domain model / the total number of texts.

[0047] In a possible implementation, the text whose total score of the classification result is less than the first threshold is determined as normal text; the text whose total score of the classification result is greater than the second threshold is determined as abnormal text (or called non-normal text). The total score is greater than or equal to 0 and less than or equal to 1. For each sample to be classified, the total scores of the classification results can be arranged in order from small to large, and compared with the first threshold one by one, and the text whose total score of the classification result is less than the first threshold is determined as normal text, and the text whose total score of the classification result is greater than the second threshold is determined as abnormal text.

[0048] Furthermore, the texts whose total scores of the classification results are between the first threshold and the second threshold are submitted to manual review. In this way, manual review can be combined with machine review, thereby improving the review speed while ensuring the review accuracy.

[0049] It can be seen from the above steps 101 to 104 that by fusing the scores of N domain models, the total score of the final classification result obtained has high robustness and high classification accuracy.

[0050] In the embodiment of the present invention, the classification method based on multi-model fusion mainly includes two processes: model training and model use. Figure 1 This mainly introduces the model usage process, and the model training is introduced below.

[0051] like Figure 2 As shown, a multi-model training method provided by an embodiment of the present invention. The method comprises the following steps:

[0052] Step 201: collect training data sets.

[0053] The training data set can also be called a sample data set, which includes positive samples and negative samples. Since some negative samples are pornographic or political, it is impossible to download labeled data directly from the Internet. Therefore, it is necessary to collect and label data from scratch. Based on this, collecting training data mainly includes the following two processes.

[0054] Process 1: Collect sample data.

[0055] In a possible implementation, sample data can be collected from multiple channels. For example, it can include but is not limited to: downloading some public sensitive word libraries from the Internet; developing crawler software to crawl some training data from the Internet, mainly as normal samples in the training data set; purchasing data from major data trading centers; sharing data with friendly companies; and some online manually reviewed data provided by customers.

[0056] The data collected from multiple channels can cover multiple fields, such as but not limited to the cultural field, education field, entertainment field, medical field, financial field, news field, etc. Therefore, when the text to be classified is subsequently classified, personal privacy information, advertising information, politically sensitive information, pornographic information, gambling information, dangerous chemical information, violent and bloody information, terrorist information and other illegal text information can be successfully identified.

[0057] Process 2: labeling sample data, or sample data labeling.

[0058] In order to improve the efficiency and accuracy of manual review, the present invention adopts the method of machine preprocessing and manual secondary review. The machine first uses keywords to filter the easily recognizable text and first identifies a part of the positive and negative sample data in the text. The remaining data is then submitted to manual review. The data after manual review will be sampled again for a second manual cross-review, and the accuracy of the manual review will be judged based on the results of the second review.

[0059] It is understandable that the collected training data set can determine the upper limit of the accuracy of the trained domain model.

[0060] Step 202: train N domain models, where N is an integer greater than 1.

[0061] Among them, one domain corresponds to one domain model, and one domain model is used to classify text in one domain. For example, domain model 1 is used to classify pornographic texts, model domain 2 is used to classify politically sensitive texts, domain model 3 is used to classify advertising texts, and so on. The so-called domain model is a visual representation of conceptual classes within a domain or objects in the real world. It is also called a conceptual model, a domain object model, or an analysis object model. It focuses on analyzing the problem domain itself, discovering important business domain concepts, and establishing relationships between business domain concepts.

[0062] In a possible implementation, TextCNN can be used to train the domain model using the training data collected in step 201 as samples.

[0063] Step 203: training a Conditional Random Field (CRF) labeling sequence based on N domain models.

[0064] In a possible implementation, the CRF annotation sequence can be defined by formulating. Specifically, N domain models can be used as labels and the domain model feature layer can be used as an input sequence to train the CRF annotation sequence.

[0065] Through the above steps 201 to 203, N domain models and CRF annotation sequences can be trained. Furthermore, a single model can be responsible for the classification task of a single domain, and CRF is responsible for annotating the probability of each task corresponding to each domain model, which helps to improve the accuracy of classification. Moreover, the domain model obtained based on this training helps to improve the scope of classification coverage, and can review more than 95% of the text types in the text field, such as traditional Chinese characters, non-Chinese, certificate information, Buddhist scriptures, medical classics, ancient poems, etc.

[0066] In order to evaluate the effectiveness of the model, a good evaluation method is needed, and an evaluation standard to measure the generalization ability of the model is also needed. Evaluation indicators are a very important part of machine learning tasks. Different machine learning tasks have different evaluation indicators, and the same machine learning task also has different evaluation indicators, and each indicator has a different focus.

[0067] In the embodiment of the present invention, the evaluation of the domain model uses the traditional F1-score, which is defined based on the harmonic mean of precision and recall, and its value is closer to the smaller value of precision (P) and recall (R). When both precision and accuracy are high, the F1 value will also be high. The precision (P) and recall (R) indicators sometimes conflict with each other, so they need to be considered comprehensively. The F1-score value reaches its best value of 1 and its worst value is 0.

[0068] Furthermore, in the embodiment of the present invention, two new metrics are added to the evaluation of the domain model, which are described in detail below.

[0069] Indicator 1: missed detection rate; where missed detection rate = the number of illegal texts that are mistakenly identified as normal texts by the domain model / the total number of texts.

[0070] Indicator 2: false alarm rate; false alarm rate = data where the model mistakenly identifies normal text as illegal text / total number of texts.

[0071] In a possible implementation, a first threshold and a second threshold may be determined based on the two metrics, wherein the first threshold and the second threshold are greater than or equal to 0 and less than or equal to 1. Specifically, the missed detection rate is calculated using the true label, and when the missed detection rate reaches one in ten thousand for the first time, the score corresponding to the text is determined to be threshold 1. The false alarm rate is calculated using the true label, and when the false alarm rate reaches one in one thousand for the first time, the score corresponding to the text is determined to be threshold 2.

[0072] Based on the two newly added metrics mentioned above, the missed detection rate of text can be less than one in ten thousand, the false detection rate can be less than one in a thousand, and a response can be achieved in seconds.

[0073] In order to further understand the classification method based on multi-model fusion proposed in the embodiment of the present invention, the following is an example of file review, specifically reviewing whether the files uploaded and shared by users violate regulations. The method includes the following steps:

[0074] Step 301: receiving a file to be reviewed that is uploaded by a user.

[0075] This step 301 can be understood as a service entrance, wherein the file that the user requests to upload is the file to be reviewed.

[0076] Step 302: load balancing the files to be reviewed according to the load of the parsing server.

[0077] In a possible implementation, multiple text parsing servers may be pre-deployed. It should be noted that the specific number of deployed text parsing servers may be adjusted according to actual needs (eg, actual page views).

[0078] Step 303: parse the file to be reviewed to obtain the text content.

[0079] Specifically, text content may be extracted from the file to be reviewed, and the text content is the text to be reviewed.

[0080] Step 304: perform regular matching on the text and the keyword vocabulary to filter out some normal text or abnormal text.

[0081] In one possible implementation, for example, a blacklist and / or a whitelist is set, and some abnormal text is filtered out by regular matching with keywords in the blacklist, and some normal text is filtered out by regular matching with keywords in the whitelist, and some normal text or abnormal text is directly returned.

[0082] Step 305: input the text into the corresponding domain model according to the length of the text, and determine the weight of each model.

[0083] In this example, it is taken that there are three domain models, that is, N=3.

[0084] In one possible implementation, if the text is determined to be a long text, the naive Bayes model is used to review the long text; if the text is determined to be a short text, the deep learning model is used to review the short text. Among them, the text whose length is greater than the third threshold can be set as a long text, and the text whose length is less than or equal to the third threshold can be set as a short text, wherein the third threshold can be, for example, 400 words. It should be noted that the specific word count limit for long texts and short texts can be adjusted according to the model accuracy. Furthermore, the naive Bayes model mainly relies on the computing resources of the CPU, and the deep learning model mainly relies on the computing resources of the GPU. For example, NVIDIA GPU V100 can be used, and a single card can process 40 texts per second.

[0085] Step 306: Determine the total score of the classification result according to the N scores output by the N classification models and the weights of the N domain models.

[0086] Exemplarily, the total score of the classification result = (N1×Q1+N2×Q2+N3×Q3) / 3, where N1 is the score output by domain model 1, N2 is the score output by domain model 2, N3 is the score output by domain model 3, Q1 is the weight of domain model 1, Q2 is the weight of domain model 2, and Q3 is the weight of domain model 3.

[0087] Step 307: classify the text according to the total score of the classification result, the relationship between the first threshold and the second threshold.

[0088] The text may be classified as normal text or abnormal text. The specific process may be referred to the introduction of the above step 204, which will not be repeated here.

[0089] Step 308: encapsulate the classification result and convert it into a binary classification result.

[0090] Step 309, returning the binary classification result to the external interface.

[0091] The external interface is used to provide classification results to the user equipment.

[0092] Through the above steps 301 to 309, personal privacy information, advertising information, politically sensitive information, pornographic information, gambling information, dangerous chemical information, violent and bloody information, terrorist information and other illegal text information can be successfully identified. The number of correctly opened texts accounts for more than 85% of the total number of texts, the missed detection rate is less than one in ten thousand, and the false alarm rate is less than one in a thousand, saving more than 10% of labor costs. In addition, there are few restrictions on the length of text, supporting a single article length of millions and responding in seconds. Furthermore, by integrating the scores of N domain models, the total score of the final classification result is obtained, which helps to improve the accuracy of classification.

[0093] It is understandable that, in order to implement the functions in the above-mentioned embodiments, the classification device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the modules and method steps of each example described in the embodiments disclosed in this application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0094] Based on the same technical concept as above, the embodiment of the present invention also provides a classification device. Figure 4 , is a schematic diagram of the structure of a classification device provided in an embodiment of the present invention. The classification device can be used to execute the method in the above method embodiment, and thus can also achieve the beneficial effects possessed by the above method embodiment.

[0095] like Figure 4 As shown, the classification device 400 includes an input unit 401, a determination unit 402 and a classification unit 403. The classification device 400 can be used to perform Figure 1 , Figure 2 or Figure 3 The method shown.

[0096] When the classification device 400 is used to implement the above Figure 1 In the method embodiment shown, the input unit 401 is used to input the text to be classified into N different domain models to obtain N scores, where N is an integer greater than 1; the determination unit 402 is used to determine the weights of the N domain models; and determine the total score of the classification result based on the N scores and the weights of the N domain models; the classification unit 403 is used to classify the text to be classified based on the relationship between the total score of the classification result, the first threshold and the second threshold, wherein the first threshold is less than the second threshold, the first threshold is related to the missed detection rate, and the second threshold is related to the false alarm rate.

[0097] In a possible implementation, the classification device further includes a matching unit, which is used to perform regular matching on the text to be classified with a keyword vocabulary to filter out some normal text or abnormal text.

[0098] In a possible implementation, the determination unit 402 is specifically configured to determine the target domain to which the to-be-classified text belongs through a conditional random field (CRF) annotation sequence; and set the weight of the domain model corresponding to the target domain to be greater than the weights of the remaining N-1 domain models.

[0099] In one possible implementation, the N domain models include a naive Bayes model and a deep model; the input unit 401 is specifically used to input the text to be classified into the naive Bayes model if the length of the text to be classified is not greater than a third threshold; if the length of the text to be classified is greater than the third threshold, input the text to be classified into the deep learning model.

[0100] In a possible implementation, the missed detection rate = the number of illegal texts misidentified as normal texts by the domain model / the total number of texts; the false alarm rate = the number of normal texts misidentified as illegal texts by the domain model / the total number of texts.

[0101] In a possible implementation, the classification unit 403 is specifically configured to: determine text whose total score of the classification result is less than the first threshold as normal text; and determine text whose total score of the classification result is greater than the second threshold as abnormal text.

[0102] In a possible implementation, the detection unit may further include a collection unit for collecting sample data through different channels and labeling the sample data; wherein the data includes at least one of the following: personal privacy information; advertising information; politically sensitive information; pornographic information; gambling information; hazardous chemical information; violent and bloody information; and terrorist information.

[0103] For a more detailed description of the input unit 401, the determination unit 402 and the classification unit 403, please refer to Figure 1 The relevant descriptions in the method embodiment shown are directly obtained and will not be repeated here.

[0104] Based on the same technical concept, the embodiment of the present application also provides a classification device, such as Figure 5 As shown, the classification device 500 includes at least one processor 501 and a communication interface 502, and a memory 503 connected to the at least one processor. The specific connection medium between the processor 501 and the memory 503 is not limited in the embodiment of the present application. Figure 5 For example, the processor 501 and the memory 503 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0105] In an embodiment of the present application, the memory 503 stores instructions that can be executed by at least one processor 501. By executing the instructions stored in the memory 503, at least one processor 501 can execute the steps included in the aforementioned classification method based on multi-model fusion.

[0106] Among them, the processor 501 is the control center of the classification device 500, and can use various interfaces and lines to connect various parts of the classification device, and realize data processing by running or executing instructions stored in the memory 503 and calling data stored in the memory 503. Optionally, the processor 501 may include one or more processing units, and the processor 501 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application program, etc., and the modem processor mainly processes the issued instructions. It is understandable that the above-mentioned modem processor may not be integrated into the processor 501. In some embodiments, the processor 501 and the memory 503 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on independent chips.

[0107] The processor 501 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiment of the classification method based on multi-model fusion can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in the processor for execution.

[0108] The memory 503 is a non-volatile computer-readable storage medium, which can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 503 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 503 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 503 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0109] Based on the same technical concept, an embodiment of the present application also provides a computer-readable storage medium, which stores a computer program that can be executed by a classification device. When the program runs on the classification device, the classification device executes the steps of the above-mentioned classification method based on multi-model fusion.

[0110] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0111] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0112] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0113] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0114] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A classification method based on multi-model fusion, characterized in that: include: Input the text to be classified into N different domain models to obtain N scores, where N is an integer greater than 1; Using the N domain models as labels and the feature layers of the N domain models as input sequences, training a conditional random field (CRF) annotation sequence, and determining the target domain to which the to-be-classified text belongs through the conditional random field (CRF) annotation sequence; setting the weight of the domain model corresponding to the target domain to be greater than the weights of the remaining N-1 domain models; Determining a total score of the classification result according to the N scores and the weights of the N domain models; The text to be classified is classified according to the relationship between the total score of the classification result, the first threshold and the second threshold, wherein the first threshold is smaller than the second threshold, the first threshold is related to the missed detection rate, and the second threshold is related to the false alarm rate.

2. The method according to claim 1, characterized in that Before inputting the to-be-classified text into N different domain models to obtain N scores, the method further includes: Regular matching is performed between the text to be classified and the keyword vocabulary to filter out some normal text or abnormal text.

3. The method according to claim 1, characterized in that The N domain models include a naive Bayes model and a deep model; The text to be classified is input into N different domain models to obtain N scores, including: If the length of the text to be classified is not greater than a third threshold, inputting the text to be classified into a naive Bayes model; If the length of the text to be classified is greater than the third threshold, the text to be classified is input into the deep learning model.

4. The method according to claim 1, characterized in that The missed detection rate = the number of illegal texts misjudged as normal texts by the domain model / the total number of texts; the false alarm rate = the number of normal texts misjudged as illegal texts by the domain model / the total number of texts.

5. The method according to claim 4, characterized in that The classifying the text to be classified according to the relationship between the total score of the classification result, the first threshold and the second threshold includes: Determine the text whose total score of the classification result is less than the first threshold as normal text; The text whose total score of the classification result is greater than the second threshold is determined as abnormal text.

6. The method according to claim 1, characterized in that Before inputting the text to be classified into N different domain models, the method further includes: Collect sample data through different channels and annotate the sample data; The data includes at least one of the following: Personal privacy information; Advertising information; Politically sensitive information; Pornographic information; Gambling information; hazardous chemicals information; Violent and bloody information; Terrorism information.

7. A classification device, characterized in that: include: An input unit, used to input the text to be classified into N different domain models to obtain N scores, where N is an integer greater than 1; A determination unit is used to use the N domain models as labels and the feature layers of the N domain models as input sequences, train a conditional random field (CRF) annotation sequence, and determine the target domain to which the to-be-classified text belongs through the conditional random field (CRF) annotation sequence; and set the weight of the domain model corresponding to the target domain to be greater than the weights of the remaining N-1 domain models; and determining a total score of the classification result according to the N scores and the weights of the N domain models; The classification unit is used to classify the text to be classified according to the total score of the classification result, the relationship between the first threshold and the second threshold, wherein the first threshold is smaller than the second threshold, the first threshold is related to the missed detection rate, and the second threshold is related to the false alarm rate.

8. A classification device, characterized in that: It comprises a processor and a transceiver, wherein the processor is connected to a memory, the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the classification device executes the method as claimed in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program or instruction. When the computer program or instruction is executed by the classification device, the classification device executes the method according to any one of claims 1 to 6.

Images