News chapter text classification method and device, computer device and storage medium

Abstract

The application relates to the technical field of natural language processing text classification, and provides a news chapter text classification method and device, computer equipment and a storage medium, the method comprises the following steps: preprocessing original news chapter text to obtain training news chapter text; performing feature representation and word feature construction on the training news chapter text to obtain a feature vector; based on a linear programming constraint condition, optimizing a word transformation matrix according to the feature vector and inter-word Mahalanobis distance, and constructing a Mahalanobis word transformation distance function; training a classifier according to the Mahalanobis word transformation distance function, predicting the classification label of to-be-classified news chapter text by using the trained classifier, and determining the category of the to-be-classified news chapter text. The method can improve the accuracy of news chapter text classification.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of natural language processing text classification, and particularly relates to a label chapter text classification method and device, computer equipment and a storage medium. BACKGROUND

[0002] Text classification is a process of dividing natural language text corpus into given categories according to predetermined intentions or standards. Label chapter text usually has two structural forms. One is semi-structured text containing structural information and natural language text information. For example, chapter text mainly has two structural sections, i.e., a chapter structure section and a chapter body structure section. The title is mainly expressed in the form of a phrase and the content is concise and important. The body is mainly expressed in the form of a narrative complete sentence, but the content is complex. The other is unstructured text having only one field such as a title or a body.

[0003] The existing text classification method can be roughly divided into two methods from the perspective of technical development, i.e., a traditional machine learning-based method and a modern machine learning-based method. The traditional machine learning-based method includes a traditional support vector machine, a decision tree, a random forest, a Bayesian, a naive k-nearest neighbor, etc. Generally, this method first performs feature modeling on the input sample through a feature engineering process, manually constructs a feature representation vector of the sample, and then classifies the manually constructed feature vector sample by using a traditional machine learning classifier. The modern machine learning-based method is mainly a feature self-learning related method. The method first performs feature self-learning (supervised or unsupervised) on the input sample, and then classifies the input sample by using a classifier. Most supervised feature learning methods are based on neural networks. In this case, a neural network classifier is generally connected to the neural network to form a neural network-based text classifier.

[0004] However, the modern machine learning-based method has a huge number of parameters and often requires a large amount of labeled data. It is difficult to use on a small-scale data set, which reduces the classification effect. The feature engineering method based on manual construction generally regards different words as unrelated individuals and lacks semantic understanding of words. At the same time, the method needs to manually construct features and usually uses Euclidean distance to measure the distance. However, the classifier is generally sensitive to the constructed features, which reduces the stability and generalization of the classifier. In addition, the Euclidean distance is sensitive to the dimension, and the distance is biased on the multi-dimensional features with inconsistent dimensions, which reduces the performance and the accuracy of the classification. SUMMARY

[0005] Therefore, it is necessary to provide a label chapter text classification method, device, computer equipment and storage medium capable of improving the accuracy of text classification.

[0006] The application provides a label chapter text classification method, which comprises the following steps:

[0007] preprocessing original news chapter text to obtain training news chapter text;

[0008] performing feature representation and word feature construction on the training news chapter text to obtain a feature vector;

[0009] optimizing a word transformation matrix based on a linear programming constraint condition and according to the feature vector and a Mahalanobis distance between words to construct a Mahalanobis word transformation distance function;

[0010] training a classifier according to the Mahalanobis word transformation distance function, predicting a classification label of to-be-classified news chapter text by using the trained classifier, and determining a category of the to-be-classified news chapter text.

[0011] In one of the embodiments, the feature vector includes a word feature vector, a word feature covariance matrix, and a text vector; and the performing feature representation and word feature construction on the training news chapter text to obtain a feature vector includes:

[0012] performing feature processing on the training news chapter text to obtain a word feature vector and a word feature covariance matrix;

[0013] flattening an embedded structure of the training news chapter text into a one-level structure to obtain a news chapter text in a data format of a binary tuple;

[0014] calculating a semi-structured-term frequency matrix and a semi-structured-inverse document frequency matrix of the news chapter text;

[0015] determining a semi-structured-term frequency-inverse document frequency matrix according to the semi-structured-term frequency matrix and the semi-structured-inverse document frequency matrix and drawing the semi-structured-term frequency-inverse document frequency matrix into a vector to obtain a text vector.

[0016] In one of the embodiments, the calculating a semi-structured-term frequency matrix and a semi-structured-inverse document frequency matrix of the news chapter text includes:

[0017] constructing a null semi-structured-term frequency matrix and a null semi-structured-inverse document frequency matrix with the number of words in the news chapter text as the number of rows and the number of structural segments as the number of columns;

[0018] calculating the term frequency of each word in each structural segment text based on the number of occurrences of the word in the structural segment text, and writing the term frequency as a matrix element into the null semi-structured-term frequency matrix to obtain a semi-structured-term frequency matrix;

[0019] calculating the inverse document frequency of each word in each structural segment text based on the size of the dictionary and the number of structural segments, and writing the inverse document frequency as a matrix element into the null semi-structured-inverse document frequency matrix to obtain a semi-structured-inverse document frequency matrix.

[0020] In one embodiment, the semi-structured term frequency matrix and the semi-structured inverse document frequency matrix are used to determine a semi-structured term frequency-inverse document frequency matrix and to form a vector, resulting in a text vector, including:

[0021] The semi-structured term frequency matrix and the semi-structured inverse document frequency matrix are used to calculate a semi-structured term frequency-inverse document frequency matrix and to form a one-dimensional vector, resulting in a text vector, according to the following formula:

[0022]

[0023] where x is the semi-structured term frequency-inverse document frequency matrix corresponding to the text vector, Q is the semi-structured term frequency matrix, P is the semi-structured inverse document frequency matrix, W Q and W P are structure information transfer sharing parameter matrices with the same dimensions as Q and P, ☉ is an element-wise product, 1 |F| is a vector with all elements equal to 1 with a dimension of |F|, |F| is the number of structure segments, and ||·||1 is the matrix 1-norm.

[0024] In one embodiment, before calculating the semi-structured term frequency matrix and the semi-structured inverse document frequency matrix of the signature chapter text, the method further includes removing entity words in the signature chapter text.

[0025] In one embodiment, the word transformation matrix is optimized based on the linear programming constraint condition and the Mahalanobis distance between words to construct a Mahalanobis word transformation distance function, including:

[0026] Obtaining a predetermined linear programming constraint condition;

[0027] Under the constraint of the linear programming constraint condition, the word transformation matrix is optimized using the feature vector and the Mahalanobis distance between words to construct a Mahalanobis word transformation distance function, according to the following formula:

[0028]

[0029] where MWTD(x, y) is the Mahalanobis word transformation distance between text vector x and text vector y, T is the word transformation matrix, and the matrix element T ij is the word transformation weight of the i-th word in text vector x to the j-th word in text vector y, ω i is the word feature vector of the i-th word in text vector x, ω j is the word feature vector of the j-th word in text vector y, B is the word feature covariance matrix, is the Mahalanobis distance between the i-th word in text vector x and the j-th word in text vector y.

[0030] In one embodiment, the linear programming constraint condition comprises: the word variable matrix is a non-negative matrix, a sum of vector elements of an i-th row vector of the word variable matrix is equal to a value of an i-th element of the text vector x, and a sum of vector elements of a j-th column vector of the word variable matrix is equal to a value of a j-th element of the text vector y.

[0031] A news chapter text classification device, comprising:

[0032] A preprocessing module, configured to preprocess original news chapter texts to obtain training news chapter texts;

[0033] A feature processing module, configured to perform feature representation and word feature construction on the training news chapter texts to obtain feature vectors;

[0034] A construction module, configured to optimize a word variable matrix based on a linear programming constraint condition and according to the feature vectors and inter-word Mahalanobis distances, and to construct a Mahalanobis word variable distance function;

[0035] A classification module, configured to train a classifier according to the Mahalanobis word variable distance function, to predict a classification label of a to-be-classified news chapter text by using the trained classifier, and to determine a category of the to-be-classified news chapter text.

[0036] The application further provides a computer device, comprising a processor and a memory, wherein the memory stores a computer program, and the processor implements the steps of the news chapter text classification method according to any one of the above embodiments when executing the computer program.

[0037] The application further provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the news chapter text classification method according to any one of the above embodiments when executed by a processor.

[0038] The above text classification method, device, computer equipment and storage medium for news chapter text, after preprocessing the original news chapter text to obtain the training news chapter text, performing feature representation and word feature construction on the training news chapter text to obtain a feature vector, then optimizing a word transformation matrix based on a linear programming constraint condition according to the feature vector and the Mahalanobis distance between words, constructing a Mahalanobis word transformation distance function, further training a classifier according to the Mahalanobis word transformation distance function, and determining the category by using the trained classifier to predict the classification label of the to-be-classified news chapter text. The method trains the classifier by constructing a Mahalanobis word transformation distance calculation model by combining the Mahalanobis distance and the word transformation linear programming method, realizes text classification, solves the problem of inconsistent dimension of traditional multi-feature fusion distance by using the dimensionless characteristics of the Mahalanobis distance, can capture the semantic similarity of words, does not need a large amount of labeled data, makes the model parameter small and easy to train and use, and further improves the model interpretability based on the word transformation matrix solution of the linear programming matrix, thereby greatly improving the accuracy of text classification. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 An application environment diagram of the news chapter text classification method in an embodiment.

[0040] Figure 2 A flowchart of the news chapter text classification method in an embodiment.

[0041] Figure 3 A flowchart of the step of performing feature representation and word feature construction on the training news chapter text to obtain a feature vector in an embodiment.

[0042] Figure 4 A structural block diagram of the news chapter text classification device in an embodiment. DETAILED DESCRIPTION

[0043] In order to make the objects, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0044] The news chapter text classification method provided in the present application can be applied to, for example Figure 1In the application environment shown, the application environment involves a terminal 102 and a server 104. The terminal 102 communicates with the server 104 through a network. The terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices, and the server 104 can be implemented by a stand-alone server or a server cluster composed of multiple servers. The above-mentioned text classification method of the section can be implemented by the terminal 102 alone, or can be implemented by the server 104. Taking the server 104 as an example, specifically, the server 104 preprocesses the original text of the section, to obtain training text of the section; the server 104 performs feature representation and word feature construction on the training text of the section, to obtain a feature vector; the server 104 optimizes a word transformation matrix based on a linear programming constraint condition, according to the feature vector and the Mahalanobis distance between words, to construct a Mahalanobis word transformation distance function; the server 104 trains a classifier according to the Mahalanobis word transformation distance function, and predicts a classification label of a text of a section to be classified by using the trained classifier, to determine the category of the text of the section to be classified.

[0045] In one embodiment, as shown in Figure 2 A text classification method of a section is provided, which is applied to a server as an example for illustration, and includes the following steps:

[0046] In step S201, the original text of the section is preprocessed to obtain training text of the section.

[0047] The text of the section refers to text including information of a section of bidding and tender, which usually includes two structural forms. One is semi-structured text including structural information and natural language text information, such as text of the section mainly including two structural sections of a section structure section and a section body structure section, the title is mainly expressed in the form of a phrase and the content is concise and important, and the body is mainly expressed in the form of a narrative complete sentence but the content is complicated. The other is unstructured text having only one field such as a title or a body. Therefore, the text classification method of the embodiment can be applied to text belonging to the two structural forms, and is not limited to text of the section of bidding and tender. The original text of the section of bidding and tender is original text data of the section of bidding and tender without preprocessing, and the training text of the section of bidding and tender is text of the section of bidding and tender after preprocessing. Since the text is used for training classification subsequently, it is called training text of the section of bidding and tender.

[0048] Specifically, the preprocessing of the embodiment mainly includes conventional text preprocessing, dirty data cleaning, complex-simplified conversion, stop word removal, word segmentation, and word type reduction. That is, when the server obtains the original chapter text, the word sequence of the text is output, and the process does not change the structure information of the text. The dirty data cleaning is realized by using preset cleaning rules, a word table, or a simple algorithm. The cleaning targets in the embodiment include but are not limited to HTML tags, garbled characters, and character expressions irrelevant to the actual required business. The complex-simplified conversion is an operation of converting traditional Chinese characters in the text into simplified Chinese characters. The operation is determined according to actual different business purposes and needs. The stop word removal is a process of removing stop words in the text by using a given stop word table. The operation can remove part of the noise. Similarly, whether the operation needs to be performed can be determined according to actual business purposes and needs. Specifically, the text representation based on the traditional method needs to remove stop words. The word segmentation is a process of segmenting a text sentence. A text is segmented into a word sequence. Chinese text can be analyzed by using semantic analysis technology. The word segmentation can improve the information understanding granularity and be beneficial to improving the effect. The word type reduction mainly normalizes different forms to prototypes.

[0049] In step S202, feature representation and word feature construction are performed on the training chapter text to obtain a feature vector.

[0050] The feature representation refers to text representation, that is, representing the text as a feature vector. The word feature construction refers to a process of outputting a word feature vector and a word feature covariance matrix after a certain feature processing on the preprocessed text.

[0051] Specifically, after the text is preprocessed to obtain the training chapter text, the training chapter text is represented as a feature vector. Specifically, a text feature representation model can be used to represent the text as a vector. Meanwhile, the preprocessed training chapter text is processed by features, such as feature engineering, feature learning, or feature fusion, to output a word feature vector and a word feature covariance matrix.

[0052] In step S203, based on a linear programming constraint condition, a word transformation matrix is optimized according to the feature vector and the Mahalanobis distance between words to construct a Mahalanobis word transformation distance function.

[0053] The linear programming is a method for researching an extreme value problem of a linear objective function under a linear constraint condition. In this embodiment, the word transition matrix is an optimization variable of the linear programming, and the Mahalanobis word transition distance function constructed based on the optimized word transition matrix is the final linear objective function. The inter-word Mahalanobis distance refers to a Mahalanobis distance between two words. The traditional Mahalanobis distance is a dimensionless distance calculation model, and the Mahalanobis word transition distance function (MWTD) in this embodiment is a text distance calculation model combining the Mahalanobis distance and the linear programming.

[0054] Specifically, after obtaining the feature vectors of the training news chapter texts, the Mahalanobis distance between the words in different training news chapter texts is first calculated. Under the linear programming constraint condition, the Mahalanobis distance and the determined feature vector are used to optimize the unknown and optimized variable word transition matrix. When the optimal word transition matrix is obtained, the Mahalanobis word transition distance function is constructed with the word transition matrix as a fixed parameter.

[0055] In step S204, the classifier is trained according to the Mahalanobis word transition distance function, the trained classifier is used to predict the classification label of the to-be-classified news chapter text, and the category of the to-be-classified news chapter text is determined.

[0056] Specifically, after obtaining the Mahalanobis word transition distance function, the training news chapter text and the Mahalanobis word transition distance function are input into the classifier for training. The classifier can be selected based on the neighbor analysis or the distance-based classifier according to actual needs. Different selection of classification methods has different processes. Taking the K-nearest neighbor classifier as an example, the process is roughly as follows: first, the Mahalanobis word transition distance result between different training news chapter texts is calculated according to the Mahalanobis word transition distance function, and then the Mahalanobis word transition distance result and the corresponding label information are used to train the classifier, so as to obtain the trained classifier. Subsequently, when the text classification is performed, the to-be-classified news chapter text is obtained, the to-be-classified news chapter text is directly input into the classifier, the distance between the to-be-classified news chapter text and the standard sample is calculated, the classification label of the to-be-classified news chapter text is predicted based on the distance, and thus the category of the to-be-classified news chapter text is obtained.

[0057] The above news chapter text classification method, after preprocessing the original news chapter text to obtain the training news chapter text, performing feature representation and word feature construction on the training news chapter text to obtain a feature vector, then optimizing a word variable matrix based on a linear programming constraint condition according to the feature vector and the Mahalanobis distance between words, constructing a Mahalanobis word variable distance function, further training a classifier according to the Mahalanobis word variable distance function, and determining the category by using the trained classifier to predict the classification label of the to-be-classified news chapter text. The method combines the Mahalanobis distance and the word variable linear programming method to construct a Mahalanobis word variable distance calculation model to train the classifier to realize text classification, solves the problem of dimension inconsistency of traditional multi-feature fusion distance bias by using the dimensionless characteristics of the Mahalanobis distance, can capture the semantic similarity of words, does not require a large amount of labeled data, makes the model parameter small and easy to train and use, and further improves the model interpretability based on the word variable matrix solution of the linear programming matrix, thereby greatly improving the accuracy of text classification.

[0058] In one embodiment, the feature vector includes a word feature vector, a word feature covariance matrix, and a text vector, as shown in Figure 3 As shown in FIG. 2, step S202 includes the following steps:

[0059] Step S301, performing feature processing on the training news chapter text to obtain a word feature vector and a word feature covariance matrix.

[0060] Specifically, word feature construction is a process of inputting preprocessed text, performing feature engineering, feature learning or feature fusion, and then outputting a word feature vector and a feature covariance matrix. The output vector can come from feature engineering only or feature learning only, or from feature fusion. Feature engineering is a feature constructed by manual analysis, feature learning is a word feature vector learned by a self-learning technology such as word2vec, and feature fusion is a fusion of feature vectors output by feature engineering and feature vectors output by feature learning. The fusion can be a vector operation such as vector splicing. The covariance matrix of features from feature learning only can be given or estimated by a self-learning model, and the covariance matrix of features containing feature vectors of feature engineering can be calculated or estimated according to feature semantics.

[0061] Step S302, flattening the embedded structure of the training news chapter text into a one-level structure to obtain a news chapter text in a binary tuple data format.

[0062] Specifically, when a text feature representation model represents a text as a vector to obtain a text vector, the structure of the preprocessed training news chapter text data is first flattened, mainly the embedded structure is flattened into a one-level structure, and the one-level structure does not need to be processed. At this time, the data format of the news chapter text is equivalent to a binary tuple <structure segment name:natural language text word sequence>.

[0063] Step S303, calculate the semi-structured term frequency matrix and the semi-structured inverse document frequency matrix of the chapter text of the label information.

[0064] In the embodiment, the semi-structured term frequency matrix is determined based on the conventional TF (term frequency) idea, and the semi-structured inverse document frequency matrix is determined based on the conventional IDF (inverse document frequency) idea, which are referred to as SA (structure-aware)-TF and SA-IDF respectively. TF is a characteristic quantity for representing the weight of a word in a document, and IDF represents the distinguishing degree of a word to a document.

[0065] Specifically, after the structure of the training chapter text of the label information is flattened, the semi-structured term frequency matrix (SA-TF) and the semi-structured inverse document frequency matrix (SA-IDF) are calculated based on the TF idea and the IDF idea respectively by using the flattened chapter text of the label information. The conventional TF is obtained by dividing the number of occurrences of a word in a document by the total number of occurrences of all words in the document, while the SA-TF in the embodiment is determined based on the number of occurrences of a word in a structural segment. Similarly, the difference between the SA-IDF in the embodiment and the conventional IDF is that the SA-IDF in the embodiment is determined based on the number of structural segments.

[0066] Step S304, determine the semi-structured term frequency-inverse document frequency matrix based on the semi-structured term frequency matrix and the semi-structured inverse document frequency matrix, and pull it into a vector to obtain a text vector.

[0067] Specifically, after the SA-TF matrix and the SA-IDF matrix are determined, the SA-TF-IDF matrix can be determined based on the SA-TF matrix and the SA-IDF matrix.

[0068] In one embodiment, step S304 includes: calculating the semi-structured term frequency-inverse document frequency matrix based on the semi-structured term frequency matrix and the semi-structured inverse document frequency matrix; and pulling the semi-structured term frequency-inverse document frequency matrix into a one-dimensional vector to obtain a text vector.

[0069] Specifically, unlike the conventional TF-IDF obtained by multiplying TF and IDF, the calculation formula of the SA-TF-IDF matrix corresponding to the text vector in the embodiment is as follows:

[0070]

[0071] wherein x is the text vector corresponding to the SA-TF-IDF matrix, Q is the semi-structured term frequency matrix, P is the semi-structured inverse document frequency matrix, and W Q and W Pis a structural information transfer sharing parameter matrix with the same dimension as Q and P, is an element-wise product, 1 |F| is a vector with all elements being 1 and the dimension |F|, |F| is the number of structural segments, ||·||1 is the 1-norm of a matrix, the so-called 1-norm of a matrix A is defined as follows:

[0072]

[0073] where A ij denotes the matrix element of the ith row and jth column of matrix A. In this embodiment, based on the TF-IDF idea, the word TF-IDF information and the structural information of the chapter text are fused to determine the SA-TF-IDF, which improves the representation effect of the text words and integrates the structural features, thereby improving the accuracy of classification.

[0074] In one embodiment, step S303 comprises: taking the number of words in the chapter text as the number of rows and the number of structural segments as the number of columns to construct the null semi-structural-term frequency matrix and the null semi-structural-inverse document frequency matrix; based on the number of occurrences of each word in the text of each structural segment, the term frequency of each word in the text of each structural segment is calculated, and the term frequency is written into the null semi-structural-term frequency matrix as a matrix element to obtain the semi-structural-term frequency matrix; based on the size of the dictionary and the number of structural segments, the inverse document frequency of each word in the text of each structural segment is calculated, and the inverse document frequency is written into the null semi-structural-inverse document frequency matrix as a matrix element to obtain the semi-structural-inverse document frequency matrix.

[0075] Specifically, when determining the SA-TF matrix and the SA-IDF matrix, first, the null SA-TF matrix and the null SA-IDF matrix with null matrix elements are constructed, and the matrix size depends on the number of words in the chapter text and the number of structural segments. The number of rows and columns of the constructed matrix is determined by taking the number of words as the number of rows and the number of structural segments as the number of columns. Then, the matrix element in the null SA-TF matrix is the term frequency TF of each word in the text in the structural segment, and the matrix element in the null SA-IDF matrix is the inverse document frequency IDF determined based on the size of the dictionary and the number of structural segments. That is, the matrix element [Q S ] ij in the null SA-TF matrix is the term frequency TF of the ith word in the jth structural segment, and the matrix element [P D ] ij in the null SA-IDF matrix is the inverse document frequency IDF of the ith word calculated in all text sets of the jth structural segment, and the calculation formula is as follows:

[0076]

[0077] where [Q S ]ij denotes the matrix element of the i-th row and j-th column of the SA-TF matrix, denotes the number of occurrences of the i-th word in the j-th structural segment of the text of the news chapter s, k is the total number of words, i = {1, 2, …, k}.

[0078]

[0079] where, [P D ] ij denotes the matrix element of the i-th row and j-th column of the SA-IDF matrix, D denotes a set of texts, d denotes a text in D, w i is the i-th word, f j is the j-th structural segment, |V| is the size of the dictionary, the specific value depends on the size of the dictionary actually used, |F| is the number of structural segments of the news chapter text s, for example, a semi-structured document with "title" and "body" has |F| = 2 structural segments.

[0080] Then, when all the matrix elements are calculated, the corresponding matrix elements are written in the null SA-TF matrix and the null SA-IDF matrix respectively, to obtain the SA-TF matrix and the SA-IDF matrix.

[0081] In an embodiment, before step S303, it further includes: removing entity words in the news chapter text.

[0082] Specifically, when there are entity words in the news chapter text, in order to avoid the influence of the entity words, it is necessary to further remove the entity words in the text. The entity words can be obtained by named entity recognition on the text in the preprocessing step, including names, place names, organization names, etc.

[0083] In an embodiment, step S203 includes: obtaining a preset linear programming constraint condition; under the constraint of the linear programming constraint condition, a word transformation matrix is obtained by using the feature vector and the inter-word Mahalanobis distance, and a Mahalanobis word transformation distance function is constructed.

[0084] Specifically, in the present embodiment, the MWTD distance of two text vectors x and y is given by a linear programming objective of the word transformation matrix from the words in one text vector to the words in the other text vector and the inter-word Mahalanobis distance. Based on this idea, the word transformation distance is optimized, and the optimized word transformation distance is constructed into the MWTD function, and the calculation formula is as follows:

[0085]

[0086] where, MWTD(x, y) is the Mahalanobis word transformation distance of the text vector x and the text vector y, T is the word transformation matrix, the matrix element T ijis the word variant weight of the i-th word in the text vector x to the j-th word in the text vector y, ω i is the word feature vector of the i-th word in the text vector x, ω j is the word feature vector of the j-th word in the text vector y, B is the word feature covariance matrix, is the inter-word Mahalanobis distance between the i-th word in the text vector x and the j-th word in the text vector y. The pre-set linear programming constraint conditions include three, the first is that the word variant matrix T is a non-negative matrix, that is, all matrix elements in the word variant matrix T are non-negative. The second is that for the i-th row vector in the word variant matrix T, the sum of the vector elements is equal to the value of the i-th element of the text vector x, and each row vector in the word variant matrix T needs to satisfy this condition. The third is that for the j-th column vector in the word variant matrix T, the sum of the vector elements is equal to the value of the j-th element of the text vector y, and each column vector in the word variant matrix T needs to satisfy this condition.

[0087] It should be understood that, although Figures 2-3 the steps in the flowchart of the method are displayed in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise explicitly stated herein, there is no strict order limitation for the execution of these steps, and these steps can be executed in other orders. Moreover, Figures 2-3 at least part of the steps in the method can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately executed with at least part of other steps or steps or stages in other steps.

[0088] In one embodiment, as shown in Figure 4 , a text classification device for news chapter is provided, comprising:

[0089] A preprocessing module 401 is configured to preprocess the original news chapter text to obtain a training news chapter text.

[0090] A feature processing module 402 is configured to perform feature representation and word feature construction on the training news chapter text to obtain a feature vector.

[0091] A construction module 403 is configured to optimize a word variant matrix based on a linear programming constraint condition and according to the feature vector and the inter-word Mahalanobis distance, and construct a Mahalanobis word variant distance function.

[0092] A classification module 404 is configured to train a classifier according to the Mahalanobis word variant distance function, predict a classification label of a to-be-classified news chapter text by using the trained classifier, and determine a category of the to-be-classified news chapter text.

[0093] In one embodiment, the feature processing module 402 is further configured to perform feature processing on the training news chapter text to obtain a word feature vector and a word feature covariance matrix; flatten the nested structure of the training news chapter text into a one-level structure to obtain the news chapter text in a data format of a binary tuple; calculate a semi-structure-term frequency matrix and a semi-structure-inverse document frequency matrix of the news chapter text; determine a semi-structure-term frequency-inverse document frequency matrix based on the semi-structure-term frequency matrix and the semi-structure-inverse document frequency matrix and pull the semi-structure-term frequency-inverse document frequency matrix into a vector to obtain a text vector.

[0094] In one embodiment, the feature processing module 402 is further configured to construct a null semi-structure-term frequency matrix and a null semi-structure-inverse document frequency matrix by taking the number of words in the news chapter text as the number of rows and the number of structure sections as the number of columns; calculate the term frequency of each word in each structure section based on the number of occurrences of the word in the text of the structure section, and write the term frequency as a matrix element into the null semi-structure-term frequency matrix to obtain the semi-structure-term frequency matrix; and calculate the inverse document frequency of each word in each structure section based on the size of the dictionary and the number of structure sections, and write the inverse document frequency as a matrix element into the null semi-structure-inverse document frequency matrix to obtain the semi-structure-inverse document frequency matrix.

[0095] In one embodiment, the feature processing module 402 is further configured to calculate a semi-structure-term frequency-inverse document frequency matrix based on the semi-structure-term frequency matrix and the semi-structure-inverse document frequency matrix and pull the semi-structure-term frequency-inverse document frequency matrix into a one-dimensional vector to obtain a text vector.

[0096] In one embodiment, the feature processing module 402 is further configured to remove entity words in the news chapter text.

[0097] In one embodiment, the construction module 403 is further configured to obtain a preset linear programming constraint condition; and obtain a word transformation matrix by optimizing the feature vector and the Mahalanobis distance between words under the constraint of the linear programming constraint condition, and construct a Mahalanobis word transformation distance function.

[0098] The specific limitation of the text classification device for the chapter of news can be referred to the limitation of the method for classifying the chapter of news, which will not be repeated here. Each module in the above text classification device for the chapter of news can be realized by software, hardware and combination thereof in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operation corresponding to each module. Based on such understanding, all or part of the processes in the above embodiment method can also be completed by the computer program to instruct the related hardware, and the computer program can be stored in a computer readable storage medium. When the processor executes the computer program, the steps of each text classification method for the chapter of news can be realized. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form, etc.

[0099] In one embodiment, a computer device, which can be a server, is provided, including a processor, a memory and a network interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is configured to store data. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement a method for classifying the chapter of news. Illustratively, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present application. One or more modules can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the computer device.

[0100] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor and the like, which is a control center of the computer device and connects all parts of the computer device through various interfaces and lines.

[0101] The memory can be used to store the computer program and / or modules, and the processor realizes various functions of the computer device by running or executing the computer program and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), etc.; and the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0102] Those skilled in the art can understand that the computer device structure shown in the embodiment only relates to part of the structure related to the application scheme, and does not constitute a limitation on the computer device to which the application is applied. The specific computer device can include more or fewer components, or combine certain components, or have a different component arrangement.

[0103] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor realizing the following steps when executing the computer program:

[0104] The original news chapter text is preprocessed to obtain training news chapter text;

[0105] The training news chapter text is subjected to feature representation and word feature construction to obtain a feature vector.

[0106] optimizing the word variable matrix according to the eigenvector and the Mahalanobis distance between words based on the linear programming constraint condition, and constructing a Mahalanobis word variable distance function;

[0107] training a classifier according to the Mahalanobis word variable distance function, predicting the classification label of the to-be-classified news chapter text by using the trained classifier, and determining the category of the to-be-classified news chapter text.

[0108] In an embodiment, when the processor executes the computer program, the following steps are also implemented: performing feature processing on the training news chapter text to obtain a word eigenvector and a word feature covariance matrix; flattening the nested structure of the training news chapter text into a one-level structure to obtain a news chapter text in a data format of a binary tuple; calculating a semi-structure-term frequency matrix and a semi-structure-inverse document frequency matrix of the news chapter text; determining a semi-structure-term frequency-inverse document frequency matrix based on the semi-structure-term frequency matrix and the semi-structure-inverse document frequency matrix and stretching the semi-structure-term frequency-inverse document frequency matrix into a vector to obtain a text vector.

[0109] In an embodiment, when the processor executes the computer program, the following steps are also implemented: taking the number of words in the news chapter text as the number of rows and the number of structure sections as the number of columns to construct a null semi-structure-term frequency matrix and a null semi-structure-inverse document frequency matrix; calculating the term frequency of each word in each structure section based on the number of occurrences of the word in the text of the structure section, and writing the term frequency as a matrix element into the null semi-structure-term frequency matrix to obtain the semi-structure-term frequency matrix; and calculating the inverse document frequency of each word in each structure section based on the size of the dictionary and the number of structure sections, and writing the inverse document frequency as a matrix element into the null semi-structure-inverse document frequency matrix to obtain the semi-structure-inverse document frequency matrix.

[0110] In an embodiment, when the processor executes the computer program, the following steps are also implemented: calculating a semi-structure-term frequency-inverse document frequency matrix based on the semi-structure-term frequency matrix and the semi-structure-inverse document frequency matrix and stretching the semi-structure-term frequency-inverse document frequency matrix into a one-dimensional vector to obtain a text vector.

[0111] In an embodiment, when the processor executes the computer program, the following steps are also implemented: removing entity words in the news chapter text.

[0112] In an embodiment, when the processor executes the computer program, the following steps are also implemented: obtaining a preset linear programming constraint condition; and optimizing a word variable matrix by using an eigenvector and a Mahalanobis distance between words under the constraint of the linear programming constraint condition to construct a Mahalanobis word variable distance function.

[0113] In an embodiment, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. When the computer program is executed by a processor, the following steps are implemented:

[0114] preprocessing the original news chapter text to obtain training news chapter text;

[0115] performing feature representation and word feature construction on the training news chapter text to obtain a feature vector;

[0116] optimizing a word transformation matrix based on a linear programming constraint condition and the feature vector and the Mahalanobis distance between words to construct a Mahalanobis word transformation distance function;

[0117] training a classifier according to the Mahalanobis word transformation distance function, predicting a classification label of the to-be-classified news chapter text by using the trained classifier, and determining the category of the to-be-classified news chapter text.

[0118] In an embodiment, the computer program, when executed by the processor, further implements the following steps: performing feature processing on the training news chapter text to obtain a word feature vector and a word feature covariance matrix; flattening the embedded structure of the training news chapter text into a one-level structure to obtain the news chapter text in the format of a binary tuple; calculating a semi-structured-term frequency matrix and a semi-structured-inverse document frequency matrix of the news chapter text; and determining a semi-structured-term frequency-inverse document frequency matrix based on the semi-structured-term frequency matrix and the semi-structured-inverse document frequency matrix and stretching the semi-structured-term frequency-inverse document frequency matrix into a vector to obtain a text vector.

[0119] In an embodiment, the computer program, when executed by the processor, further implements the following steps: constructing a null semi-structured-term frequency matrix and a null semi-structured-inverse document frequency matrix by taking the number of words in the news chapter text as the number of rows and the number of structural sections as the number of columns; calculating the term frequency of each word in each structural section based on the number of occurrences of the word in the text of the structural section, and writing the term frequency as a matrix element into the null semi-structured-term frequency matrix to obtain the semi-structured-term frequency matrix; and calculating the inverse document frequency of each word in each structural section based on the size of the dictionary and the number of structural sections, and writing the inverse document frequency as a matrix element into the null semi-structured-inverse document frequency matrix to obtain the semi-structured-inverse document frequency matrix.

[0120] In an embodiment, the computer program, when executed by the processor, further implements the following steps: calculating a semi-structured-term frequency-inverse document frequency matrix based on the semi-structured-term frequency matrix and the semi-structured-inverse document frequency matrix and stretching the semi-structured-term frequency-inverse document frequency matrix into a one-dimensional vector to obtain a text vector.

[0121] In an embodiment, the computer program, when executed by the processor, further implements the following steps: removing entity words in the news chapter text.

[0122] In an embodiment, the computer program, when executed by the processor, further implements the following steps: obtaining a preset linear programming constraint condition; and optimizing a word transformation matrix based on the linear programming constraint condition, the feature vector, and the Mahalanobis distance between words to construct a Mahalanobis word transformation distance function.

[0123] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0124] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0125] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for those skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A method for classifying text in a bidding notice section, characterized in that, include: Preprocess the original target message section text to obtain the training target message section text; The training target text is subjected to feature representation and word feature construction to obtain feature vectors; Based on linear programming constraints, the word transformation matrix is ​​optimized using the feature vectors and the Mahalanobis distance between words to construct the Mahalanobis word transformation distance function. Specifically, this includes: obtaining preset linear programming constraints; under the constraints of the linear programming conditions, optimizing the word transformation matrix using the feature vectors and the Mahalanobis distance between words to construct the Mahalanobis word transformation distance function, with the calculation formula as follows: Where MWTD(x,y) is the Mahalanobis word transition distance between text vector x and text vector y, and T is the word transition matrix, with matrix elements T ij ω is the word transformation weight from the i-th word in text vector x to the j-th word in text vector y. i Let ω be the word feature vector of the i-th word in the text vector x. j Let B be the word feature vector of the j-th word in the text vector y, and let B be the word feature covariance matrix. This is the Mahalanobis distance between the i-th word in text vector x and the j-th word in text vector y. The pre-defined linear programming constraints include three conditions: First, the word transformation matrix T is a non-negative matrix, meaning all elements in T are non-negative; second, for the i-th row vector in T, the sum of its elements equals the value of the i-th element of text vector x, and this condition must be satisfied for every row vector in T; third, for the j-th column vector in T, the sum of its elements equals the value of the j-th element of text vector y, and this condition must be satisfied for every column vector in T. A classifier is trained based on the Mahalanobis word-change distance function, and the trained classifier is used to predict the classification label of the text to be classified, thereby determining the category of the text to be classified.

2. The method according to claim 1, characterized in that, The feature vector includes word feature vectors, word feature covariance matrices, and text vectors; the process of performing feature representation and word feature construction on the training target text to obtain the feature vector includes: The training target text is processed to obtain word feature vectors and word feature covariance matrices. The embedded structure of the training target text is flattened into a first-level structure to obtain the target text in binary format; Calculate the semi-structured term frequency matrix and the semi-structured inverse document frequency matrix of the text of the marked section; The semi-structure-term-inverse-document frequency matrix is ​​determined based on the semi-structure-term frequency matrix and the semi-structure-inverse document frequency matrix, and then stretched into a vector to obtain the text vector.

3. The method according to claim 2, characterized in that, The calculation of the semi-structured term frequency matrix and the semi-structured inverse document frequency matrix of the target text includes: Using the number of words in the text of the marked section as the number of rows and the number of structural segments as the number of columns, construct a null-valued semi-structure-term frequency matrix and a null-valued semi-structure-inverse document frequency matrix; Based on the number of times the word appears in the text of the structure segment, the word frequency of each word in the text of each structure segment is calculated, and the word frequency is written as a matrix element into the null semi-structure-word frequency matrix to obtain the semi-structure-word frequency matrix; Based on the size of the dictionary and the number of the structural segments, the inverse document frequency of each word in the text of each structural segment is calculated, and the inverse document frequency is written as a matrix element into the null semi-structure-inverse document frequency matrix to obtain the semi-structure-inverse document frequency matrix.

4. The method according to claim 2, characterized in that, The step of determining the semi-structure-term-inverse-document frequency matrix based on the semi-structure-term frequency matrix and the semi-structure-inverse document frequency matrix, and then vectorizing it to obtain the text vector, includes: The semi-structure-term-inverse-document frequency matrix is ​​calculated based on the semi-structure-term-inverse-document frequency matrix and then stretched into a one-dimensional vector to obtain the text vector. The calculation formula is as follows: Where x is the text vector corresponding to the semi-structure-term frequency-inverse document frequency, Q is the semi-structure-term frequency matrix, P is the semi-structure-inverse document frequency matrix, and W is the text vector corresponding to the semi-structure-term frequency matrix. Q and W P It is a structural information transfer shared parameter matrix with the same dimension as Q and P, ☉ is an element-wise product, 1 |F| It is a vector of dimension |F| in which all elements are 1, |F| is the number of structural segments, and ||·||1 is the 1-norm of the matrix.

5. The method according to claim 2, characterized in that, Before calculating the semi-structured term frequency matrix and semi-structured inverse document frequency matrix of the bidding information section text, the method further includes: removing entity words from the bidding information section text.

6. The method according to claim 1, characterized in that, The linear programming constraints include: the word transformation matrix is ​​a non-negative matrix, the sum of the vector elements in the i-th row of the word transformation matrix is ​​equal to the value of the i-th element of the text vector x, and the sum of the vector elements in the j-th column of the word transformation matrix is ​​equal to the value of the j-th element of the text vector y.

7. A text classification device for a type of notice, characterized in that, include: The preprocessing module is used to preprocess the original target information section text to obtain the training target information section text; The feature processing module is used to perform feature representation and word feature construction on the training target text to obtain feature vectors; The module is used to construct a Mahalanobis word transition distance function by optimizing the word transition matrix based on the feature vector and the inter-word Mahalanobis distance under linear programming constraints. Specifically, it is used to obtain the preset linear programming constraints; under the constraints of the linear programming constraints, the word transition matrix is ​​obtained by optimizing the feature vector and the inter-word Mahalanobis distance, and the Mahalanobis word transition distance function is constructed. The calculation formula is as follows: Where MWTD(x,y) is the Mahalanobis word transition distance between text vector x and text vector y, and T is the word transition matrix, with matrix elements T ij ω is the word transformation weight from the i-th word in text vector x to the j-th word in text vector y. i Let ω be the word feature vector of the i-th word in the text vector x. j Let B be the word feature vector of the j-th word in the text vector y, and let B be the word feature covariance matrix. This is the Mahalanobis distance between the i-th word in text vector x and the j-th word in text vector y. The pre-defined linear programming constraints include three conditions: First, the word transformation matrix T is a non-negative matrix, meaning all elements in T are non-negative; second, for the i-th row vector in T, the sum of its elements equals the value of the i-th element of text vector x, and this condition must be satisfied for every row vector in T; third, for the j-th column vector in T, the sum of its elements equals the value of the j-th element of text vector y, and this condition must be satisfied for every column vector in T. The classification module is used to train a classifier based on the Mahalanobis word-change distance function, and to use the trained classifier to predict the classification label of the text to be classified, thereby determining the category of the text to be classified.

8. A computer device comprising a processor and a memory, the memory storing a computer program, characterized in that, The processor is used to implement the text classification method for the label section as described in any one of claims 1-6 when executing the computer program.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the text classification method for the label section as described in any one of claims 1-6.

Images