Intelligent quality problem information matching method and system based on text understanding

Abstract

The invention provides an intelligent quality problem information matching method and system based on text understanding, and the method comprises the steps: carrying out the data preprocessing, and carrying out the data dimension reduction of the preprocessed sample data; performing feature extraction, wherein the feature extraction comprises feature extraction according to distance measurement, feature extraction based on information measurement, feature extraction based on dependency measurement and feature extraction based on consistency measurement; vectorizing the quality problem information by using a vector space model; constructing a feature extraction algorithm; calculating Chinese text similarity; constructing a similarity calculation module; and a problem prediction software model is constructed, and intelligent matching of quality problem information is carried out. Unified processing and fusion are carried out on the data through the ET L process. And aiming at the fused high-quality data, researching a multi-type text intelligent understanding technology based on machine learning and a text intelligent understanding technology based on combination of rules and statistics, and obtaining core data in quality problem information in a question and answer mode.

Description

Technical Field

[0001] The present invention relates to the technical field of data information processing, and in particular to a method and system for intelligent matching of quality problem information based on text understanding. Background Art

[0002] Information resource exchange and sharing refers to the exchange and sharing of information and information products between information systems at different levels and departments. It is to share information, a resource that is becoming increasingly important in the Internet era, with others in order to achieve more reasonable resource allocation and save costs. Specifically, information resource exchange and sharing is based on information system technology and transmission technology, which enables information resources to be exchanged and shared between information systems at different levels and departments, which can improve the utilization rate of information resources and avoid repeated waste in information collection, storage and management.

[0003] In the process of information resource exchange and sharing, information resources usually have characteristics such as multi-source, heterogeneous and massive, which can easily cause the problem of "information islands" and is not conducive to the efficient use of information resources across applications, dimensions and levels. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a quality problem information intelligent matching method and system based on text understanding, and uniformly processes and integrates data through the ETL process. For the integrated high-quality data, various types of text intelligent understanding technologies based on machine learning and text intelligent understanding technologies based on the combination of rules and statistics are studied, and the core data in the quality problem information is obtained through question-answering.

[0005] The first object of the present invention is to provide a quality problem information intelligent matching method based on text understanding, including data preprocessing, and further comprising the following steps:

[0006] Step 1: Perform data dimension reduction on the preprocessed sample data;

[0007] Step 2: performing feature extraction, wherein the feature extraction includes feature extraction based on distance measurement, feature extraction based on information measurement, feature extraction based on dependency measurement, and feature extraction based on consistency measurement;

[0008] Step 3: Use the vector space model to vectorize the quality problem information;

[0009] Step 4: Construct feature extraction algorithm;

[0010] Step 5: Calculate the similarity of Chinese texts;

[0011] Step 6: Construct similarity calculation module;

[0012] Step 7: Build a problem prediction software model and perform intelligent matching of quality problem information.

[0013] Preferably, the data preprocessing includes text cleaning, word segmentation, and stop word removal of unstructured text to obtain text data with a standardized and unified format.

[0014] In any of the above solutions, preferably, the data dimension reduction is represented as

[0015] 1) It represents a sample set in the common dimensional space. represents a data set in the space d = (d < < D), where N is the sample size, i is the i-th sample, and D is the number of features;

[0016] 2) Dimensionality reduction mapping, M:X→Y, x→y=M(x), is called y as the low-dimensional representation of x.

[0017] In any of the above schemes, preferably, step 1 includes principal component analysis and independent component analysis,

[0018] The principal component analysis includes: for a given data set Among them, x i ∈R N is an N-dimensional column vector, M represents the number of samples;

[0019] Assume that the given data has been zeroed, that is:

[0020] The covariance matrix is ​​defined as:

[0021] Since C is a semi-positive definite matrix, it is diagonalized, that is, C = V∧V T ,in,

[0022] ∧=diag(λ1,λ2,…λ n ),VV T =1;

[0023] Assuming that the rank of the semi-positive definite matrix C is d, that is, rank(C) = d, then the semi-positive definite matrix C has d non-zero eigenvalues, denoted by λ1≥λ2≥…λ d >0;

[0024] Select the eigenvectors v1≥v2≥…v corresponding to the first k largest eigenvalues k The eigenvector matrix V = {v1≥v2≥…v k};

[0025] The sample data X is represented by the eigenvector matrix V.

[0026] In any of the above schemes, preferably, the independent component analysis includes:

[0027] Assume that the data S∈R consists of S independent elements n , we observed data x, x = AS,

[0028] Where A is the unknown mixing matrix and x is the observed data {x i ; i = 1, 2, ..., m}, a total of m samples,

[0029] Assume that each signal source S i The distribution of is given by the density function p S Given, we get the joint distribution function of the signal S:

[0030]

[0031] W is the pseudo-inverse matrix of A, and the joint distribution function of the signal x is obtained:

[0032]

[0033] For a random variable z, its cumulative distribution function is defined as Then the probability density function of z is derived from the cumulative distribution function, that is, p z (z) = F'(z);

[0034] The rectangular matrix W is the parameter in the model. Given a training set {x i ; i=1,2,…,m}, the log-likelihood estimate of the sample is as follows:

[0035]

[0036] According to the gradient descent algorithm, Derive the random skill increase learning rule, for a training set x (i) , the update rule is as follows, where a is the learning rate;

[0037]

[0038] By calculating S i =Wx (i) Restore the original signal.

[0039] In any of the above solutions, preferably, the Relief algorithm is used to implement the feature extraction based on the distance metric, and the hypothesis interval is defined as the maximum distance that the decision surface can move while keeping the sample classification unchanged, expressed as:

[0040]

[0041] Among them, M(d) and H(d) are the nearest neighbor points of the same type and different type as d respectively;

[0042] Assume D = {d1, d2, ... d n} is the text set to be classified, d i ={x1,x2,…x n} corresponds to its i-th document, x j represents the jth eigenvalue;

[0043] Randomly select m samples from all training sets, calculate the hypothesis interval of the samples on each feature, and use the accumulated value as the weight of the feature. Sample d updates the feature x j The weight can be expressed as:

[0044]

[0045] Since text features are discrete values, the function diff is defined as:

[0046]

[0047] In any of the above schemes, it is preferred that information gain, mutual information, x 2 The statistics and expected cross entropy measure the information metric;

[0048] The information gain is used to calculate the weight of the feature in the decision tree. The feature is the difference between the entropy without considering the feature and the entropy after considering the feature. The information entropy is defined as follows:

[0049]

[0050] The calculation formula of IG is:

[0051]

[0052] Among them, P(c i ) indicates category C i The probability of appearing in the training set, P(x) is the probability of feature x appearing, P(c|x) represents the probability of a document not containing feature x in the training set, and P(c|x) represents the category C when feature x appears. i The probability of Category C i The number of documents in the training set that do not contain feature t divided by the number of documents in the training set that do not contain feature x.

[0053] In any of the above schemes, preferably, the mutual information is used to measure the mutual dependence between two signals in one information, and is used to calculate the dependence between feature x and category c, and the mutual information of feature x and each category is integrated as the weight of the feature value;

[0054] The mutual information MI(t,c) between feature x and category c is defined as follows:

[0055]

[0056] The approximate calculation formula is:

[0057]

[0058] Among them, E is the number of times feature x and category c appear at the same time, B is the number of times feature x appears but category c does not appear, F is the number of times category c appears but feature x does not appear, and N is the total number of documents;

[0059] When there are multiple categories, the mutual information of x for c is calculated as follows:

[0060]

[0061] In any of the above schemes, preferably, the x 2 The statistic is used to measure the lack of independence between feature items and categories and calculate the CHI value of feature item x for category c. The formula is:

[0062]

[0063] Where D is the number of documents that neither contain feature x nor belong to category c, and N is the total number of documents.

[0064] In any of the above solutions, preferably, the calculation formula of the expected cross entropy is:

[0065]

[0066] Among them, P(c i |x) means that when word x appears in the text, the text belongs to category c i The probability, P(c i ) is of category c i Probability of occurrence.

[0067] In any of the above schemes, preferably, the feature extraction based on dependency measurement is to extract the optimal evaluation criterion, and the formula is:

[0068] T0=argmaxJ(o),st|Y|≤z

[0069] Among them, z is the upper limit of the number of selected features, o is the set of selected features, and J(o) is the evaluation criterion.

[0070] In any of the above solutions, preferably, the feature extraction based on consistency measurement includes the following sub-steps:

[0071] Step 21, specify a maximum number of selections K and inconsistency rate δ in the initial stage, and set the initial subset Z;

[0072] Step 22, randomly select a candidate subset Z' in Z;

[0073] Step 23, if the number of features contained in the candidate subset Z' is less than the number of features contained in Z, proceed to the next step, otherwise return to step 22;

[0074] Step 24, calculate the inconsistency rate, if J(o)≤δ, replace the best subset;

[0075] Step 25, determine whether the number of cycles reaches a threshold, if so, end, otherwise return to step 22.

[0076] In any of the above solutions, preferably, the workflow of the similarity calculation module includes the following sub-steps:

[0077] Step 61: Obtaining the items to be checked for duplicates according to the user's similarity analysis request or the item intelligent retrieval request;

[0078] Step 62: Filter the approved projects in the index library according to the application type of the quality problem information to be checked;

[0079] Step 63: The quality problem information to be checked for duplicates and the reported quality problem information obtained through filtering are calculated based on the project similarity method to obtain the similarity value of each project.

[0080] The second object of the present invention is to provide a quality problem information intelligent matching system based on text understanding, including an index library construction module, an intelligent retrieval module, a similarity calculation module and a determination module.

[0081] Therefore, the index library construction module is used to obtain all quality problem information from the database, including both approved and unapproved types;

[0082] The intelligent search module is used to match the most relevant items in the search index library according to the query keywords input by the user and return the information of these items;

[0083] The determination module is used to analyze the set of similarity values ​​obtained according to a specific threshold after the information to be checked for duplicates passes through the similarity calculation module, and then give the project similarity result and return the list of similar quality problem information to the user;

[0084] The system adopts the method described in the first purpose to realize intelligent matching of quality problem information based on text understanding.

[0085] Preferably, the workflow of the similarity calculation module includes the following sub-steps:

[0086] Step 601: Obtaining the items to be checked for duplicates according to the user's similarity analysis request or the item intelligent retrieval request;

[0087] Step 602: Filter the approved projects in the index library according to the application type of the quality problem information to be checked;

[0088] Step 603: The quality problem information to be checked for duplicates and the reported quality problem information obtained through filtering are calculated based on the project similarity method to obtain similarity values ​​for each project.

[0089] In any of the above solutions, preferably, the workflow of the determination module includes the following sub-steps:

[0090] Step 611: receiving the similarity of each question information processed by the project similarity calculation module;

[0091] Step 612: Compare the similarity value with the specified threshold. If it is greater than the threshold, add the corresponding project ID and field information to the corresponding list;

[0092] Step 613: sort the final list obtained and return it to the user;

[0093] Step 614: Check similar project information and highlight the parts with the same keywords as those in the project to be checked for duplicates.

[0094] The present invention proposes a method and system for intelligent matching of quality problem information based on text understanding, which provides application function services such as duplicate identification of the same problem, discovery of repeated problems, recommendation of treatment measures, intelligent data cleaning, and prediction of potential problems for the quality problem similarity intelligent analysis system. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] Figure 1 The present invention is a flowchart of a preferred embodiment of the method for intelligently matching quality problem information based on text understanding according to the present invention.

[0096] Figure 2 It is a schematic diagram of a framework of an embodiment of feature extraction of the quality problem information intelligent matching method based on text understanding according to the present invention.

[0097] Figure 3This is a flow system architecture diagram of a preferred embodiment of the quality problem information intelligent matching system based on text understanding according to the present invention.

[0098] Figure 4 The present invention is a flowchart of an embodiment of the interaction between modules of the quality problem information intelligent matching system based on text understanding according to the present invention.

[0099] Figure 5 The present invention is a flowchart of an embodiment of index library construction of the quality problem information intelligent matching method based on text understanding according to the present invention.

[0100] Figure 6 The tree structure diagram of an index library file library according to an embodiment of the quality problem information intelligent matching method based on text understanding of the present invention.

[0101] Figure 7 The figure is a flow chart of an embodiment of intelligent retrieval of the quality problem information intelligent matching method based on text understanding according to the present invention.

[0102] Figure 8 The figure is a flow chart of an embodiment of similarity calculation of the quality problem information intelligent matching method based on text understanding according to the present invention.

[0103] Fig. 9 The present invention is a flowchart of an embodiment of the operation of the determination module of the quality problem information intelligent matching system based on text understanding according to the present invention. DETAILED DESCRIPTION

[0104] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0105] Embodiment 1

[0106] This embodiment provides a quality problem information intelligent matching method based on text understanding, such as Figure 1 As shown, step 100 is executed, data preprocessing, including text cleaning, word segmentation, and stop word removal of unstructured text to obtain text data with a standardized and unified format.

[0107] Step 110 is executed to perform data dimension reduction on the preprocessed sample data. The data dimension reduction is expressed as

[0108] 1) It represents a sample set in the common dimensional space. represents a data set in the space d = (d < < D), where N is the sample size, i is the i-th sample, and D is the number of features;

[0109] 2) Dimensionality reduction mapping, M:X→Y, x→y=M(x), is called y as the low-dimensional representation of x.

[0110] The step 110 includes principal component analysis and independent component analysis.

[0111] The principal component analysis includes: for a given data set Among them, x i ∈R N is an N-dimensional column vector, M represents the number of samples;

[0112] Assume that the given data has been zeroed, that is:

[0113] The covariance matrix is ​​defined as:

[0114] Since C is a semi-positive definite matrix, it is diagonalized, that is, C = V∧V T ,in,

[0115] ∧=diag(λ1,λ2,…λ n ),VV T =1;

[0116] Assuming that the rank of the semi-positive definite matrix C is d, that is, rank(C) = d, then the semi-positive definite matrix C has d non-zero eigenvalues, denoted by λ1≥λ2≥…λ d >0;

[0117] Select the eigenvectors v1≥v2≥…v corresponding to the first k largest eigenvalues k The eigenvector matrix V = {v1≥v2≥…v k};

[0118] The sample data X is represented by the eigenvector matrix V.

[0119] The independent component analysis includes:

[0120] Assume that the data S∈R consists of S independent elements n , we observed data x, x = AS,

[0121] Where A is the unknown mixing matrix and x is the observed data {x i ; i = 1, 2, ..., m}, a total of m samples, assuming that each signal source S i The distribution of is given by the density function p S Given, we get the joint distribution function of the signal S:

[0122]

[0123] W is the pseudo-inverse matrix of A, and the joint distribution function of the signal x is obtained:

[0124]

[0125] For a random variable z, its cumulative distribution function is defined as Then the probability density function of z is derived from the cumulative distribution function, that is, p z (z) = F'(z);

[0126] The rectangular matrix W is the parameter in the model. Given a training set {x i ; i=1,2,…,m}, the log-likelihood estimate of the sample is as follows:

[0127]

[0128] According to the gradient descent algorithm, Derive the random skill increase learning rule, for a training set x (i) , the update rule is as follows, where a is the learning rate;

[0129]

[0130] By calculating S i =Wx (i) Restore the original signal.

[0131] Execute step 120 to perform feature extraction, wherein the feature extraction includes feature extraction based on distance measurement, feature extraction based on information measurement, feature extraction based on dependency measurement, and feature extraction based on consistency measurement;

[0132] The Relief algorithm is used to implement the feature extraction based on the distance metric, and the hypothesis interval is defined as the maximum distance that the decision surface can move while keeping the sample classification unchanged, expressed as:

[0133]

[0134] Among them, M(d) and H(d) are the nearest neighbor points of the same type and different type as d respectively;

[0135] Assume D = {d1, d2, ... d n} is the text set to be classified, d i ={x1,x2,…x n} corresponds to its i-th document, x j represents the jth eigenvalue;

[0136] Randomly select m samples from all training sets, calculate the hypothesis interval of the samples on each feature, and use the accumulated value as the weight of the feature. Sample d updates the feature x j The weight can be expressed as:

[0137]

[0138] Since text features are discrete values, the function diff is defined as:

[0139]

[0140] Using information gain, mutual information, x 2 The statistics and expected cross entropy measure the information metric;

[0141] The information gain is used to calculate the weight of the feature in the decision tree. The feature is the difference between the entropy without considering the feature and the entropy after considering the feature. The information entropy is defined as follows:

[0142]

[0143] The calculation formula of IG is:

[0144]

[0145] Among them, P(c i ) indicates category C i The probability of appearing in the training set, P(x) is the probability of feature x appearing, P(c|x) represents the probability of a document not containing feature x in the training set, and P(c|x) represents the category C when feature x appears. i The probability of Category C i The number of documents in the training set that do not contain feature t divided by the number of documents in the training set that do not contain feature x.

[0146] The mutual information is used to measure the mutual dependence between two signals in a message, and is used to calculate the dependence between feature x and category c, and the mutual information between feature x and each category is integrated as the weight of the feature value;

[0147] The mutual information MI(t,c) between feature x and category c is defined as follows:

[0148]

[0149] The approximate calculation formula is:

[0150]

[0151] Among them, E is the number of times feature x and category c appear at the same time, B is the number of times feature x appears but category c does not appear, F is the number of times category c appears but feature x does not appear, and N is the total number of documents;

[0152] When there are multiple categories, the mutual information of x for c is calculated as follows:

[0153]

[0154] The x 2 The statistic is used to measure the lack of independence between feature items and categories and calculate the CHI value of feature item x for category c. The formula is:

[0155]

[0156] Where D is the number of documents that neither contain feature x nor belong to category c, and N is the total number of documents.

[0157] The calculation formula of the expected cross entropy is:

[0158]

[0159] Among them, P(c i |x) means that when word x appears in the text, the text belongs to category c i The probability, P(c i ) is of category c i Probability of occurrence.

[0160] The feature extraction based on dependency measurement is to extract the optimal evaluation criterion, and the formula is:

[0161] T0=argmaxJ(o),st|Y|≤z

[0162] Among them, z is the upper limit of the number of selected features, o is the set of selected features, and J(o) is the evaluation criterion.

[0163] The feature extraction based on consistency measurement includes the following sub-steps:

[0164] Execute step 121, specify a maximum selection number K and inconsistency rate δ in the initial stage, and set an initial subset Z;

[0165] Execute step 122 to randomly select a candidate subset Z' in Z;

[0166] Execute step 123, if the number of features contained in the candidate subset Z' is less than the number of features contained in Z, proceed to the next step, otherwise return to step 22;

[0167] Execute step 124 to calculate the inconsistency rate, and if J(o)≤δ, replace the best subset;

[0168] Execute step 125 to determine whether the number of cycles reaches the threshold. If so, end the process; otherwise, return to step 22.

[0169] Execute step 130 to vectorize the quality problem information using a vector space model;

[0170] Execute step 140 to construct a feature extraction algorithm;

[0171] Execute step 150 to calculate the Chinese text similarity;

[0172] Execute step 160 to construct a similarity calculation module. The workflow of the similarity calculation module includes the following sub-steps:

[0173] Execute step 161 to obtain the items to be checked for duplicate content according to the user similarity analysis request or the item intelligent retrieval request;

[0174] Execute step 162 to filter the approved projects in the index library according to the application type of the quality problem information to be checked for duplicate quality;

[0175] Execute step 163 to calculate the similarity value of each project based on the project similarity calculation method for the quality problem information to be checked for duplicate and the reported quality problem information obtained by filtering.

[0176] Step 170: construct a problem prediction software model and perform intelligent matching of quality problem information.

[0177] Embodiment 2

[0178] This embodiment provides a quality problem information intelligent matching system based on text understanding, including an index library construction module, an intelligent retrieval module, a similarity calculation module and a determination module.

[0179] Therefore, the index library construction module is used to obtain all quality problem information from the database, including both approved and unapproved types.

[0180] The intelligent search module is used to match the most relevant items in the search index library according to the query keywords input by the user and return the information of these items.

[0181] The workflow of the similarity calculation module includes the following sub-steps:

[0182] Step 601: Obtaining the items to be checked for duplicates according to the user's similarity analysis request or the item intelligent retrieval request;

[0183] Step 602: Filter the approved projects in the index library according to the application type of the quality problem information to be checked;

[0184] Step 603: The quality problem information to be checked for duplicates and the reported quality problem information obtained through filtering are calculated based on the project similarity method to obtain similarity values ​​for each project.

[0185] The determination module is used to analyze the set of similarity values ​​obtained according to a specific threshold after the information to be checked for duplicates passes through the similarity calculation module, and then give the project similarity result and return the list of similar quality problem information to the user. The workflow of the determination module includes the following sub-steps:

[0186] Step 611: receiving the similarity of each question information processed by the project similarity calculation module;

[0187] Step 612: Compare the similarity value with the specified threshold. If it is greater than the threshold, add the corresponding project ID and field information to the corresponding list;

[0188] Step 613: sort the final list obtained and return it to the user;

[0189] Step 614: Check similar project information and highlight the parts with the same keywords as those in the project to be checked for duplicates.

[0190] Embodiment 3

[0191] A quality problem information intelligent matching method based on text understanding, comprising:

[0192] 1. Data preprocessing

[0193] The text data of quality problem information, as a carrier of information, cannot be understood by computers in an unstructured format. The purpose of text data preprocessing is to remove formatting tags, word segmentation, and stop words from unstructured text to obtain text data with a standardized and unified format in preparation for the next step of classification. The text, as a carrier of information, cannot be understood by computers in an unstructured format. The purpose of text preprocessing is to clean, word segmentation, and stop words from unstructured text to obtain text data with a standardized and unified format in preparation for the next step of classification.

[0194] The entries obtained by the word segmentation process are expressed in a structured form to provide standardized data for feature extraction calculation. The text after word segmentation is composed entirely of entries, which are effective expressions of the text theme. In order to facilitate the computer to distinguish different entries, special characters can be used to separate the entries.

[0195] During the data preprocessing stage, the quality of the original data will directly affect the final linking results. Different data sets often describe the same data in different ways. Normalizing this data is an important step to improve the accuracy of subsequent links. Common data preprocessing methods include:

[0196] (1) Syntax regularization, mainly including syntax matching and comprehensive attributes, etc. The specific situation is as follows:

[0197] 1) Syntax matching: such as the representation method of equipment name;

[0198] 2) Comprehensive attributes: such as the way of expressing the cause of the fault.

[0199] (2) Data normalization, mainly including elimination of special characters, error checking and data normalization, etc. The details are as follows:

[0200] 1) Eliminate special symbols: remove spaces, "", "", "- and other symbols;

[0201] 2) Error checking: topological errors of the input error class;

[0202] 3) Data normalization: Replace nicknames and abbreviations with formal names.

[0203] 1.1 Word segmentation method

[0204] In the text after formatting tags are removed, the text needs to be segmented to continue processing. In English, words are separated by spaces, which is easier to process in terms of semantic accuracy and complexity. Since the characteristics of the Chinese language are that words are composed of characters, sentences are composed of phrases, and texts are composed of sentences, when processing Chinese text, the article must be segmented first, and words with independent meanings are separated from sentences as the basic units for identifying semantics. Commonly used segmentation algorithms are mainly divided into three categories: segmentation methods based on string matching, segmentation methods based on understanding, and segmentation methods based on statistics. According to whether it is combined with the part-of-speech tagging process, it can be divided into a simple segmentation method and an integrated method combining segmentation and tagging.

[0205] (1) Word segmentation method based on string matching. This method is also called mechanical word segmentation method. It matches the Chinese character string to be analyzed with the entries in a "sufficiently large" machine dictionary according to a certain strategy. If a certain string is found in the dictionary, the match is successful. Several commonly used mechanical word segmentation methods are as follows: forward maximum matching method (from left to right); reverse maximum matching method (from right to left); minimum segmentation (minimizing the number of words segmented in each sentence); bidirectional maximum matching method (scanning twice from left to right and from right to left).

[0206] (2) Understanding-based word segmentation method. This word segmentation method achieves the effect of word recognition by letting the computer simulate human understanding of sentences. Its basic idea is to perform syntactic and semantic analysis while segmenting words, and use syntactic information and semantic information to deal with ambiguous phenomena. It usually includes three parts: word segmentation subsystem, syntactic and semantic subsystem, and general control part. Under the coordination of the general control part, the word segmentation subsystem can obtain syntactic and semantic information about words, sentences, etc. to judge word segmentation ambiguity, that is, it simulates the process of human understanding of sentences. This word segmentation method requires the use of a large amount of language knowledge and information. Due to the generality and complexity of Chinese language knowledge, it is difficult to organize various language information into a form that can be directly read by machines. Therefore, the word segmentation system based on understanding is still in the experimental stage.

[0207] (3) Word segmentation method based on statistics. A word is a stable combination of characters. Therefore, in a context, the more times adjacent characters appear at the same time, the more likely they are to form a word. Therefore, the frequency or probability of adjacent co-occurrence of characters can better reflect the credibility of the word. The frequency of the combination of adjacent co-occurring characters in the corpus can be counted to calculate their mutual occurrence information. Define the mutual occurrence information of two characters and calculate the probability of adjacent co-occurrence of two Chinese characters X and Y. The mutual occurrence information reflects the closeness of the combination relationship between Chinese characters. When the closeness is higher than a certain threshold, it can be considered that this character group may form a word. This method only needs to count the frequency of the character group in the corpus and does not require a segmentation dictionary. Therefore, it is also called dictionary-free word segmentation or statistical word extraction method.

[0208] 1.2 Commonly used word segmentation tools

[0209] Among the many Chinese word segmentation tools, the most famous one is the "Language Cloud" system of Harbin Institute of Technology. This system integrates a variety of natural language processing technologies and provides Chinese word segmentation, part-of-speech tagging, named entity recognition and other operations in the form of cloud services. The word segmentation accuracy rate is as high as 97.3%. Many Internet companies, such as Baidu, Tencent, Huawei, Kingsoft, etc. are users of its commercial version.

[0210] NLPIR (ICTCLAS2015) developed by Dr. Zhang Huaping and others from the Institute of Computing Technology of the Chinese Academy of Sciences also performs well in word segmentation, and also provides functions such as part-of-speech tagging, named entity recognition, and user dictionary. Functions such as microblog word segmentation, new word discovery, and keyword extraction are provided free of charge to developers for non-commercial use and research.

[0211] 1.3 Text segmentation function design

[0212] By comparing the processing performance and stability of the above two word segmentation tools, the project was implemented using ICTCLAS of the Chinese Academy of Sciences as the word segmentation tool. In stand-alone mode, the input of the word segmentation tool is a text file, and the output is a word segmentation vector expressed in "word / part of speech".

[0213] The implementation process of this function is as follows: Figure 6 .3 quality. Specifically, documents is the document data read by the program from Hbase. The data has passed the denoising process and is a list of text strings in the body of the web page. The Parallelize method is used to parallelize the data to form a JavaRDD type data set. Like the denoising function, the document is of String object type before and after word segmentation, which belongs to a one-to-one mapping relationship, so the map function is selected as the parallelization operation. The func method in the Map function performs the word segmentation of a single document. SegmentProxy is the driver class of the word segmentation function of ICTCLAS of the Chinese Academy of Sciences, which is used to call the word segmentation function.

[0214] 1.4 Remove stop words

[0215] After the text is segmented, not all the word strings are useful. Some words or characters need to be automatically filtered out. These words or characters are called StopWords. These stop words are generally input manually, and the generated stop words will form a stop word list. However, there is no clear stop word list that can be applied to all tools. Some tools even explicitly avoid using stop words to support phrase search. For a given purpose, any category of words can be selected as stop words. In general, stop words can be roughly divided into two categories. One category is the function words contained in human language. These function words are extremely common. Compared with other words, function words have no practical meaning, such as 'the', 'is', 'at', 'which', 'on' and 'this', 'that', 'of', 'get', 'ground' in Chinese. The other category of words includes vocabulary words, such as 'want' and 'after', 'should', 'to', 'some', 'after', 'one', etc. These words are widely used, but have no obvious category meaning. If it is not removed, it will easily be selected as a feature word and affect the classification accuracy.

[0216] 1.5 Design of stop word removal function

[0217] The stop word removal function is similar to the word segmentation function and is a map operation. First, the stop word list is input as a string and declared as a broadcast variable type to inform Spark that the table needs to be copied to each computing node. Then, an RDD dataset is generated with the document vector after word segmentation as an element, and the words in the document are traversed in the map. If the word is included in the stop word list, the word is deleted from the document vector. Finally, the document after the stop words are removed is returned.

[0218] 2. Data Dimensionality Reduction

[0219] Dimensionality reduction methods include feature selection and feature extraction. The former refers to selecting a set of feature subsets that are most beneficial to classification from the original high-dimensional features (variables or attributes) according to a certain criterion, and the latter aims to find a mapping to transform the original space into a space with a lower dimension and map the original data to this space, so as to concentrate the classification information scattered in the original features on a small number of new features. The difference between the two is that the features selected by feature selection still belong to the original feature set, while the new features generated by feature extraction are a combination of the original features and often do not belong to the original feature set. It can be seen that feature selection can be understood as a special form of feature extraction. This article only focuses on feature extraction and regards it as "dimensionality reduction" in a narrow sense. Dimensionality reduction of high-dimensional data has become a core issue of common concern in many fields such as machine learning, pattern recognition, data mining and information retrieval.

[0220] The basic idea of ​​data dimensionality reduction is to perform some transformation operation on the sample data in the input space, map the high-dimensional sample data to a low-dimensional space, and finally obtain a low-dimensional representation of the original data space in the low-dimensional space. In order to facilitate the study of data dimensionality reduction below, a mathematical description of data dimensionality reduction is given: (1) It represents a sample set in the common dimensional space. It represents a data set in the space d = (d << D);

[0221] (2) Dimensionality reduction mapping, M:C→Y, x→y=M(x); it is called y as the low-dimensional representation of x.

[0222] 2.1 Principal Component Analysis (PCA)

[0223] The basic idea of ​​principal component analysis is to construct a series of linear combinations of the original variables to form several comprehensive geometric indicators, so as to remove the correlation of the data and make the low-dimensional data retain the variance information of the original high-dimensional data to the greatest extent.

[0224] For a given data set Among them, x i ∈R N is an N-dimensional column vector, where M represents the number of samples.

[0225] Assume that the given data has been zeroed, that is:

[0226] The covariance matrix is ​​defined as:

[0227] Since C is a semi-positive definite matrix, it is diagonalized, that is, C = V∧V T ,in,

[0228] ∧=diag(λ1,λ2,…λ n ), VV T =1;

[0229] Assume that the rank of the positive semi - definite matrix C is d, that is, rank(C)=d. Then the positive semi - definite matrix C has d non - zero eigenvalues, denoted as λ1≥λ2≥…λ d >0. Select the eigenvectors v1≥v2≥…v k corresponding to the first k larger eigenvalues to form the eigenvector matrix V = {v1≥v2≥…v k}. We can use the eigenvector matrix V to represent the sample data X, thus completing the reduction of the data from n - dimension to k - dimension (k << n), achieving the purpose of feature extraction and reducing the variable dimension.

[0230] The principal component analysis (PCA) method is characterized by easy calculation and strong interpretability. The criterion for the PCA algorithm to measure information is the variance value of the data: the larger the variance, the more key information it contains; otherwise, the less information it contains. Therefore, PCA is a process of coordinate transformation that projects the coordinates of high - dimensional data onto the direction with the largest data variance to form a new coordinate system to represent the coordinates of the data.

[0231] 2.2 Independent Component Analysis (ICA)

[0232] Independent Component Analysis is a new signal processing method proposed by Jutten and Herault in the 1990s. The ICA algorithm originally originated from Blind Source Separation, that is, obtaining the source signal s from the collected mixed signal x.

[0233] Assume that the data S ∈ R n consisting of S independent components, and the data x we observe,

[0234] x = AS,

[0235] where A is the unknown mixing matrix, x is the observed data {x i ; i = 1,2,…,m}, a total of m samples, and our goal is to recover the original signal S (i) . We assume that the distribution of each signal source S i is given by the density function p S . From this, we can obtain the joint distribution function of the signal S:

[0236]

[0237] Since we assume that the signal sources are independent of each other, from the previous x = AS = W-1 s, W is the pseudo-inverse matrix of A, we can get the joint distribution function of signal x

[0238]

[0239] From the knowledge of probability, we know that for a random variable z, its cumulative distribution function (cdf) can be defined as Then the probability density function of z can be derived from the cumulative distribution function, that is; p z (z) = F'(z). In order to find s i The probability density function needs to specify a cumulative distribution function F(s). F(s) needs to meet two conditions: 1) the function is monotonically increasing; 2) the function value range is within the range of [0.1]. Since the ICA algorithm cannot be applied to Gaussian distribution data, the cumulative distribution function we choose cannot be a Gaussian function. The Sigmoid function has the above properties, so the Sigmoid function is used in the algorithm.

[0240] Then Ps(s)P S (s) is obtained by the derivation of the Sigmoid function, that is, p s (s) = g′(s).

[0241] The rectangular matrix W is the parameter in the model. Given a training set {x i ; i=1,2,…,m}, the log-likelihood estimate of the sample is as follows:

[0242]

[0243] The next step is to find W. According to the gradient descent algorithm, We can easily derive the random skill ascent learning rule, for a training set x (i) , the update rule is as follows, where a is the learning rate.

[0244]

[0245] When the algorithm iterates to find W, by calculating S i =Wx (i) To restore the original signal. PCA is essentially a problem of basis transformation, so that the transformed data has the maximum variance; PCA is not only to reduce the dimensionality of high-dimensional data, but more importantly, it removes noise through dimensionality reduction. ICA is to find the independent parts of the signal (no orthogonality is required), corresponding to high-order statistical component analysis. ICA reduces the dimensionality of the original data and extracts independent attributes, looking for a linear transformation z = Wx, so that the independence of the components of z is maximized.

[0246] In summary, we can see that compared with PCA, ICA can remove the correlation between the processed components, suppress Gaussian noise, and are statistically independent of each other. Therefore, it can better characterize the random statistical characteristics of the variables.

[0247] 3. Feature extraction

[0248] After the documents are preprocessed, they cannot be compared because the length and keywords of each article are different, and they cannot be directly classified. Text feature selection is a key technology in the text classification process. It obtains several keywords that best represent the article by performing statistical analysis on the article, and replaces the article with these keywords. This process is called feature extraction for text classification, which is a feature dimensionality reduction method applied to text. The text feature selection process can be roughly divided into four parts: generating text feature subsets, evaluating text feature subsets, termination conditions, and optimal text feature subsets. Figure 2 shown.

[0249] Feature selection algorithms can be divided into four categories according to the evaluation function: distance measurement, information measurement, dependency measurement and consistency measurement.

[0250] 3.1 Feature extraction algorithm based on distance metric

[0251] Distance metrics are usually considered as measures of separability, difference, or recognition. The most commonly used distance metrics are Euclidean distance, S-order Minkowski measure, Chebychev distance, square distance, etc. In a binary classification problem, for feature X and feature Y, if the difference in the conditional probabilities of the two classes caused by X is greater than Y, then X is better than Y. Because the purpose of feature selection is to find features that separate the two classes as much as possible. If the difference is 0, then X and Y are indistinguishable. The more well-known algorithm in this category is the Relief algorithm.

[0252] The Relief algorithm is a weighted search feature subset algorithm proposed by Kira and Rendell. It assigns a weight to each feature and uses the weight to represent the correlation between the feature and the category. The hypothesis margin is defined as the maximum distance that the decision surface can move while keeping the sample classification unchanged, which can be expressed as:

[0253]

[0254] Where M(d) and H(d) are the nearest neighbor points of the same type and different type of d, respectively. By calculating the hypothetical interval of the training sample, the classification value of the feature can be approximately evaluated. The Relief algorithm uses this feature to assign a certain weight to each feature in the feature set. The specific idea is as follows: Assume D = {d1, d2, ... dn} is the text set to be classified, d i ={x1,x2,…x n} corresponds to its i-th document, x j Relief algorithm randomly selects m samples from all training sets, calculates the hypothesis interval of the samples on each feature, and adds them up as the weight of the feature. Sample d updates feature x j The weight can be expressed as:

[0255]

[0256] 由于文本特征为离散值 , function diff 定义为 :

[0257]

[0258] From the formula, we can see that when a feature contributes more to classification, it means that the distance between samples of the same type is closer, while the distance between samples of different types is farther, and the weight is correspondingly larger; on the contrary, if the feature contributes less to classification, the weight should be smaller or even approach zero after calculating a large number of samples.

[0259] 3.2 Feature extraction algorithm based on information measurement

[0260] Information metrics are usually measured using information gain (IG), mutual information (MI), CHI statistics, and expected cross entropy (ECE). Information gain is defined as the difference between prior uncertainty and expected posterior uncertainty. It can effectively select key features and eliminate irrelevant features. Mutual information describes the strength of the interdependence between two random variables. CHI statistics describe the independence between features and categories. Expected cross entropy is similar to information gain, but it differs from information gain in that it does not consider the situation where a word does not occur in a document.

[0261] 1) Information Gain

[0262] Information gain is a concept in machine learning that is used to calculate the weight of features in a decision tree. It is defined as the average value of the class feature vector. All the information that this feature can provide for classification is represented by the difference between the entropy without considering this feature and the entropy after considering this feature. The definition of information entropy is as follows:

[0263]

[0264] The calculation formula of IG is:

[0265]

[0266] P(c i ) indicates category C i The probability of occurrence in the training set; P(x) is the probability of feature x occurring, that is, the probability of documents containing x in the training set; and P(c|x) indicates the probability of a document not containing feature x in the training set; P(c|x) indicates that the document is of category C when feature x appears i The probability of category C i The number of documents that contain feature x divided by the total number of documents that contain feature x in the training set; It is the number of documents in category Ci that do not contain feature t divided by the number of documents in the training set that do not contain feature xx.

[0267] Information gain predicts the category of a document by counting the number of times a certain feature value appears or does not appear in a certain article. The greater the information gain of a feature, the greater its contribution to classification. The disadvantage of information gain is that it takes into account the situation where the feature does not occur, and does not contribute much to judging the text category, and introduces unnecessary interference, especially when the category distribution and feature value distribution are highly unbalanced. If the vast majority of classes are negative classes, the vast majority of features do not appear, and the function result is determined by the feature values ​​that do not appear, then the effect of the information gain algorithm will be greatly reduced.

[0268] 2) Mutual Information

[0269] Mutual information is a concept in information theory, which is used to measure the degree of mutual dependence between two signals in a message. In the field of feature selection, people often use it to calculate the degree of dependence between feature x and category c, and combine the mutual information of feature x and each category as the weight of the feature value. The mutual information MI(t,c) between feature x and category c is defined as follows:

[0270]

[0271] The approximate calculation formula is:

[0272]

[0273] Among them: E is the number of times feature x and category c appear at the same time; B is the number of times feature x appears but category c does not appear; F is the number of times category c appears but feature x does not appear; N is the total number of documents.

[0274] When x and c are unrelated, that is, x and c are independent in distribution, then MI(x,c) = 0; when the appearance of a feature item depends on a certain category, the MI with that category will be large; when a feature rarely appears in a certain category, their mutual information is negative, that is, negatively correlated. When there are multiple categories, the mutual information of x for c can be calculated using the following formula:

[0275]

[0276] Mutual information is more inclined to consider rare feature words when performing feature selection. However, since rare words may contain less category information than more frequently occurring feature words, the classification performance may be poor when using the mutual information method to select feature sets for text classification.

[0277] 3)x 2 Statistics

[0278] x 2 (CHI-square) statistics are used to measure the lack of independence between features and categories, that is, x 2 The larger the value, the smaller the independence between the two and the greater the correlation. It assumes that the relationship between feature item x and category c conforms to the first-order degree of freedom x 2 Distribution, the original hypothesis is that x and c are unrelated, then the higher the calculated CHI value, the more you should choose the alternative hypothesis, that is, the greater the correlation, the greater the category information. The CHI value of feature item x for category c can be calculated by the following formula:

[0279]

[0280] Among them, A is the co-occurrence frequency of feature x and category c, B is the number of documents in the text set that contain feature x and do not belong to category c. C is the number of documents in category c that do not contain feature x, D is the number of documents that neither contain feature x nor belong to category c, and N is the total number of documents. 2 The advantage of statistics is that it takes into account the existence and non-existence of feature items at the same time, so it is better than mutual information in terms of information representation. However, since it counts whether feature x appears in the document and does not pay attention to its word frequency, it exaggerates the role of low-frequency words and has no obvious effect on the classification of low-frequency words.

[0281] 4) Expected Cross Entropy (ECE)

[0282] The expected cross entropy reflects the probability distribution of text categories and the distance between the probabilities of text categories in a certain situation. The larger the expected cross entropy of feature x, the greater the impact on the text classification distribution.

[0283] The expected cross entropy calculation formula is as follows:

[0284]

[0285] Among them, P(c i |x) means that when word x appears in the text, the text belongs to category c i The probability of P(c i ) is of category c i The probability of occurrence; P(c i |x) value is large, the category and feature are strongly correlated. If P(c i ) is small, then the feature has a greater effect on classification. However, the disadvantage of ECE is that it does not consider the case where the text feature does not appear.

[0286] 3.3 Feature extraction algorithm based on dependency measurement

[0287] The Hibert-Schmidt (HISC) dependency criterion in dependency measurement is often used as a key method to evaluate the correlation between metric features and categories. Its core idea is that a good feature should maximize this correlation. Based on this, the feature selection problem can be viewed as a combinatorial optimization problem:

[0288] T0=argmaxJ(o),st|Y|≤z(3.3-1)

[0289] Among them, n is the upper limit of the number of selected features, x is the set of selected features, and J(x) is the evaluation criterion. This type of feature extraction algorithm also often uses statistically related coefficients, such as Pearson correlation coefficient, probability error, Fisher score, linear determinability analysis, least squares regression error, etc. to express the separability of features for categories.

[0290] (1) Feature extraction algorithm based on consistency measurement

[0291] Given two text samples, if their feature values ​​are the same but they belong to different categories, they are said to be inconsistent; otherwise, they are said to be consistent. The consistency metric uses inconsistency to measure. Its purpose is not to maximize the separability of the class, but to try to retain the recognition ability of the original features, that is, to find the smallest subset with the same category distinction ability as the full set. It has the advantages of monotony, speed, and redundancy removal. However, it is sensitive to noisy data. Typical algorithms in this regard include LVF, Focus, etc.

[0292] The LVF algorithm is a method that uses probability to select features. It first randomly selects a feature subset and then uses the consistency criterion to evaluate it to achieve feature selection. The core of the LVF algorithm is the inconsistency criterion. In specific applications, a suitable inconsistency rate δ is manually set to make the discriminant function: J(o)≤δ hold.

[0293] The specific process of the LVF algorithm is as follows:

[0294] 1) In the initial stage, a maximum number of selections K and inconsistency rate δ are specified, and the initial subset Z is set;

[0295] 2) Randomly select a candidate subset Z' in X;

[0296] 3) If the number of features contained in the candidate subset Z' is less than the number of features contained in Z, proceed to the next step, otherwise return to (2);

[0297] 4) Calculate the inconsistency rate. If J(o)≤δ, replace the best subset;

[0298] 5) Determine whether the number of cycles reaches the threshold, if so, end the process, otherwise return to (2);

[0299] In each cycle of the LVF algorithm, a subset is randomly selected from the feature set X and whether the inconsistency condition is met. It does not traverse all features by randomly generating candidate sets and can only reduce the error rate by increasing the number of iterations.

[0300] 4. Text vector representation

[0301] In order for the quality problem information text to be understood by the computer, it must be vectorized and represented in digital form. This is essentially a process of converting unstructured natural language text into structured vector text. After the concept of information retrieval was proposed, a series of text representation models based on documents and queries emerged, with the representative vector space model. These models start from different perspectives and use different methods to deal with issues such as feature weighting and similarity calculation.

[0302] 4.1 Vector Space Model

[0303] The vector space model (VSM) represents documents in the form of vectors. Currently, this model is the main model for text representation. The classic vector space model was proposed by Salton et al. in the late 1960s and successfully applied to the famous SMART system. With the expansion of information, it has been widely used in information retrieval, automatic indexing, and text classification. A basic assumption of the vector space model is that the category described in the text is only related to the frequency of the term in the document, but not to the position and order of these words or phrases in the text. One of its basic ideas is to represent the text with a bag of words (BOW), taking each term as one dimension of the feature space, treating the text as a vector in the feature space, and then using the modulus or angle between two vectors to measure the similarity between two texts.

[0304] The more important concepts in the vector space model include: feature items, text, feature item weights, text similarity, etc.

[0305] Document: refers to general text or text fragments. It is usually pre-processed plain text. In the following text, the term "text" is used to refer to plain text objects, and no distinction is made between documents and texts.

[0306] Feature term: The content of a text is often represented by the basic language units (characters, words, phrases, etc.) it contains, that is, d = {x1, x2…x n}, these basic language units are collectively referred to as the characteristic items of text.

[0307] TermWeight: For a text containing n terms, d = {x1, x2…x n}, these n feature items are after feature extraction and dimensionality reduction. These feature items are often given a certain weight w to indicate their importance in the text d, that is, d = {x1, w1; x2, w2; ... x n ,w n ;}, abbreviated as d = {w1,w2…w n}.

[0308] Text similarity: The degree of relevance between two texts d1 and d2 is often measured by their similarity Sim(d1, d2). When the text is represented as a vector space model, the distance between the vectors can be used to represent their similarity. The commonly used similarity calculation formulas are:

[0309] (1) Inner product between vectors

[0310]

[0311] (2) Cosine of the vector angle

[0312]

[0313] The characteristic of the vector space model is that it ignores the mutual order relationship between feature items, which brings advantages in representation, calculation and processing, but it also inevitably loses a lot of information about text structure and semantics.

[0314] 4.2 Weight calculation of feature items

[0315] The weight calculation of feature items is the most important part of the vector space model. When assigning weights to each feature item, the weight of the feature item that can represent the category information in the text should be larger. However, it should be avoided that when a strong feature appears, other features are completely submerged by it, which will cause an abnormal amplification of information. The simplest method for calculating the weight of feature items is Boolean weight, that is, the feature item appears as 1 otherwise it is 0. This method is simple and easy to use, but it cannot reflect the importance of each feature item in the document. Another method is for experts or users to assign weights manually based on their own experience and domain knowledge. This method is more arbitrary and inefficient. However, with the increase in the dimension of text information and the diversification of text content, this method has been less used, but it is still used in special fields. The third method is to compare the feature item with other features and texts in all categories to obtain its own weight. This type of method is currently the most studied and applied method, and its representative is the TF-IDF algorithm. The so-called TF-IDF algorithm is based on the following assumption: the most meaningful words for distinguishing documents should be those that appear frequently in the document but less frequently in other documents in the entire document collection. Therefore, if the feature space coordinate system takes TF (TermFrequency) word frequency as a measure, the characteristics of similar texts can be reflected. In addition, considering the ability of words to distinguish different categories, the TF-IDF method believes that the smaller the frequency of a word in a text, the greater its ability to distinguish texts of different categories. Therefore, the concept of inverse document frequency IDF (InverseDocumentFrequency) is introduced, and the product of TF and IDF is used as the value measure of the feature space coordinate system, and it is used to adjust the weight TF. The purpose of adjusting the weight is to highlight important words and suppress minor words. For a word ix in a specific file, its importance can be expressed as:

[0316]

[0317] In the above formula, n i,jIndicates that the feature word x i In text d i The denominator of the number of occurrences is the sum of the number of feature words. Inverse document frequency is a measure of the general importance of a word. The IDF of a feature word can be obtained by dividing the total number of documents containing the word by the sum of the number of documents containing the word and then taking the logarithm of the quotient, as follows:

[0318]

[0319] Where |D| is the total number of texts in the corpus, |j:x i ∈d j | represents the total number of texts containing the feature word ix. However, since the frequency of a word may be 0 in a certain article after feature extraction, the denominator is usually 1+|j:x. i ∈d j | instead. So the TF-IDF weight formula is:

[0320] W i,j =tf i,j idf i (4.1-4)

[0321] Usually, in order to eliminate the influence of large differences in weights in documents and other factors, the vectors are often normalized, as shown in the following formula:

[0322]

[0323] In essence, TF-IDF is a weighted method that attempts to suppress noise, and simply assumes that words with low text frequency are more important, and words with high text frequency are more useless. Obviously, this is not entirely correct. Moreover, the simple structure of TF-IDF cannot effectively reflect the importance of words and the distribution of feature words, making it unable to complete the function of weight adjustment well, so the accuracy of TF-IDF method is not very high. In addition, the position information of words is not reflected in the TFIDF algorithm.

[0324] 5. Feature extraction algorithm implementation

[0325] When constructing a vector space model of text, the most intuitive method is to use words as its feature dimension, because words are the smallest meaningful language components that can act independently. However, for Chinese text, this simplest method has a problem that needs to be overcome: like many non-English languages, such as Japanese and Korean, Chinese sentences are often expressed as a string, in which there is no obvious separation symbol between words. Therefore, if you want to construct a vector space model of Chinese text, you first need to perform word segmentation to divide the sentence into smaller meaningful language units: words.

[0326] At present, practical Chinese word segmentation methods can be roughly divided into three categories: dictionary word segmentation, statistical word segmentation and mixed word segmentation.

[0327] (1) Dictionary word segmentation

[0328] The basic idea of ​​the dictionary segmentation method is to match the string to be analyzed with the entries in a "sufficiently large" segmentation dictionary according to a certain strategy. If a string is found in the segmentation dictionary, a word is identified.

[0329] According to the different scanning directions, the string matching word segmentation method can be divided into forward matching (from left to right) and reverse matching (from right to left); according to different length priority matching situations, it can be divided into maximum (longest) matching and minimum (shortest) matching.

[0330] Common dictionary-based methods are:

[0331] ①Forward Maximum Matching Method

[0332] This method is usually referred to as the MM method. In the MM method, the string is processed from left to right, and the algorithm is shown in Table 1 below.

[0333] Table 1 Forward maximum matching method

[0334]

[0335]

[0336] ②Reverse Maximum Matching Method

[0337] This method is usually referred to as RMM. The basic principle of RMM is the same as that of MM, but the difference is the direction of word segmentation, which is from right to left. The algorithm is shown in Table 2.

[0338] Table 2 Reverse maximum matching method

[0339]

[0340] ③ Minimum vocabulary

[0341] Using the above two word segmentation rules will often produce different results when processing the same sentence, and the minimum word method will choose the scenario with the least result words.

[0342] (2) Statistical word segmentation

[0343] The statistical word segmentation method does not rely on the word segmentation dictionary, but uses the word frequency information in the corpus to segment Chinese text. If we look at it from a formal perspective, a word is a stable combination of characters. Therefore, in the context, the more times adjacent characters appear at the same time, the more likely it is to form a word. Therefore, the frequency or probability of adjacent co-occurrence of characters can better reflect the credibility of the word. The frequency of the combination of adjacent co-occurring characters in the corpus can be counted to calculate their mutual occurrence information. Define the mutual occurrence information of two characters and calculate the adjacent co-occurrence probability of two Chinese characters X and Y. The mutual occurrence information reflects the closeness of the combination relationship between Chinese characters. When the closeness is higher than a certain threshold, it can be considered that this word group may constitute a word. This method only needs to count the frequency of word groups in the corpus, and does not require a word segmentation dictionary, so it is also called a dictionary-free word segmentation method or a statistical word extraction method. However, this method also has certain limitations. It often extracts some common word groups with high co-occurrence frequency but not words, such as "this one", "one of", "some", "my", "many", etc., and the recognition accuracy of common words is poor, and the time and space overhead is large.

[0344] (3) Mixed word segmentation

[0345] The hybrid method is a method that combines dictionary segmentation and statistical segmentation. The hybrid method uses a basic segmentation dictionary (common word dictionary) for string matching segmentation, and uses statistical methods to identify some new words, that is, combining string frequency statistics with string matching. It not only takes advantage of the fast speed and high efficiency of matching segmentation, but also takes advantage of the advantages of dictionary-free segmentation in combining context to identify new words and automatically eliminate ambiguity. The hybrid method takes advantage of the advantages of dictionaries and statistics and has been widely used in Chinese word segmentation software.

[0346] 6. Chinese text similarity calculation

[0347] Natural Language Processing

[0348] This solution uses mathematical models to calculate the similarity of Chinese texts and analyzes and processes natural language from a data perspective.

[0349] Feature extraction

[0350] This solution directly searches for identical strings without using a word segmentation program, and calculates the similarity based on this.

[0351] Representation Model

[0352] After obtaining the feature points of Chinese text, it is necessary to express them as a mathematical model that can be recognized by computers. The most commonly used one is the vector space model (VSM), which is expressed as

[0353] D=(W1,W2,…W n )

[0354] Where D represents the text, each dimension represents an independent feature point, and W i Indicates the value of the text at the feature point i. At present, the vector space model has been widely used in various fields such as information retrieval, text classification, text clustering, text mining, etc. that require mathematical models to represent text.

[0355] Feature weights

[0356] After determining the feature point i, in the vector space model, it is also necessary to determine the value W of the feature point i In the field of information processing, TFIDF is often used to represent feature weights.

[0357] TFIDF is a technique commonly used in information retrieval and data mining to calculate weights. TFIDF is a statistical method used to assess the importance of a word to a document in a document set. Term frequency (TF) refers to the number of times a given word appears in the document. This number is usually normalized to prevent it from being biased towards long documents. (The same word may have a higher term frequency in a long document than in a short document, regardless of whether the word is important or not.) For the TF of a word in a particular document, its importance can be expressed as:

[0358] tf i,j =n i,j / ∑ k n k,j , n i,j Indicates the number of times word i appears in j

[0359] Inverse Document Frequency (IDF) is a measure of the general importance of a word. The IDF of a particular word can be calculated by dividing the total number of documents by the number of documents containing the word and taking the logarithm of the quotient:

[0360] idf i =log(|D| / {d:t i ∈d}), the denominator contains the number of articles representing word i

[0361] TFIDF, on the other hand, comprehensively balances the importance and specificity of keywords:

[0362] tfidf i,j =tf i,j *idf i

[0363] A high word frequency within a particular document, and a low document frequency of that word in the entire document collection, can produce a high-weighted TFIDF. Therefore, TFIDF tends to filter out common words and retain important words.

[0364] If the total number of words in a document is 80, and the word "Chinese word segmentation" appears 10 times, then the TF of "Chinese word segmentation" in the document is 10 / 80=0.125. At the same time, if "Chinese word segmentation" appears in 100 documents and the total number of documents is 200, its IDF is

[0365] log(200 / 100) = 0.301. Finally, TFIDF = TF*IDF = 0.125*0.301 = 0.038.

[0366] Similarity calculation

[0367] By calculating the similarity of the representation models representing Chinese texts, we can get the similarity of Chinese texts. Similarity can be measured by similarity, which is a value between 0 and 1, with 1 indicating complete similarity and 0 indicating no correlation (of course, for different formulas, the value direction may also be different, for example, in cosine similarity, 0 indicates complete similarity and 1 indicates no correlation).

[0368] To calculate the similarity of the vector space model, you can use formulas such as cosine similarity, Jaccard similarity, and Euclidean distance similarity.

[0369] On the basis of the similarity intelligent analysis model, further software engineering practice is carried out to carry out demand development, software design, software development, software testing, etc., to build a similarity intelligent system software module that can be integrated into the equipment quality problem information system, to achieve data processing and fusion for quality problem information, intelligent understanding of semi-structured text of quality problems, and similar intelligent analysis of quality problems.

[0370] 7. Similarity module construction

[0371] 7.1 System Architecture

[0372] This system consists of four parts: quality problem information index library construction module, quality problem information intelligent retrieval module, quality problem information similarity calculation module and judgment module. The system architecture is as follows Figure 3 shown.

[0373] In the information retrieval and duplicate analysis system based on intelligent algorithms, all quality problem data in the database are first segmented, unregistered words are identified, keywords are extracted, and a knowledge representation model is constructed. A quality problem information knowledge index and a quality problem information retrieval index are constructed based on the model. Then, when the system receives a quality problem information similarity analysis request, the knowledge representation of the items to be tested and other items is obtained from the quality problem information knowledge index. The knowledge representation of the quality problem information to be tested (newly reported) can also be obtained from the quality problem information retrieval index through the quality problem information intelligent retrieval module. The similarity calculation module then calculates their similarity values ​​one by one. Finally, the similarity determination module in the system analyzes the results and returns the final set of similar items. In order to better understand this system, the details of the interaction implementation of each module of the entire system are as follows. Figure 4 shown.

[0374] 7.2 Project Index Library Building Module

[0375] The system first obtains all quality problem information from the database, including both approved and unapproved projects; then indexes these project applications and adds them to the project index library. The processing flow of the project index library construction module is as follows:

[0376] (1) Read the project's unique identifier ID value, application type, application year, project establishment status, project name, main research content and other field information from the information management system database;

[0377] (2) Then, the project name and main research content in the project field information are respectively identified based on unknown words, extracted based on keywords, and established based on project knowledge representation;

[0378] (3) Establish an index for the above knowledge representation model based on Apache Lucene and add it to the corresponding index library (project knowledge index library, project retrieval index library) according to the index requirements.

[0379] The index library construction process is as follows Figure 5 shown.

[0380] The index library of this design can be divided into three categories: statistics, general and search. The statistics index library is used to detect the similarity of projects, the general index library is used to check the specific word segmentation of projects, and the search index library (retrieval index library) is used for intelligent retrieval of projects to obtain related projects for horizontal duplication checking. The file tree structure of the project index library is as follows: Figure 6 shown.

[0381] The index file structure diagram is described as follows:

[0382] (1) The nodes in the second layer represent the project index library. The statistics index library stores the statistical values of unregistered words and keywords; the general index library stores the word segmentation situation of project application forms; the search index library stores project retrieval information.

[0383] (2) The statistics index file has 8 fields: the id field, which is the unique identifier of the project; the type field, which is the application type of the project; the year field, which is the application year of the project; the state field, which is the project establishment situation; the subject field, which is the location information of the project, including the project name and the main research content of the project; the flag field, which identifies whether the word is an unregistered word or a general keyword; the word field, which is the keyword or unregistered word of the project; the times field, which is the weight of the keyword or unregistered word of the project.

[0384] (3) The general index file has 2 fields: the id field, which is the unique identifier of the project; the word field, which is the word segmentation situation of the project application form.

[0385] (4) The search index file has 3 fields: the id field, which is the unique identifier of the project; the projectname field, which is the keyword of the project name;

[0386] the projectdetail field, which is the keyword of the main research content of the project.

[0387] 7.3 Intelligent Retrieval Module for Quality Problem Information

[0388] The intelligent retrieval module for quality problem information is based on the established project retrieval index library. The main function of this module is to match the most relevant projects in the retrieval index library according to the query keywords input by the user and return the information of these projects, which is convenient for the user to view and check the similarity of these relevant projects, so as to improve the efficiency of project similarity calculation.

[0389] The specific process of this module is as follows:

[0390] (1) The user inputs a query text for retrieval.

[0391] (2) Based on the Chinese Analyzer of Apache Lucene, the query text is segmented.

[0392] (3) Stop word processing is performed on the result obtained in step (2), such as filtering out words like "de" (of), "ne" (呢), etc., to obtain a preliminary set of query keywords, thereby improving the accuracy and efficiency of retrieval.

[0393] (4) Based on step (3), semantic expansion is performed on the words in the keyword set to obtain their corresponding synonyms and near-synonyms, thereby forming the final query keyword set.

[0394] (5) Calculate the relevance between the keyword set and the items in the item retrieval index library based on the relevance function.

[0395] (6) The project list is returned to the user in descending order of relevance, and the user can perform a horizontal check for duplicates on this batch of related projects.

[0396] The process corresponding to the above steps is as follows Figure 7 shown.

[0397] 7.4 Quality Problem Information Similarity Calculation Module

[0398] The main process of the similarity calculation module is as follows:

[0399] (1) Based on the user's similarity analysis request or the project intelligent retrieval request, the project to be checked for duplicates is obtained.

[0400] (2) Filter the projects that have been approved in the index library according to the application type of the quality problem information to be checked.

[0401] (3) The quality problem information to be checked for duplicates (newly reported) and the reported quality problem information obtained by filtering are calculated based on the project similarity method to obtain the similarity value of each project.

[0402] The process of the quality problem information similarity calculation module is as follows: Figure 8 shown.

[0403] 7.5 Judgment Module

[0404] The main task of the judgment module is to analyze the set of similarity values ​​obtained according to a specific threshold after the information to be checked for duplicates passes through the similarity calculation module, and then give the project similarity result and return a list of similar quality problem information to the user.

[0405] The process of the judgment module includes the following steps:

[0406] (1) Accept the similarity of each question information after being processed by the project similarity calculation module.

[0407] (2) Compare the similarity value with the specified threshold. If it is greater than the threshold, add the corresponding project ID and field information to the corresponding list.

[0408] (3) Sort the final list and return it to the user.

[0409] (4) Users can view similar project information, and we will highlight the parts with the same keywords as the project to be checked for duplicates.

[0410] The main process of the judgment module is as follows Fig. 9 shown.

[0411] 8. Problem prediction software model construction

[0412] The prediction model conducts predictive analysis based on key information and similarity results to predict potential quality problems in the future and realize the intelligent prediction function of related problem data.

[0413] Predictive analytics is a statistical or data mining solution that includes algorithms and techniques that can be used on structured and unstructured data to determine future outcomes. It can be deployed for many other purposes such as prediction, optimization, forecasting and simulation, and can also provide a variety of information for planning processes.

[0414] Exponential smoothing was proposed by Brown. He believed that the trend of time series is stable or regular, so the time series can be reasonably extended. He believed that the recent past trend will continue in the future to some extent, so he puts a larger weight on the recent data.

[0415] Basic principle: Exponential smoothing is a type of moving average method. Its characteristic is that it gives different weights to past observations, that is, the weights of recent observations are larger than those of distant observations. According to the number of smoothing times, exponential smoothing is divided into single exponential smoothing, double exponential smoothing, and triple exponential smoothing. But their basic ideas are: the predicted value is the weighted sum of previous observations, and different weights are given to different data, with new data given larger weights and old data given smaller weights.

[0416] According to the number of smoothing times, exponential smoothing is divided into single exponential smoothing, double exponential smoothing and triple exponential smoothing.

[0417] Exponential smoothing forecast

[0418] When there is no obvious trend change in the time series, an exponential smoothing forecast can be used.

[0419] The prediction formula is: yt+1'=ayt+(1-a)yt', where yt+1' is the predicted value of period t+1, i.e. the smoothed value St of the current period (period t); yt is the actual value of period t; yt' is the predicted value of period t, i.e. the smoothed value St-1 of the previous period.

[0420] This formula can also be written as: yt+1'=yt'+a(yt-yt'). It can be seen that the next period forecast value is the sum of the current period forecast value and the current period actual value with a discount and the forecast value error.

[0421] Quadratic Exponential Smoothing Forecast

[0422] Secondary exponential smoothing is a smoothing of primary exponential smoothing. It is suitable for time series with linear trends. Its prediction formula is:

[0423] yt+m=(2+am / (1-a))yt'-(1+am / (1-a))yt=(2yt'-yt)+m(yt'-yt)a / (1-a)In the formula, yt=ayt-1'+(1-a)yt-1Obviously, quadratic exponential smoothing is a straight line equation with an intercept of (2yt'-yt) and a slope of (yt'-yt)a / (1-a), and the independent variable is the number of predicted days.

[0424] The basic formula of quadratic exponential smoothing is St = αSt + (1-α)St-1 Yt+T = at + btT at = 2St-St bt = (α / 1-α)(St-St).

[0425] St--the exponential smoothing value of the tth period;

[0426] St-1--the second exponential smoothing value of the tth period;

[0427] α--smoothing coefficient;

[0428] Yt+T--the predicted value of the t+T period;

[0429] T--number of periods moving backward from period t

[0430] Triple exponential smoothing forecast

[0431] The triple exponential smoothing forecast is a smoothing based on the double exponential smoothing.

[0432] The prediction formula is: yt+m=(3yt'-3yt+yt)+[(6-5a)yt'-(10-8a)yt+(4-3a)yt]*am / 2(1-a)2+(yt'-2yt+yt')*a2m2 / 2(1-a)2 where yt=ayt-1+(1-a)yt-1

[0433] Their basic idea is that the predicted value is the weighted sum of previous observations, and different weights are given to different data, with larger weights given to new data and smaller weights given to old data.

[0434] In order to better understand the present invention, the above is described in detail in conjunction with the specific embodiments of the present invention, but it is not intended to limit the present invention. Any simple modifications made to the above embodiments based on the technical essence of the present invention still fall within the scope of the technical solution of the present invention. Each embodiment in this specification focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referenced to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A quality problem information intelligent matching method based on text understanding, including data preprocessing, characterized in that: The following steps are also included: Step 1: Perform data dimension reduction on the preprocessed sample data; Step 2: performing feature extraction, wherein the feature extraction includes feature extraction based on distance measurement, feature extraction based on information measurement, feature extraction based on dependency measurement, and feature extraction based on consistency measurement; Step 3: Use the vector space model to vectorize the quality problem information; Step 4: Construct feature extraction algorithm; Step 5: Calculate the similarity of Chinese texts; Step 6: Construct similarity calculation module; Step 7: Build a problem prediction software model and perform intelligent matching of quality problem information.

2. The quality problem information intelligent matching method based on text understanding according to claim 1 is characterized in that: The data dimension reduction is expressed as 1) It represents a sample set in the common dimensional space. represents a data set in the space d = (d < < D), where N is the sample size, i is the i-th sample, and D is the number of features; 2) Dimensionality reduction mapping, M:X→Y, x→y=M(x), is called the low-dimensional representation of x as y.

3. The quality problem information intelligent matching method based on text understanding as claimed in claim 2, characterized in that: The step 1 includes principal component analysis and independent component analysis. The principal component analysis includes: for a given data set Among them, x i ∈R N is an N-dimensional column vector, M represents the number of samples; Assume that the given data has been zeroed, that is: The covariance matrix is ​​defined as: Since C is a semi-positive definite matrix, it is diagonalized, that is, C = V∧V T ,in, ∧=diag(λ1,λ2,…λ n ),VV T =1; Assuming that the rank of the semi-positive definite matrix C is d, that is, rank(C) = d, then the semi-positive definite matrix C has d non-zero eigenvalues, denoted by λ1≥λ2≥…λ d >0; Select the eigenvectors v1≥v2≥…v corresponding to the first k largest eigenvalues k The eigenvector matrix V = {v1≥v2≥…v k }; The sample data X is represented by the eigenvector matrix V.

4. The quality problem information intelligent matching method based on text understanding as claimed in claim 3 is characterized in that: The independent component analysis includes: Assume that the data S∈R consists of S independent elements n , we observed data x, x = AS, Where A is the unknown mixing matrix and x is the observed data {x i ; i = 1, 2, ..., m}, a total of m samples, Assume that each signal source S i The distribution of is given by the density function p S Given, we get the joint distribution function of the signal S: W is the pseudo-inverse matrix of A, and the joint distribution function of the signal x is obtained: For a random variable z, its cumulative distribution function is defined as Then the probability density function of z is derived from the cumulative distribution function, that is, p z (z) = F'(z); The rectangular matrix W is the parameter in the model. Given a training set {x i ; i=1,2,…,m}, the log-likelihood estimate of the sample is as follows: According to the gradient descent algorithm, Derive the random skill increase learning rule, for a training set x (i) , the update rule is as follows, where a is the learning rate; By calculating S i =Wx (i) Restore the original signal.

5. The quality problem information intelligent matching method based on text understanding as claimed in claim 4 is characterized in that: The Relief algorithm is used to implement the feature extraction based on the distance metric, and the hypothesis interval is defined as the maximum distance that the decision surface can move while keeping the sample classification unchanged, expressed as: Among them, M(d) and H(d) are the nearest neighbor points of the same type and different type of d respectively; Assume D = {d1, d2, ... d n } is the text set to be classified, d i ={x1,x2,…x n } corresponds to its i-th document, x j represents the jth eigenvalue; Randomly select m samples from all training sets, calculate the hypothesis interval of the samples on each feature, and use the accumulated value as the weight of the feature. Sample d updates the feature x j The weight can be expressed as: Since text features are discrete values, the function diff is defined as:

6. The quality problem information intelligent matching method based on text understanding according to claim 5 is characterized in that: Using information gain, mutual information, x 2 The statistics and expected cross entropy measure the information metric; The information gain is used to calculate the weight of the feature in the decision tree. The feature is the difference between the entropy without considering the feature and the entropy after considering the feature. The information entropy is defined as follows: The calculation formula of IG is: Among them, P(c i ) indicates category C i The probability of appearing in the training set, P(x) is the probability of feature x appearing, P(c|x) represents the probability of a document not containing feature x in the training set, and P(c|x) represents the category C when feature x appears. i The probability of Category C i The number of documents in the training set that do not have feature t divided by the number of documents in the training set that do not have feature x.

7. The quality problem information intelligent matching method based on text understanding according to claim 6 is characterized in that: The mutual information is used to measure the mutual dependence between two signals in a message, and is used to calculate the dependence between feature x and category c, and the mutual information between feature x and each category is integrated as the weight of the feature value; The mutual information MI(t,c) between feature x and category c is defined as follows: The approximate calculation formula is: Among them, E is the number of times feature x and category c appear at the same time, B is the number of times feature x appears but category c does not appear, F is the number of times category c appears but feature x does not appear, and N is the total number of documents; When there are multiple categories, the mutual information of x for c is calculated as follows:

8. The quality problem information intelligent matching method based on text understanding according to claim 7 is characterized in that: The x 2 The statistic is used to measure the lack of independence between feature items and categories and calculate the CHI value of feature item x for category c. The formula is: Where D is the number of documents that neither contain feature x nor belong to category c, and N is the total number of documents.

9. The quality problem information intelligent matching method based on text understanding according to claim 8, characterized in that: The calculation formula of the expected cross entropy is: Among them, P(c i |x) means that when word x appears in the text, the text belongs to category c i The probability, P(c i ) is of category c i Probability of occurrence.

10. A quality problem information intelligent matching system based on text understanding, including an index library construction module, characterized in that: It also includes an intelligent retrieval module, a similarity calculation module and a judgment module. Therefore, the index library construction module is used to obtain all quality problem information from the database, including both approved and unapproved types; The intelligent search module is used to match the most relevant items in the search index library according to the query keywords input by the user and return the information of these items; The determination module is used to analyze the set of similarity values ​​obtained according to a specific threshold after the information to be checked for duplicates passes through the similarity calculation module, and then give the project similarity result and return the list of similar quality problem information to the user; The system adopts the method as claimed in claim 1 to realize intelligent matching of quality problem information based on text understanding.

Images