Data preprocessing method after text data classification

Abstract

The invention relates to a data preprocessing method after text data classification, which belongs to the technical field of data processing, and comprises the following steps: S1) collecting various text data to obtain a set of the various text data, and constructing a text data model; s2) finding out keywords of various types of text data in the text data model, taking respective keywords of the various types of text data as respective classification targets of the various types of text data, and then classifying the various types of text data in the text data model; s3) performing text filtering based on various types of text data classified in the step S2); wherein text filtering is carried out on various types of classified text data by adopting a line filtering method; s4) carrying out data clustering on the filtered text; the method has the beneficial effects that various types of classified text data lines are subjected to text filtering by adopting a filtering method, and the preprocessing of automatically classified big data is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to a data preprocessing method after text data classification. Background Art

[0002] Data preprocessing refers to some processing of data before the main data processing. For example, before converting or enhancing the acquired observation data, the irregular distribution network is first converted into a regular network through interpolation to facilitate computer calculation.

[0003] Data preprocessing is the process of cleaning and transforming raw data to make it suitable for analysis and modeling. Raw data often includes missing data, outliers, and other inconsistencies, such as formatting issues. Preprocessing steps include data cleaning, data normalization, and data transformation. The goal of data preprocessing is to improve the accuracy and efficiency of downstream analysis and modeling.

[0004] With the continuous progress of modern society, especially the rapid development of the Internet, the number of various network resources has shown the characteristics of huge quantity, wide variety, and rapid changes. The Internet has entered the era of big data. At present, in addition to the huge amount of big data in the Internet application environment, the proportion of unstructured data is increasing, and the number of resources is increasing linearly. Rapidly and accurately realizing the automatic classification of big data in the Internet application environment has become a hot topic in current data technology research. Preprocessing of big data after automatic classification is the basis for solving the problem of big data classification. Summary of the invention

[0005] The technical problem to be solved by the present invention is how to preprocess the big data after automatic classification. The present invention provides a data preprocessing method after text data classification, which classifies various types of text data in a text data model, and uses a filtering method to filter the classified various types of text data to achieve preprocessing of the big data after automatic classification.

[0006] In order to achieve the above object, the present invention is implemented by the following technical solutions:

[0007] A data preprocessing method after text data classification comprises the following steps:

[0008] Step S1): Collect various types of text data, obtain a collection of various types of text data, and construct a text data model;

[0009] Step S2): Find out the keywords of each type of text data in the text data model, use the keywords of each type of text data as the classification target of each type of text data, and then classify the various types of text data in the text data model; wherein the keyword stringing method is used to classify the various types of text data;

[0010] Step S3): performing text filtering on the various types of text data classified in step S2); wherein a line filtering method is used to perform text filtering on the various types of text data classified;

[0011] Step S4): cluster the filtered text.

[0012] Optionally, in step S1), a text data model is constructed by using the following steps:

[0013] Step 1): constructing a three-dimensional text data model; wherein the three-dimensional text data model is a cylindrical data model;

[0014] Step 2): On the text data model, separate the various types of collected text data, and place the various types of text data on the corresponding contour lines of the cylindrical data model.

[0015] Optionally, in step S2), keywords of various types of text data are found by using the following steps:

[0016] Step R1): The words with the most occurrences in each type of text data are used as keywords for each type of text data;

[0017] Step R2): Mark the keywords of each type of text data, and use the keywords of each type of text data as the classification targets of each type of text data;

[0018] Step R3): Classify the keywords of each type of text data to determine the type of each type of text data.

[0019] Optionally, in step S2), the keyword string signing method is the following steps:

[0020] Step Q1): dividing a plurality of contour lines on the cylindrical data model;

[0021] Step Q2): string corresponding keywords on each of the contour lines, and connect the keywords in series through each contour line.

[0022] Optionally, in step S2), the keyword string signing method adopts the following formula (1):

[0023] (1);

[0024] in, is the i-th contour line, Keywords on the i-th contour line, is the i-th contour line The corresponding keywords are , is the total number of contour lines, For the connecting line, Connect the contour lines. A data model representing a cylindrical shape;

[0025] or It represents that in the cylindrical data model, the keywords corresponding to each contour line are connected in series through connecting lines.

[0026] Optionally, in step S3), the line filtering method performs text filtering on the classified text data of each category, using the following steps:

[0027] Step 00): Filter the text data corresponding to each contour line;

[0028] Step 11): Scan and filter the text data on each contour line along the linear direction.

[0029] Optionally, the line filtering method performs text filtering on the classified text data of each category, using the following formula (2):

[0030] (2);

[0031] in, is a text data selected from different categories of text data, Representing different categories, is the text filtering function, Filter a text data selected from different categories of text data, is a contour line selected from different contour lines, and the linear direction of the selected contour line is along one direction, Represents different contour lines;

[0032] or Filter the text data on the selected contour line along one direction.

[0033] Optionally, in step S4), the various types of text data are merged.

[0034] Beneficial effects of the present invention:

[0035] The present invention constructs a text data model, finds out the keywords of various types of text data in the text data model, uses the keywords of each type of text data as the classification targets of each type of text data, and then classifies the various types of text data in the text data model, and uses a filtering method to perform text filtering on the classified types of text data. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 It is a work flow chart of the present invention;

[0038] Figure 2 It is a structural diagram of the text data model of the present invention. DETAILED DESCRIPTION

[0039] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0040] Example 1

[0041] like Figure 1-Figure 2 As shown, this implementation provides a data preprocessing method after text data classification, comprising the following steps:

[0042] Step S1): Collect various types of text data, obtain a collection of various types of text data, and construct a text data model;

[0043] Step S2): Find out the keywords of each type of text data in the text data model, use the keywords of each type of text data as the classification target of each type of text data, and then classify the various types of text data in the text data model; wherein the keyword stringing method is used to classify the various types of text data;

[0044] Step S3): performing text filtering on the various types of text data classified in step S2); wherein a line filtering method is used to perform text filtering on the various types of text data classified;

[0045] Step S4): cluster the filtered text.

[0046] Specifically, in step S1), the text data model is constructed by the following steps:

[0047] Step 1): constructing a three-dimensional text data model; wherein the three-dimensional text data model is a cylindrical data model;

[0048] Step 2): On the text data model, separate the various types of collected text data, and place the various types of text data on the corresponding contour lines of the cylindrical data model.

[0049] In step S2), keywords of various types of text data are found by using the following steps:

[0050] Step R1): The words with the most occurrences in each type of text data are used as keywords for each type of text data;

[0051] Step R2): Mark the keywords of each type of text data, and use the keywords of each type of text data as the classification targets of each type of text data;

[0052] Step R3): Classify the keywords of each type of text data to determine the type of each type of text data.

[0053] In step S2), the keyword string signing method is as follows:

[0054] Step Q1): dividing a plurality of contour lines on the cylindrical data model;

[0055] Step Q2): string corresponding keywords on each of the contour lines, and connect the keywords in series through each contour line.

[0056] In step S2), the keyword string signing method uses the following formula (1):

[0057] (1);

[0058] in, is the i-th contour line, Keywords on the i-th contour line, is the i-th contour line The corresponding keywords are , is the total number of contour lines, For the connecting line, Connect the contour lines. A data model representing a cylindrical shape;

[0059] or It represents that in the cylindrical data model, the keywords corresponding to each contour line are connected in series through connecting lines.

[0060] In step S3), the line filtering method performs text filtering on the classified text data of each category, using the following steps:

[0061] Step 00): Filter the text data corresponding to each contour line;

[0062] Step 11): Scan and filter the text data on each contour line along the linear direction.

[0063] In addition, the line filtering method is used to filter the classified text data using the following formula (2):

[0064] (2);

[0065] in, is a text data selected from different categories of text data, Representing different categories, is the text filtering function, Filter a text data selected from different categories of text data, is a contour line selected from different contour lines, and the linear direction of the selected contour line is along one direction, Represents different contour lines;

[0066] or Filter the text data on the selected contour line along one direction.

[0067] In step S4), various types of text data are merged, that is, various types of text data are merged and stored in one storage.

[0068] Example 2

[0069] like Figure 2 As shown, based on Example 1, the various types of text data are respectively Figure 2 Find the six cylinders in the six cylinders, find the keywords in each of the six cylinders, use the respective keywords as the classification targets of each type of text data, and then classify the various types of text data in the text data model.

[0070] Among them, the six cylinders are stringed together using the contour lines of the cylindrical data model, that is, each cylinder is stringed together using the contour lines.

[0071] The upper end line (connecting line) and the lower end line (connecting line) of the cylindrical data model connect the contour lines (vertical lines of the cylindrical data model) in series.

[0072] Example 3

[0073] like Figure 2 As shown, based on Example 1, the classified text data of various types are subjected to text filtering, which is to perform text filtering on the six cylinders on each contour line along a length direction of the contour line.

[0074] Only the text data on a certain contour line is selected for text filtering. Therefore, the corresponding text data on a certain contour line is selected for text filtering. There are corresponding categories ,In this way, text filtering is performed on the text data corresponding to a certain contour line.

[0075] Throughout the specification, references to "one embodiment," "an embodiment," "an example," or "an example" mean that a particular feature, structure, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment of the present invention. Therefore, the phrases "one embodiment," "an embodiment," "an example," or "an example" appearing in various places throughout the specification do not necessarily all refer to the same embodiment or example. In addition, particular features, structures, or characteristics may be combined in one or more embodiments or examples in any suitable combination and / or sub-combination. In addition, it will be appreciated by those of ordinary skill in the art that the figures provided herein are for illustrative purposes and that the figures are not necessarily drawn to scale. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0076] The terms "first" and "second" used in the present invention are only used to distinguish the corresponding components for the sake of clarity of description, and are not intended to limit any order or emphasize importance, etc. In addition, the term "connected" used in this article can be directly connected or indirectly connected via other components without special explanation.

[0077] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope of the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A data preprocessing method after text data classification, characterized in that: The steps include: Step S1): Collect various types of text data, obtain a collection of various types of text data, and construct a text data model; Step S2): Find out the keywords of each type of text data in the text data model, use the keywords of each type of text data as the classification target of each type of text data, and then classify the various types of text data in the text data model; wherein the keyword stringing method is used to classify the various types of text data; Step S3): performing text filtering on the various types of text data classified in step S2); wherein a line filtering method is used to perform text filtering on the various types of text data classified; Step S4): cluster the filtered text.

2. The data preprocessing method after text data classification according to claim 1 is characterized in that: In the step S1), the text data model is constructed by the following steps: Step 1): constructing a three-dimensional text data model; wherein the three-dimensional text data model is a cylindrical data model; Step 2): On the text data model, separate the various types of collected text data, and place the various types of text data on the corresponding contour lines of the cylindrical data model.

3. The data preprocessing method after text data classification according to claim 1 is characterized in that: In step S2), the keywords of the various types of text data are found by using the following steps: Step R1): The words with the most occurrences in each type of text data are used as keywords for each type of text data; Step R2): Mark the keywords of each type of text data, and use the keywords of each type of text data as the classification targets of each type of text data; Step R3): Classify the keywords of each type of text data to determine the type of each type of text data.

4. The data preprocessing method after text data classification according to claim 1 is characterized in that: In step S2), the keyword string signing method comprises the following steps: Step Q1): dividing a plurality of contour lines on the cylindrical data model; Step Q2): string corresponding keywords on each of the contour lines, and connect the keywords in series through each contour line.

5. The data preprocessing method after text data classification according to claim 1 is characterized in that: In step S2), the keyword string signing method adopts the following formula (1): (1)； in, is the i-th contour line, Keywords on the i-th contour line, is the i-th contour line The corresponding keywords are , is the total number of contour lines, For the connecting line, Connect the contour lines. A data model representing a cylindrical shape; or It represents that in the cylindrical data model, the keywords corresponding to each contour line are connected in series through connecting lines.

6. The data preprocessing method after text data classification according to claim 1 is characterized in that: In the step S3), the line filtering method performs text filtering on the classified text data of each category, using the following steps: Step 00): Filter the text data corresponding to each contour line; Step 11): Scan and filter the text data on each contour line along the linear direction.

7. A data preprocessing method after text data classification according to any one of claims 1 or 6, characterized in that: The line filtering method performs text filtering on the classified text data of each category, using the following formula (2): (2)； in, is a text data selected from different categories of text data, Representing different categories, is the text filtering function, Filter a text data selected from different categories of text data, is a contour line selected from different contour lines, and the linear direction of the selected contour line is along one direction, Represents different contour lines; or Filter the text data on the selected contour line along one direction.

8. The data preprocessing method after text data classification according to claim 1 is characterized in that: In the step S4), the various types of text data are merged.

Images