A data classification method, system and related device

Abstract

The present application discloses a data classification method, system and related devices for quickly and accurately classifying data. The present application method includes: obtaining data source information, extracting data from the data source information, and obtaining target data; obtaining a full subject word library and a department subject word library; judging whether the target data has a subject based on the full subject word library; if so, extracting the subject from the full subject word library; if not, classifying the target data through a preset learning model to obtain a data classification result; when the target data has a subject, judging whether the subject is unique; if so, taking the subject as the data classification result; if not, determining the department subject of the target data based on the department subject word library; when the subject is not unique, judging whether the subject overlaps with the department subject; if so, taking the subject or the department subject as the data classification result; if not, classifying the subject and the department subject through a preset learning model to obtain a data classification result.

Description

Technical Field

[0001] The present application relates to the field of big data information technology, and in particular to a data classification method, system and related devices. Background Art

[0002] With the development of information technology, the scale and application scenarios of big data in government departments have shown explosive growth. In the government system, the data sources are extensive and complex. The business systems of government departments at all levels, social public service platforms, and Internet of Things devices, etc., cover many sensitive and important areas such as citizens' personal information, urban planning and construction, and policy implementation. Different types of data vary greatly in importance, sensitivity, and usage rights. For example, the leakage of personal privacy data such as resident ID numbers and home addresses will seriously damage the rights and interests of citizens. Policy planning data that has not been released by government departments must also be strictly confidential. Therefore, it is urgent to classify and standardize government data, which will help government departments manage data assets, clarify management strategies for different types of data, improve data security, and prevent data leakage risks.

[0003] In the existing technology, data classification is mostly a manual operation. First, it is necessary to manually aggregate data from multiple data sources, such as collecting data from different government systems. Then, the data is preliminarily distinguished and labeled based on experience or simple predefined rules, and then detailed classification is performed based on this to finally obtain the data classification results.

[0004] However, manually-led data classification has many drawbacks. The entire classification process relies heavily on manual labor, which requires a lot of effort to organize when collecting data. The classification process is highly subjective and there is no unified classification standard. As a result, when faced with massive amounts of data, it is both time-consuming and labor-intensive and prone to errors. It is impossible to obtain accurate data classification results, which makes it difficult to meet the needs of government departments for fast and accurate data classification. Summary of the invention

[0005] In order to solve the above technical problems, the present application provides a data classification method, system and related devices.

[0006] The technical solution provided in this application is described below:

[0007] The first aspect of the present application provides a data classification method, the method comprising:

[0008] Acquire data source information, perform data extraction on the data source information, and obtain target data;

[0009] Obtain full subject thesaurus and departmental subject thesaurus;

[0010] Determining whether the target data has a subject based on the full subject word library;

[0011] If yes, extract the subject from the full subject word library;

[0012] If not, classify the target data using a preset learning model to obtain a data classification result;

[0013] When it is determined that the subject exists in the target data, determining whether the extracted subject is unique;

[0014] If yes, the subject is used as the data classification result;

[0015] If not, determining the departmental subject of the target data based on the departmental subject word library;

[0016] When it is determined that the subject is not unique, determining whether the subject overlaps with the department subject;

[0017] If yes, the subject or the department subject is used as the data classification result;

[0018] If not, the subject and the department subject are classified by the preset learning model to obtain the data classification result.

[0019] Optionally, the acquiring of data source information and performing data extraction on the data source information to obtain target data includes:

[0020] Acquire data source information, perform data processing on the data source information, and extract the data source information after data processing to obtain target data.

[0021] Optionally, the acquiring of data source information, performing data processing on the data source information, and extracting the processed data source information to obtain target data includes:

[0022] Obtain data source information, identify and remove duplicate data, redundant information and missing values ​​of the data source information;

[0023] Unifying the data format of the data source information from which the duplicate data, the redundant information, and the missing values ​​have been removed;

[0024] The data source information that has completed the data format unification is extracted, and the data source information is spliced ​​with space characters to obtain target data.

[0025] Optionally, judging whether the target data has a subject based on the full subject word library includes:

[0026] The target data is segmented, and the segmented target data is input into the full subject word library to determine whether there is a subject.

[0027] Optionally, determining the department subject of the target data based on the department subject word library includes:

[0028] Determining whether the target data has a department theme based on the department theme word library;

[0029] If so, determining the departmental subject of the target data;

[0030] If not, the subject is classified by a preset learning model to obtain a data classification result.

[0031] Optionally, the full subject word library includes primary categories, secondary categories and tertiary categories.

[0032] Optionally, the first-level category is the e-government knowledge domain and departmental functional division and industry category;

[0033] The secondary category and the tertiary category are the knowledge branches to which the subject words in the full subject word library belong.

[0034] A second aspect of the present application provides a data classification system, the system comprising:

[0035] A first extraction unit is used to obtain data source information, perform data extraction on the data source information, and obtain target data;

[0036] An acquisition unit is used to acquire the full subject word library and the departmental subject word library;

[0037] A first judging unit, configured to judge whether the target data has a subject based on the full subject word library;

[0038] A second extraction unit is used for extracting the subject from the full subject word library if yes;

[0039] A first classification unit, for classifying the target data by using a preset learning model to obtain a data classification result if no;

[0040] A second judging unit, configured to judge whether the extracted subject is unique when it is determined that the subject exists in the target data;

[0041] A second classification unit, for taking the subject as the data classification result if yes;

[0042] a second determining unit, for determining the department subject of the target data based on the department subject word library if no;

[0043] A third judgment unit, configured to judge whether the subject and the department subject overlap when it is determined that the subject is not unique;

[0044] A third classification unit, for taking the subject or the department subject as the data classification result;

[0045] The fourth classification unit is used to classify the subject and the department subject through the preset learning model to obtain the data classification result.

[0046] A third aspect of the present application provides a data classification device, the device comprising:

[0047] Processor, memory, input-output unit, and bus;

[0048] The processor is connected to the memory, the input and output unit, and the bus;

[0049] The memory stores a program, and the processor calls the program to execute any one of the methods.

[0050] A fourth aspect of the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed on a computer, the program executes the first aspect and any optional method in the first aspect.

[0051] It can be seen from the above technical solutions that this application has the following advantages:

[0052] This application automates the process of data collection and key information extraction, reduces manual intervention, and improves efficiency. Multiple judgments are made on the data through the full subject vocabulary and department subject vocabulary, providing a unified classification standard and basis for the entire data classification process, ensuring the consistency of the classification results. The learning model is introduced and used to classify data, which improves the accuracy of classification and ensures that the data is correctly divided into corresponding categories. The entire data classification method has the characteristics of automation and intelligence, realizes fast and efficient data classification, and meets the needs of government departments for rapid data classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solution in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 A schematic diagram of an embodiment of the data classification method provided in this application;

[0055] Figure 2 A flowchart of another embodiment of the data classification method provided by the present application;

[0056] Figure 3 A schematic diagram of the structure of an embodiment of the data classification system provided by the present application;

[0057] Figure 4 A schematic diagram of the structure of an embodiment of the data classification device provided in this application. DETAILED DESCRIPTION

[0058] The present application provides a data classification method, which can classify data quickly and accurately. It should be noted that the data classification method of the present application is applied to a terminal.

[0059] See also Figure 1 The present application first provides an embodiment of a data classification method, which includes:

[0060] S101, obtaining data source information, performing data extraction on the data source information, and obtaining target data;

[0061] In this embodiment, the data source information is the data source configured by the user according to the data classification requirements, and the configured data source information includes: data source library, data source department, data source system, data table name, all fields, etc. After obtaining the data source information, the key structure and key information in the data source are screened according to the pre-set extraction rules, and finally accurate and representative data are extracted from the key structure and key information, and the data is the target data.

[0062] This embodiment avoids unnecessary waste of resources and data redundancy by acquiring target data from the data source information configured by the user, helps to accurately extract target data, and improves the reliability of data acquisition. Extracting data source information can ensure that the target data meets the requirements of subsequent data classification, laying a good foundation for the entire data classification process.

[0063] S102, obtaining a full subject word library and a departmental subject word library;

[0064] In this embodiment, the source of the full thesaurus is the Comprehensive E-Government Thesaurus, which contains 20,252 thesaurus, including 17,421 formal thesaurus and 2,831 informal thesaurus. The full thesaurus obtained based on the Comprehensive E-Government Thesaurus includes primary, secondary and tertiary categories, where the primary category is the e-government knowledge field and the functional division of departments and industry categories, and the secondary and tertiary categories are the knowledge branches to which the thesaurus in the full thesaurus belongs; and the source of the departmental thesaurus is the corresponding government departments.

[0065] The acquisition of the full subject word library of this embodiment provides a comprehensive and universal semantic framework for the entire data processing process, which can cover a wide range of concepts and terms, so that accurate operations can be performed based on this framework in subsequent data classification. At the same time, the department subject word library is more targeted, reflecting the unique business needs and semantic habits of the corresponding government departments. In the process of data classification, when the full subject word library cannot well meet the needs of a specific government department, the department subject word library can be used as a supplement to improve the accuracy and flexibility of data processing. The combination of the full subject word library and the department subject word library can better meet the needs of data classification at different levels and in different departments.

[0066] S103, judging whether the target data has a subject based on the full subject word library;

[0067] In this embodiment, subject words are taken out from the full subject word library in order and matched with the target data. The matching method can be exact matching or fuzzy matching. Exact matching requires that the vocabulary in the target data is exactly the same as the subject words, while fuzzy matching can have a certain semantic similarity.

[0068] It should be noted that, in the process of traversing the entire subject thesaurus, if at least one subject word that successfully matches the target data is found, it is determined that the target data has a subject, and step 104 is executed; if no matching subject word is found after traversing the entire subject thesaurus, it is determined that the target data does not have a subject, and step 105 is executed.

[0069] This embodiment helps to improve the accuracy of data processing by comparing the target data with the full subject word library to determine whether the subject exists. By matching by using exact matching or fuzzy matching, various possible subject situations can be considered more comprehensively, reducing omissions and misjudgments.

[0070] S104, extracting topics from the full subject word library;

[0071] In this embodiment, when it is determined that the target data has a topic, the topic is extracted from the full subject word library. Before extraction, the topic that successfully matches the target data needs to be marked. Since the topic in the full subject word library has been compared with the target data when determining whether the target data has a topic, the successfully matched topic can be directly located based on the successfully matched records, and then the successfully matched topic can be extracted.

[0072] It should be noted that if there are multiple successfully matched topics, all of them need to be extracted. For example, when it is determined that there is a topic, continue to search for all other topics that match the target data in the full subject vocabulary, conduct a comprehensive search for the existing topic-related terms and their associated terms, perform a matching analysis on all searched related topics, and set a matching threshold. If the matching degree is greater than or equal to the threshold, the related topics will be extracted. For example, the matching threshold is set to 80%, and the target data contains "education policy" related content. The "education policy" topic can be matched in the full subject vocabulary, and the "education policy and policy" topic can also be matched. Through matching analysis, the matching degree of the "education policy and policy" topic and the "education policy" topic is 90%, so the "education policy and policy" topic and the "education policy" topic are related topics.

[0073] This embodiment extracts topics from the full subject word library, clarifies the association between the target data and the subject words in the subject word library, so that the target data has clear subject attributes. The extracted topics can provide important basis for subsequent steps, and the clear topics help determine the direction and focus of the analysis.

[0074] S105, classifying the target data through a preset learning model to obtain a data classification result;

[0075] In this embodiment, when it is determined that there is no subject in the target data, the preset learning model is called to classify the target data to obtain the data classification result. First, a suitable preset learning model is selected, such as a convolutional neural network model, a recurrent neural network model, etc., and then the target data is input into the preset learning model. The preset learning model will extract features of the target data, such as extracting vocabulary features, semantic features and other features in the text data. After completing the feature extraction, the preset learning model performs classification calculations based on the extracted features, and calculates the probability that the target data belongs to each preset category through the weights and bias parameters within the preset learning model, and selects the category with the highest probability as the data classification result.

[0076] This embodiment uses a preset learning model to classify target data without a subject, which can give full play to the advantages of machine learning and process complex and diverse data. The preset learning model can learn the potential patterns and rules in the data, and even in the absence of a clear subject, it can reasonably classify according to the characteristics of the target data, which can improve the accuracy and efficiency of data classification.

[0077] S106, when it is determined that the target data has a subject, determining whether the extracted subject is unique;

[0078] In this embodiment, first, when it is determined that the subject exists, all the subjects extracted from the full subject word library are obtained, and then the number of all the subjects is counted. If the counted number is 1, that is, only one subject matching the target data is found, then the subject is determined to be unique; if the statistical result is greater than 1, it indicates that multiple matching subjects are found, and at this time, the subject is determined to be not unique. For example, the extracted subjects are the "education policy and policy" subject and the "education policy" subject, and the number of the subjects is 2, which is greater than 1, so the subject is not unique. When the determination result is unique, step 107 is executed; if the determination result is not unique, step 108 is executed.

[0079] This embodiment helps to improve the accuracy of data classification by determining whether the subject is unique, avoids incorrectly determining the data category when the subject is not unique, and ensures that the data classification result is consistent with the actual connotation of the data, thereby making the entire data classification process more rigorous and scientific.

[0080] S107, taking the theme as the data classification result;

[0081] In this embodiment, when it is determined that the extracted topic is unique, the topic is extracted as a data classification result.

[0082] It should be noted that the unique subject needs to be standardized to ensure that the format and expression of the extracted subject meet the required standards.

[0083] This embodiment uses the unique subject as the data classification result, can accurately obtain the data classification result, reduces computing resources, and improves the efficiency of data classification.

[0084] S108, determining the department subject of the target data based on the department subject word library;

[0085] In this embodiment, when it is determined that the subject is not unique, the departmental subject word library is used to determine the departmental subject of the target data. Therefore, it is necessary to first take out the departmental subject words one by one from the departmental subject word library and match them with the target data. The matching method can be exact matching or fuzzy matching. In the matching process, since the departmental subject word library is more department-specific than the full subject word library, it will pay more attention to department-specific vocabulary and concepts. When a matching departmental subject word is found, it will be determined as the departmental subject of the target data.

[0086] In this embodiment, when the existing topics matched in the full subject vocabulary are not unique, the departmental topics of the target data are determined through the departmental subject vocabulary, which can more accurately determine the topics of the target data at the department level, help improve the classification accuracy of the target data, and enable the target data to be better associated with a specific department.

[0087] S109, when it is determined that the subject is not unique, determining whether the subject overlaps with the department subject;

[0088] In this embodiment, after the topics in the full subject vocabulary and the department topics in the department subject vocabulary have been determined, it is necessary to determine whether the topics and department topics overlap. First, the topics in the full subject vocabulary are extracted, and the department topics determined in the department subject vocabulary are also extracted. Then, the extracted topics and department topics are compared. If the topics and department topics completely overlap, step 110 is executed; if they are not exactly the same, step 111 is executed.

[0089] This embodiment determines whether the subject and the department subject overlap. If the subject and the department subject overlap, this indicates that there is commonality between the full subject and the department-specific subject, which helps to integrate data resources and can uniformly process and output overlapping data to avoid duplication. If there is no overlap, the boundary between the data under the full subject and the data under the department subject can be divided, which helps to perform more accurate classification storage and query operations.

[0090] S110, taking the subject or departmental subject as the data classification result;

[0091] In this embodiment, when it is determined that the subject overlaps with the department subject, the subject or department subject is used as the data classification result. The subject or department subject is determined as the final data classification result according to actual needs. For example, if the subject "market research data analysis" overlaps with the department subject "marketing department data statistical analysis", and it is determined according to actual needs that the department subject is used as the classification result, then the data classification result of the target data is determined as "marketing department data statistical analysis".

[0092] This embodiment uses overlapping topics or department topics as data classification results to optimize the data classification system, improve data availability and management efficiency, reduce unnecessary calculation steps, and obtain accurate data classification results quickly and easily.

[0093] S111. Classify the topics and department topics through a preset learning model to obtain data classification results.

[0094] In this embodiment, when it is determined that the topic and the department topic do not overlap, the preset learning model is called to classify the topic and the department topic to obtain the data classification result. First, the topic and the department topic need to be converted into vector form. The conversion process can use word embedding technology to map the topic words to a low-dimensional vector space. After the conversion is completed, the converted topic vector and department topic vector are input into the preset learning model. The preset learning model can be a classification model based on a neural network, which is not specifically limited. Finally, the preset learning model analyzes and classifies the input topic vector and department topic vector, and outputs the data classification result.

[0095] In this embodiment, when the subject and the department subject do not overlap, classification using a preset learning model can classify the data more scientifically and accurately. By converting the subject and the department subject into vector form and using an advanced learning model, the hidden semantic relationship and features between the subject and the department subject can be mined, thereby achieving a more refined and reasonable classification. The classification method based on the learning model is not subject to excessive interference from human factors and has high objectivity and stability.

[0096] See also Figure 2 , Figure 2 Another embodiment of a data classification method provided by the present application includes:

[0097] S201, obtaining data source information, identifying and removing duplicate data, redundant information and missing values ​​from the data source information;

[0098] In this embodiment, the data structure of the acquired data source information is first parsed to clarify the data storage format, field meaning and other information, and then the data in the data source information is marked with key identifiers, and duplicate data is identified through the marked key identifiers. For example, if the key identifier fields of two or more records are exactly the same, they are determined to be duplicate data and removed. For the identification of redundant information, each field in the data is analyzed according to pre-defined business rules. If a field contains a large number of meaningless placeholders, default values, etc., it is determined to be redundant information and removed. For missing values, it is necessary to traverse each field in the data source information and count the number of similar missing values ​​such as null values ​​and specific missing identifiers in each field. Fields with too high a missing value ratio will be removed, and fields with a low missing value ratio can be filled by methods such as mean filling, mode filling or predicted filling.

[0099] This embodiment removes duplicate data of data source information to avoid multiple unnecessary operations on the same data in subsequent data processing, reduces the waste of computing resources, and improves the efficiency of data processing; removing redundant information of data source information helps to streamline the data set, making the data more concise and clear, reduces the space occupied by data storage, and can highlight key information in the data, avoid interference from irrelevant information during the data analysis process, and improve the accuracy of data analysis; processing missing values ​​of data source information ensures the integrity of the data and avoids deviations in data analysis results due to missing values.

[0100] S202, unifying the data format of the data source information from which duplicate data, redundant information and missing values ​​have been removed;

[0101] In this embodiment, firstly, the data source information from which duplicate data, redundant information and missing values ​​have been removed is obtained, and then the data format is determined for the data source information, and the target format can be set according to the subsequent data processing requirements. After the data format is determined, the data source information from which duplicate data, redundant information and missing values ​​have been removed is traversed and checked. For numerical data, the number of decimal places, whether scientific notation is used and other formats are checked, and unified adjustments are made according to the target format, for example, all numerical values ​​are unified into a format that retains two decimal places; for date data, different formats are unified into a standard format, for example, "XXXX-XX-XX" is unified into "YYYY-MM-DD"; for text data, the character encoding is unified, for example, texts under different encodings are uniformly converted into UTF-8 encoding.

[0102] This embodiment unifies the data format, reduces the complexity of the storage structure and space waste caused by format differences, can avoid errors caused by format problems, reduce the time cost of data conversion due to format incompatibility, and improve the circulation and availability of data.

[0103] S203, extracting the data source information of the completed data format in a unified manner, and concatenating the data source information with a space character to obtain target data.

[0104] In this embodiment, it is first necessary to obtain the data source information that has completed the unified data format, perform an extraction operation on it, and then use a space character to perform a splicing operation on the extracted data source information. Furthermore, usually starting from the first element of the data, each element is combined with a space character in sequence. For example, after the data source information is processed, it contains three elements, namely "element 1", "element 2" and "element 3", then the result after splicing is "element 1 element 2 element 3", and the target data can be obtained after the splicing is completed.

[0105] This embodiment helps to integrate scattered and processed data source information into a coherent target data through extraction and splicing operations on data source information with unified data format, thereby simplifying the data structure. At the same time, the spliced ​​complete target data is easier to transmit than scattered data elements, reducing data fragmentation problems during transmission and improving transmission efficiency. The integrated target data is more convenient for overall analysis and interpretation, and the coherent target data can better reflect the relationship between the data, helping to more accurately mine the potential information in the data.

[0106] S204, obtaining a full thesaurus and a departmental thesaurus;

[0107] In this embodiment, step S204 is similar to step S102 in the above embodiment and will not be described again here.

[0108] S205: segment the target data, and input the segmented target data into the full subject word library to determine whether there is a subject.

[0109] In this embodiment, after obtaining the full subject vocabulary and department subject vocabulary, when the target data is segmented, if the target data is text data, it can be determined that the segmentation rules are based on punctuation marks and specific vocabulary boundaries. For example, a text type target data containing multiple sentences can be segmented into multiple clauses according to punctuation marks such as periods and commas. After the segmentation is completed, the segmented target data are input one by one into the full subject vocabulary for judgment.

[0110] In the judgment process, for each clause, the exact match or fuzzy match method can be used to compare with the subject in the full subject word library. The exact match requires that a certain word in the clause is exactly the same as the subject word; the fuzzy match can be calculated by methods such as semantic similarity. When the semantic similarity between the clause and the subject word reaches a certain threshold, it is considered that there is a subject match. For example, there is a "data analysis" subject in the full subject word library, and the clause after segmentation is "We are conducting data analysis work". Through exact matching or fuzzy matching, it can be judged that the clause has the "data analysis" subject. When it is determined to exist, step 206 is executed; when it is determined not to exist, step 207 is executed.

[0111] This embodiment divides the target data before performing topic judgment, which can improve the accuracy of topic judgment and avoid misjudgment due to missing target data. It uses a combination of exact matching and fuzzy matching to more comprehensively identify the topics in the target data.

[0112] S206, extracting topics from the full subject word library;

[0113] In this embodiment, step S206 is similar to step S104 in the above embodiment and will not be described in detail here.

[0114] S207, classifying the target data by using a preset learning model to obtain a data classification result;

[0115] In this embodiment, step S207 is similar to step S105 in the above embodiment and will not be described again here.

[0116] S208, when it is determined that the target data has a subject, determining whether the extracted subject is unique;

[0117] In this embodiment, step S208 is similar to step S106 in the above embodiment and will not be described again here.

[0118] S209, taking the theme as the data classification result;

[0119] In this embodiment, step S209 is similar to step S107 in the above embodiment and will not be described again here.

[0120] S210, when it is determined that the subject is not unique, judging whether the target data has a department subject based on the department subject word library;

[0121] In this embodiment, when it is determined that the subject is not unique, it is determined whether the target data has a department subject based on the department subject thesaurus. First, the segmented target data is obtained, and the segmented target data is compared with the subjects in the department subject thesaurus one by one. During the comparison process, an exact match or a fuzzy match can be used. An exact match is when the segmented target data is exactly the same as a subject in the department subject thesaurus, and then it is determined that a department subject exists; a fuzzy match is through the calculation of semantic similarity and other methods. When the semantic similarity between the segmented target data and the subject words in the department subject thesaurus reaches a set threshold, it is determined that a department subject match exists. When the judgment result is existence, step 211 is executed; when the judgment result is non-existence, step 212 is executed.

[0122] In this embodiment, when the subject is not unique, judging whether the target data has a department subject based on the department subject thesaurus is a more targeted processing method. The department subject thesaurus is constructed according to the business characteristics and needs of a specific department, and can more accurately reflect department-related data subjects. Through this type of judgment method, the relationship between the target data and the department subject can be further clarified, which helps to classify the target data more accurately.

[0123] S211, extracting departmental thesaurus from departmental thesaurus;

[0124] In this embodiment, when it is determined that the target data has a department subject, the department subject record that successfully matches the target data is first found in the department subject thesaurus, and the department subject in the department subject thesaurus is completely extracted based on the record. For example, if the "sales data statistics" in the department subject thesaurus matches the target data, then the department subject of "sales data statistics" is extracted.

[0125] This embodiment extracts departmental themes from the departmental thesaurus, provides a clear direction for subsequent data analysis, and provides more targeted subject resources, further increasing the accuracy of the entire data classification process.

[0126] S212. Classify the topics through a preset learning model to obtain data classification results.

[0127] In this embodiment, when it is determined that there is no department theme in the target data, the theme is directly classified through the preset learning model. Before classification, it is necessary to select a suitable preset learning model. The preset learning model can be a classification model based on a neural network. After selecting the preset learning model, the extracted theme that is not unique and does not match the department theme is provided as input to the preset learning model. The preset learning model will extract features from the input theme, and then analyze and predict the extracted features to obtain data classification results.

[0128] In this embodiment, when the target data cannot be classified by departmental subject, the classification method based on the preset learning model can handle complex subject relationships, is not limited to the pre-defined departmental subject vocabulary, can be classified according to features, and can dig out deeper relationships of the data, which helps to improve the comprehensiveness of data classification and ensure that all data can be reasonably classified and processed.

[0129] S213, when it is determined that the subject is not unique, determining whether the subject overlaps with the department subject;

[0130] In this embodiment, step S213 is similar to step S109 in the above embodiment and will not be described again here.

[0131] S214. Use the subject or departmental subject as the data classification result;

[0132] In this embodiment, step S214 is similar to step S110 in the above embodiment and will not be described again here.

[0133] S215. Classify the topics and department topics through a preset learning model to obtain data classification results.

[0134] In this embodiment, step S215 is similar to step S111 in the aforementioned embodiment and will not be described in detail here.

[0135] The data classification system provided by this application is described in detail below. Figure 3 , Figure 3 Another embodiment of the data classification system provided by the present application includes:

[0136] The first extraction unit 301 is used to obtain data source information, perform data extraction on the data source information, and obtain target data;

[0137] An acquisition unit 302 is used to acquire a full subject word library and a department subject word library;

[0138] The first judgment unit 303 is used to judge whether the target data has a theme based on the full theme word library;

[0139] The second extraction unit 304 is used to extract the subject from the full subject word library if yes;

[0140] The first classification unit 305 is used to classify the target data by using a preset learning model to obtain a data classification result if no;

[0141] The second judging unit 306 is used to judge whether the extracted subject is unique when it is determined that the target data has a subject;

[0142] The second classification unit 307 is used for taking the subject as the data classification result if yes;

[0143] A second determining unit 308 is used to determine the department theme of the target data based on the department theme word library if no;

[0144] The third judging unit 309 is used to judge whether the subject and the department subject overlap when it is determined that the subject is not unique;

[0145] The third classification unit 310 is used for taking the subject or department subject as the data classification result if yes;

[0146] The fourth classification unit 311 is used to classify the subject and department subject through a preset learning model to obtain a data classification result if no.

[0147] Optionally, the first extraction unit 301 is specifically used for:

[0148] Obtain data source information, perform data processing on the data source information, and extract the data source information after data processing to obtain target data.

[0149] Optionally, the first extraction unit 301 is specifically used for:

[0150] Obtain data source information, identify and remove duplicate data, redundant information and missing values ​​from data source information;

[0151] Unify the data format of data source information that has removed duplicate data, redundant information, and missing values;

[0152] Extract the data source information of the completed data format, and use space characters to splice the data source information to obtain the target data.

[0153] Optionally, the first judging unit 303 is specifically configured to:

[0154] The target data is segmented and input into the full subject word library to determine whether there is a topic.

[0155] Optionally, the second determining unit 308 is specifically configured to:

[0156] Determine whether the target data has departmental thesaurus based on the departmental thesaurus;

[0157] If so, determine the departmental themes of the target data;

[0158] If not, the topics are classified through the preset learning model to obtain the data classification results.

[0159] Optionally, the full thesaurus includes primary categories, secondary categories, and tertiary categories.

[0160] Optional, the first-level categories are e-government knowledge areas and departmental functional divisions and industry categories;

[0161] The secondary and tertiary categories are the branches of knowledge to which the subject terms in the full subject database belong.

[0162] This application also provides a data classification device, see Figure 4 , Figure 4 An embodiment of a data classification device provided by the present application includes:

[0163] Processor 401, memory 402, input and output unit 403, bus 404;

[0164] The processor 401 is connected to the memory 402, the input and output unit 403 and the bus 404;

[0165] The memory 402 stores a program, and the processor 401 calls the program to execute any of the above methods.

[0166] The present application also relates to a computer-readable storage medium on which a program is stored, wherein when the program is run on a computer, the computer is caused to execute any of the above methods.

[0167] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0168] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0169] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0170] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0171] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A data classification method, characterized in that: The method comprises: Acquire data source information, perform data extraction on the data source information, and obtain target data; Obtain full subject thesaurus and departmental subject thesaurus; Determining whether the target data has a subject based on the full subject word library; If yes, extract the subject from the full subject word library; If not, classify the target data using a preset learning model to obtain a data classification result; When it is determined that the subject exists in the target data, determining whether the extracted subject is unique; If yes, the subject is used as the data classification result; If not, determining the departmental subject of the target data based on the departmental subject word library; When it is determined that the subject is not unique, determining whether the subject is completely identical to the department subject; If yes, the subject or the department subject is used as the data classification result; If not, classify the subject and the department subject by using the preset learning model to obtain the data classification result; Determining the department subject of the target data based on the department subject word library includes: Determining whether the target data has a department theme based on the department theme word library; If so, determining the departmental subject of the target data; If not, the subject is classified by a preset learning model to obtain a data classification result.

2. The method according to claim 1, characterized in that The obtaining of data source information and extracting data from the data source information to obtain target data include: Acquire data source information, perform data processing on the data source information, and extract the data source information after data processing to obtain target data.

3. The method according to claim 2, characterized in that The acquiring of data source information, performing data processing on the data source information, and extracting the processed data source information to obtain target data includes: Obtain data source information, identify and remove duplicate data, redundant information and missing values ​​of the data source information; Unifying the data format of the data source information from which the duplicate data, the redundant information, and the missing values ​​have been removed; The data source information that has completed the data format unification is extracted, and the data source information is spliced ​​with space characters to obtain target data.

4. The method according to claim 1, characterized in that: The determining whether the target data has a subject based on the full subject word library includes: The target data is segmented, and the segmented target data is input into the full subject word library to determine whether there is a subject.

5. The method according to any one of claims 1 to 4, characterized in that The full subject word library includes primary categories, secondary categories and tertiary categories.

6. The method according to claim 5, characterized in that The first-level categories are the e-government knowledge domain, departmental division of labor and industry categories; The secondary category and the tertiary category are the knowledge branches to which the subject words in the full subject word library belong.

7. A data classification system, characterized in that: The system comprises: A first extraction unit is used to obtain data source information, perform data extraction on the data source information, and obtain target data; An acquisition unit is used to acquire the full subject word library and the departmental subject word library; A first judging unit, configured to judge whether the target data has a subject based on the full subject word library; A second extraction unit is used to extract the subject from the full subject word library when the subject exists; A first classification unit is used to classify the target data by using a preset learning model to obtain a data classification result when no subject exists; A second judging unit, configured to judge whether the extracted subject is unique when it is determined that the subject exists in the target data; A second classification unit, used for taking the subject as the data classification result when the subject is unique; A second determining unit, configured to determine a departmental subject of the target data based on the departmental subject word library when the subject is not unique; A third judging unit, configured to judge whether the subject is completely the same as the department subject when it is determined that the subject is not unique; A third classification unit, used for taking the subject or the department subject as the data classification result when it is determined that they are completely identical; A fourth classification unit, for classifying the subject and the department subject by using the preset learning model to obtain the data classification result when it is determined that they are not completely the same; The second determining unit is specifically configured to: Determining whether the target data has a department theme based on the department theme word library; If so, determining the departmental subject of the target data; If not, the subject is classified by a preset learning model to obtain a data classification result.

8. A data classification device, characterized in that: The device comprises: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a program stored thereon, wherein the program, when executed on a computer, performs the method according to any one of claims 1 to 6.

Images