An intelligent data grading and classification method and system based on data portraits

Abstract

The present application relates to the technical field of data governance, and particularly relates to an intelligent data grading and classifying method and system based on data portrait, which comprises the following steps: extracting field information from a database to be classified, and inputting a large language model to extract field semantic features; determining a classification system according to the field semantic features, and constructing a classification definition of the database to be classified; obtaining a table creation statement, extracting grading data from the database to be classified according to the table creation statement, grading the grading data according to the classification definition, and generating a grading data table corresponding to the table creation statement. In view of the problem that data grading and classifying rules are prone to failure, a large language model is introduced to identify the field information in the database, so as to obtain the field semantic features of the database. In the subsequent grading process, the large language model can be used to describe the role of each field data in the database, so that the model can accurately determine the category and importance of the related data, thereby obtaining the corresponding grading data table.

Description

Technical Field

[0001] This invention relates to the field of data governance technology, specifically to an intelligent data hierarchical classification method and system based on data profiling. Background Technology

[0002] Data, as a core resource of the digital economy and information society, is considered the fifth major factor of production after land, labor, capital, and technology, and holds significant value in enterprise digital transformation. With the acceleration of digitalization, the amount of data enterprises need to process is growing exponentially. Different types of data exhibit significant differences in sensitivity and value, requiring differentiated management based on their importance and value indices. Data classification and grading, as the foundation of data governance and security protection, aims to scientifically categorize data according to its sensitivity, value, and legal compliance requirements, and to formulate corresponding security control strategies accordingly.

[0003] In the existing technology, there are already corresponding data classification and grading schemes.

[0004] For example, application number CN202410845741.3 proposes a data classification and grading method, belonging to the field of data classification and grading technology. It includes: Step 1: The user uploads data to be classified to the server, selecting fields for which no specific data is provided; Step 2: The server obtains the data fields and content. If the data field does not contain template data, proceed to Step 3; otherwise, proceed to Step 4; Step 3: The data fields are scanned using a recognition model. If the output uniquely corresponds to a certain data in the template, the field level is determined according to the correspondence in the template; otherwise, the field is considered not to belong to the template range, and Step 4 is executed; Step 4: The data fields are scanned using a recognition model, and the output is matched with the data in the template to form a regular expression rule array, which represents the matching result; Step 5: The matching process is executed to obtain the classification level; a weighted matching function is proposed. This method can more accurately classify and grade data types.

[0005] For example, application number CN202311435688.1 discloses a data classification and grading system, comprising a data source, a data management unit, a system configuration unit, an asset discovery unit, and a display unit. Its key feature is that the data source is connected to clients via a network and a local area network, and the data source includes relational databases, big data platforms, cloud databases, and enterprise internal databases. The system configuration unit has a built-in discovery module that enables automatic classification and grading across multiple industries. Custom configuration templates allow users to create discovery templates, selecting the fields and business types to be discovered based on user-defined classification and grading standards. For example, users can select only specific sensitive data types for sensitive data discovery. During data discovery, users can customize the content to be discovered using discovery templates, thus greatly improving autonomy and applicability to different industries.

[0006] However, in actual implementation, the inventors found that this type of technical solution struggles to adapt flexibly to changes in business scenarios and complex semantic data, easily leading to misjudgments and omissions, thus affecting classification accuracy. Furthermore, as the scale of rules and the amount of data increase, the system's matching and reasoning efficiency decreases, limiting overall performance. Summary of the Invention

[0007] To address the aforementioned problems in existing technologies, a smart data hierarchical classification method based on data profiling is provided. On the other hand, a system that applies this intelligent data classification method is also provided.

[0008] The specific technical solution is as follows: A smart data hierarchical classification method based on data profiling includes: Step S1: Extract field information from the database to be classified and input it into a large language model to extract semantic features of the fields; Step S2: Determine the classification system based on the semantic features of the fields, and construct the classification definition corresponding to the database to be classified; Step S3: Obtain the table creation statement input by the user, extract the data to be classified from the database to be classified according to the table creation statement, classify the data to be classified according to the classification definition, and generate a classification data table corresponding to the table creation statement.

[0009] On the other hand, step S1 includes: Step S11: Extract the field information from the database to be classified, and input it into the large language model for cleaning, labeling fields with disordered naming; Step S12: Add unified field labels to the fields with inconsistent naming to form corrected field information; Step S13: Pre-identify the corrected field information to obtain the domain identification result, and then connect the large language model to the industry knowledge base corresponding to the domain identification result to extract the semantic features of the field from the corrected field information.

[0010] On the other hand, step S2 includes: Step S21: Find the corresponding semantic features of the identified fields for each table in the database to be classified, and match the corresponding tag system based on the semantic features of the fields; Step S22: Classify according to the label system to obtain the classification system, and construct classification labels corresponding to the table; Step S23: Generate the classification definition by semantic generation based on all the classification labels.

[0011] On the other hand, step S3 includes: Step S31: Extract the data to be classified according to the table creation statement, and identify the table creation statement to determine the affected objects and the degree of influence to generate classification criteria; Step S32: Match the corresponding classification method from the external database according to the classification definition, and generate model prompt words according to the classification definition; Step S33: Input the classification method, classification criteria, and model prompt words into the large language model, so that the large language model generates a hierarchical data table corresponding to the table creation statement.

[0012] On the other hand, step S31 includes: Step S311: Extract the data to be classified according to the table creation statement, and extract multiple data columns from the data to be classified; Step S312: For each data column, search for matching entity nouns from the data to be classified; Step S313: Match the external database with the found entity name to determine the affected object pointed to by the entity name, and determine the degree of influence corresponding to the entity name.

[0013] An intelligent data hierarchical classification system based on data profiling includes: The field semantic extraction module extracts field information from the database to be classified and inputs it into a large language model to extract field semantic features; A classification determination module, which is connected to the field semantic extraction module; The classification determination module determines the classification system based on the semantic features of the fields and constructs a classification definition corresponding to the database to be classified. A hierarchical table creation module, which is connected to the classification determination module; The hierarchical table creation module obtains the table creation statement input by the user, extracts the data to be classified from the database to be classified according to the table creation statement, classifies the data to be classified according to the classification definition, and generates a hierarchical data table corresponding to the table creation statement.

[0014] On the other hand, the field semantic extraction module includes: The field recognition module extracts the field information from the database to be classified and inputs it into the large language model for cleaning, labeling fields with disordered names; A unified field annotation module, which is connected to the field recognition module; The unified field annotation module adds unified field annotations to the fields with disordered naming, forming corrected field information; A semantic extraction module, which is connected to the unified field annotation module; The semantic extraction module pre-identifies the corrected field information to obtain a domain identification result, and then connects the large language model to an industry knowledge base corresponding to the domain identification result to extract the semantic features of the corrected field information.

[0015] On the other hand, the classification determination module includes: The tag system extraction module searches for the corresponding field semantic features identified in each table of the database to be classified, and matches the corresponding tag system based on the field semantic features. A label annotation module, which is connected to the label system extraction module; The labeling module classifies according to the label system to obtain the classification system, and constructs classification labels corresponding to the table; A category definition generation module, which is connected to the label annotation module; The classification definition generation module generates the classification definition semantically based on all the classification labels.

[0016] On the other hand, the hierarchical table creation module includes: The grading basis generation module extracts the data to be graded based on the table creation statement, and identifies the table creation statement to determine the affected objects and the degree of influence to generate grading basis. The prompt word generation module is connected to the hierarchical basis generation module; The prompt word generation module matches the corresponding classification method from an external database according to the classification definition, and generates model prompt words according to the classification definition; The data table hierarchical module inputs the classification method, classification criteria, and model prompt words into the large language model, thereby enabling the large language model to generate hierarchical data tables corresponding to the table creation statements.

[0017] On the other hand, the classification basis generation module includes: A data column extraction module extracts the data to be classified according to the table creation statement, and extracts multiple data columns from the data to be classified. An entity matching module, which is connected to the data column extraction module; The entity matching module searches for matching entity names in the data to be classified for each data column; An entity lookup module, which is connected to the entity matching module; The entity lookup module matches the found entity names with an external database to determine the affected object pointed to by the entity name, and to determine the degree of influence corresponding to the entity name.

[0018] The above technical solution has the following advantages or beneficial effects: To address the issue that data classification and grading rules can easily become ineffective due to the frequent changes in business scenarios in existing technologies, this solution introduces a large-scale language model to identify field information in the database, thereby obtaining the semantic features of the database fields. Based on this, the database type is classified, the corresponding classification system is determined, and a classification definition is used for description. In the subsequent grading process, the function of each field in the database can be described to the large-scale language model based on the classification definition, so that the model can accurately determine the category and importance of relevant data, thereby splitting it into corresponding graded data tables. Attached Figure Description

[0019] Embodiments of the invention will be described more fully with reference to the accompanying drawings. However, the drawings are for illustration and explanation only and do not constitute a limitation on the scope of the invention.

[0020] Figure 1 This is an overall schematic diagram of an embodiment of the present invention; Figure 2 This is a schematic diagram of step S1 in an embodiment of the present invention; Figure 3 This is a schematic diagram of step S2 in an embodiment of the present invention; Figure 4 This is a schematic diagram of step S3 in an embodiment of the present invention; Figure 5 This is a schematic diagram of step S31 in an embodiment of the present invention; Figure 6 This is a schematic diagram of the system in an embodiment of the present invention; Figure 7 This is a schematic diagram of the field semantic extraction module in an embodiment of the present invention; Figure 8 This is a schematic diagram of the classification determination module in an embodiment of the present invention; Figure 9 This is a schematic diagram of the hierarchical table creation module in an embodiment of the present invention; Figure 10 This is a schematic diagram of the hierarchical basis generation module in an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0023] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention.

[0024] This invention includes: A smart data hierarchical classification method based on data profiling, such as Figure 1 As shown, it includes: Step S1: Extract field information from the database to be classified and input it into a large language model to extract semantic features of the fields; Step S2: Determine the classification system based on the semantic features of the fields, and construct the classification definition corresponding to the database to be classified; Step S3: Obtain the table creation statement input by the user, extract the data to be classified from the database according to the table creation statement, classify the data to be classified according to the classification definition, and generate a classification data table corresponding to the table creation statement.

[0025] Specifically, addressing the issue that data classification and grading rules are prone to failure due to the frequent changes in business scenarios in existing technologies, this solution introduces a large-scale language model to identify field information in the database, thereby obtaining the semantic features of the database fields. Based on this, the database types are classified, the corresponding classification system is determined, and a classification definition is used for description. In the subsequent grading process, the function of each field in the database can be described to the large-scale language model based on the classification definition, so that the model can accurately determine the category and importance of the relevant data, thereby splitting it into corresponding graded data tables.

[0026] In actual implementation, the above-mentioned intelligent data classification method is mainly configured as a software implementation in corresponding computer devices, such as database systems, to perform corresponding classification processing on the data sources accessed by the database system, thereby adding corresponding labels to facilitate the use of subsequent business scenarios and the configuration of security permissions.

[0027] A database system refers to a system that connects to multiple database and data warehouse data sources, such as MySQL, Hive, Oracle, and SQL Server, and has certain data processing capabilities, enabling functions such as field recognition, metadata collection, and batch attribute maintenance.

[0028] In its initial state, users configure the corresponding data sources for the database system. The platform provides automatic capture, parsing, and storage of metadata from multiple data sources, supporting unified management of metadata. After the data source configuration is complete, the platform allows users to configure metadata collection tasks for each data source. These tasks can be run manually or scheduled to run on a timer, enabling automatic discovery and updating of metadata. The platform can also identify metadata in new data sources, providing foundational support for subsequent classification and grading. Simultaneously, it enables automatic subscription or capture of data, storing it in the database to be classified.

[0029] Subsequently, for the database to be classified, which contains multiple data tables, each data table is configured with a large number of data fields according to different business dimensions. The large language model extracts the field information generated by the data fields to perform semantic recognition, thereby realizing the understanding of the content contained in the database, the type of database, etc., as field semantic features.

[0030] A large-scale language model is a pre-trained and fine-tuned large-scale language model that can access a database system through a specific data interface to obtain and recognize field information from the database. To achieve better database classification results, this embodiment also pre-tunes the large-scale language model using instructions.

[0031] Leveraging fine-tuning interfaces supported by open-source and commercial large-scale models, a high-quality domain-specific fine-tuning dataset is constructed. This dataset is then used to train models for specific classification tasks, enabling them to maintain or improve classification accuracy while simplifying prompts. The fine-tuning data includes instructions, input samples, and corresponding accurate outputs, ensuring the model's efficient understanding and execution of complex rules. Furthermore, by incorporating domain expert-designed and optimized prompt words and structures, the prompt content is dynamically adjusted, allowing the large-scale model to more accurately understand classification rules and business context. Sensitivity levels are determined based on field information, enhancing the professionalism and stability of the model's response.

[0032] Based on the semantic features of fields identified by large-scale language models, semantic vectors can be further constructed from these features and matched with relevant external classification systems to determine the classification system that the current database should adopt. For example, in the scenario of public data hierarchical classification, the number of people affected and the granularity of data relative to individuals are usually used as classification criteria. On this basis, the database is described in conjunction with specific fields, explaining the classification definition of the database under the current classification system.

[0033] Finally, based on the user's input table creation statement, the data to be classified is extracted from the database to be classified. This part of the data to be classified is a specific column of a specific data table obtained by searching the database to be classified. This part of the data, together with the classification definition, is input into a large language model for classification, and corresponding hierarchical classification field labels are added to form a hierarchical data table.

[0034] In one embodiment, such as Figure 2 As shown, step S1 includes: Step S11: Extract field information from the database to be classified, and input it into a large language model for cleaning, labeling fields with disordered names; Step S12: Add unified field labels to fields with inconsistent naming to form corrected field information; Step S13: Pre-identify the corrected field information to obtain the domain identification result, and then connect the large language model to the industry knowledge base corresponding to the domain identification result to extract the semantic features of the corrected field information.

[0035] Specifically, to achieve better recognition results, in this embodiment, field information is first extracted from the database to be classified, and a certain number of sample data rows are extracted as samples and input into a large language model for cleaning. By inputting corresponding prompt words, the field information in multiple tables input to the large language model is cross-compared to determine whether there are fields with duplicate data but confusing names. Additional label fields are added to these fields with confusing names as identifiers to avoid confusion caused by different field names in the subsequent semantic recognition process of the large language model.

[0036] Subsequently, the corrected field information is pre-identified to obtain the domain identification result. This part can use a word segmentation model to perform simple word segmentation on the field names in the corrected field information, and then perform vector matching on the semantic identification dictionary, and count the number of field names matched in each dictionary to determine the closest dictionary and the corresponding industry domain. Then, a large language model is connected to the industry knowledge base corresponding to the domain identification result to extract the field semantic features from the corrected field information to achieve a better understanding of the semantics.

[0037] In one embodiment, such as Figure 3 As shown, step S2 includes: Step S21: Find the corresponding semantic features of the identified fields for each table in the database to be classified, and match the corresponding label system based on the semantic features of the fields; Step S22: Classify according to the label system to obtain the classification system, and construct the classification labels corresponding to the table; Step S23: Generate the classification definition by semantic generation based on all the classification labels.

[0038] Specifically, in order to achieve a better classification effect on the data table, in this embodiment, the semantic features of the fields returned by the large language model are first segmented, and the fields described in the segmentation results are used to point to the corresponding tables in the database to be classified.

[0039] Subsequently, for each table in the database to be classified, the corresponding semantic features of the identified fields are searched, and the corresponding tag system is matched based on the field semantic features. This tag system mainly consists of business tag categories pre-compiled for relevant industries, such as "ID" and "phone number". There is often a certain difference between these tags and the actual field names.

[0040] Based on this, a large-scale language model is used to classify the tag system and determine the function of the same tag in the corresponding table, such as "personal information" or "contact information". Category tags are then added to each data table.

[0041] After setting category labels for all tables, high-frequency words in the category labels are obtained through hot word generation, and a large language model is used to describe the high-frequency words to generate category definitions.

[0042] In one embodiment, such as Figure 4 As shown, step S3 includes: Step S31: Extract the data to be classified based on the table creation statement, and identify the table creation statement to determine the affected objects and the degree of influence to generate the classification basis; Step S32: Match the corresponding classification method from the external database according to the classification definition, and generate model prompt words according to the classification definition; Step S33: Input the classification method, classification criteria, and model prompt words into the large language model, so that the large language model generates a hierarchical data table corresponding to the table creation statement.

[0043] Specifically, to enable a large language model to effectively classify specific fields in a database table, this embodiment first extracts the data to be classified based on the table creation statement, mainly the data tables directly mentioned in the table creation statement. Then, the fields involved in the table creation statement are extracted, and combined with rules and a general knowledge base, the large language model identifies the fields, determines the objects and degree of influence of each field, and thus defines the classification criteria.

[0044] Simultaneously, the system matches the corresponding classification method from the external database according to the classification definition, generates model prompt words according to the classification definition, describes the current data table, and adds the desired classification result to the prompt words according to the classification method.

[0045] Finally, the classification method, classification criteria, and model prompts are input into a large language model, which then generates a hierarchical data table corresponding to the table creation statement.

[0046] In one embodiment, such as Figure 5 As shown, step S31 includes: Step S311: Extract the data to be classified according to the table creation statement, and extract multiple data columns from the data to be classified; Step S312: For each data column, search for matching entity names in the data to be classified; Step S313: Match the found entity names with the external database to determine the affected objects that the entity names point to, and to determine the degree of influence corresponding to the entity names.

[0047] Specifically, in order to achieve a better generation process for the classification criteria, in this embodiment, the data to be classified is first extracted according to the table creation statement, multiple data columns are extracted from the data to be classified, and for each data column, the corresponding field part is obtained, and then the field part is identified and the entity part contained in the field is extracted.

[0048] Based on this, the entity part is input into the industry knowledge base for matching according to the pre-matched industry knowledge base, so that the model can accurately understand the affected object and the degree of impact, thereby generating the classification basis.

[0049] An intelligent data hierarchical classification system based on data profiling, such as Figure 6 As shown, it includes: Field semantic extraction module 1 extracts field information from the database to be classified and inputs it into a large language model to extract field semantic features; Classification determination module 2 is connected to field semantic extraction module 1; The classification determination module 2 determines the classification system based on the semantic features of the fields and constructs the classification definition corresponding to the database to be classified; Hierarchical table creation module 3, which is connected to category determination module 2; The hierarchical table creation module 3 obtains the table creation statement input by the user, extracts the data to be classified from the database according to the table creation statement, classifies the data to be classified according to the classification definition, and generates a hierarchical data table corresponding to the table creation statement.

[0050] Specifically, addressing the issue that data classification and grading rules are prone to failure due to the frequent changes in business scenarios in existing technologies, this solution introduces a large-scale language model to identify field information in the database, thereby obtaining the semantic features of the database fields. Based on this, the database types are classified, the corresponding classification system is determined, and a classification definition is used for description. In the subsequent grading process, the function of each field in the database can be described to the large-scale language model based on the classification definition, so that the model can accurately determine the category and importance of the relevant data, thereby splitting it into corresponding graded data tables.

[0051] In one embodiment, such as Figure 7 As shown, the field semantic extraction module 1 includes: Field recognition module 11 extracts field information from the database to be classified, inputs it into a large language model for cleaning, and labels fields with disordered names; Unified field annotation module 12, which is connected to field recognition module 11; The unified field annotation module 12 adds unified field annotations to fields with inconsistent naming, forming corrected field information; Semantic extraction module 13, which is connected to unified field annotation module 12; The semantic extraction module 13 uses a large language model to pre-identify the corrected field information to obtain the domain identification result. Then, the large language model is connected to the industry knowledge base corresponding to the domain identification result to extract the semantic features of the corrected field information.

[0052] Specifically, to achieve better recognition results, in this embodiment, field information is first extracted from the database to be classified, and a certain number of sample data rows are extracted as samples and input into a large language model for cleaning. By inputting corresponding prompt words, the field information in multiple tables input to the large language model is cross-compared to determine whether there are fields with duplicate data but confusing names. Additional label fields are added to these fields with confusing names as identifiers to avoid confusion caused by different field names in the subsequent semantic recognition process of the large language model.

[0053] Subsequently, the corrected field information is pre-identified to obtain the domain identification result. This part can use a word segmentation model to perform simple word segmentation on the field names in the corrected field information, and then perform vector matching on the semantic identification dictionary, and count the number of field names matched in each dictionary to determine the closest dictionary and the corresponding industry domain. Then, a large language model is connected to the industry knowledge base corresponding to the domain identification result to extract the field semantic features from the corrected field information to achieve a better understanding of the semantics.

[0054] In one embodiment, such as Figure 8 As shown, the classification determination module 2 includes: The tag system extraction module 21 searches for the corresponding field semantic features identified in each table of the database to be classified, and matches the corresponding tag system based on the field semantic features. Label annotation module 22, which is connected to label system extraction module 21; The labeling module 22 classifies the data according to the label system to obtain a classification system and constructs classification labels corresponding to the table; Category definition generation module 23, which is connected to label annotation module 22; The category definition generation module 23 generates the category definition semantically based on all the category labels.

[0055] Specifically, in order to achieve a better classification effect on the data table, in this embodiment, the semantic features of the fields returned by the large language model are first segmented, and the fields described in the segmentation results are used to point to the corresponding tables in the database to be classified.

[0056] Subsequently, for each table in the database to be classified, the corresponding semantic features of the identified fields are searched, and the corresponding tag system is matched based on the field semantic features. This tag system mainly consists of business tag categories pre-compiled for relevant industries, such as "ID" and "phone number". There is often a certain difference between these tags and the actual field names.

[0057] Based on this, a large-scale language model is used to classify the tag system and determine the function of the same tag in the corresponding table, such as "personal information" or "contact information". Category tags are then added to each data table.

[0058] After setting category labels for all tables, high-frequency words in the category labels are obtained through hot word generation, and a large language model is used to describe the high-frequency words to generate category definitions.

[0059] In one embodiment, such as Figure 9 As shown, the hierarchical table creation module 3 includes: The grading basis generation module 31 extracts the data to be graded based on the table creation statement and identifies the table creation statement to determine the affected objects and the degree of influence to generate the grading basis. Prompt word generation module 32, which is connected to hierarchical basis generation module 31; The prompt word generation module 32 matches the corresponding classification method from the external database according to the classification definition, and generates model prompt words according to the classification definition; The data table hierarchical module 33 inputs the classification method, classification basis, and model prompt words into the large language model, thereby enabling the large language model to generate hierarchical data tables corresponding to the table creation statements.

[0060] Specifically, to enable a large language model to effectively classify specific fields in a database table, this embodiment first extracts the data to be classified based on the table creation statement, mainly the data tables directly mentioned in the table creation statement. Then, the fields involved in the table creation statement are extracted, and combined with rules and a general knowledge base, the large language model identifies the fields, determines the objects and degree of influence of each field, and thus defines the classification criteria.

[0061] Simultaneously, the system matches the corresponding classification method from the external database according to the classification definition, generates model prompt words according to the classification definition, describes the current data table, and adds the desired classification result to the prompt words according to the classification method.

[0062] Finally, the classification method, classification criteria, and model prompts are input into a large language model, which then generates a hierarchical data table corresponding to the table creation statement.

[0063] In one embodiment, such as Figure 10 As shown, the grading basis generation module 31 includes: The data column extraction module 311 extracts the data to be classified according to the table creation statement, and extracts multiple data columns from the data to be classified. Entity matching module 312, which is connected to data column extraction module 312; The entity matching module 312 searches for matching entity names in the data columns from the data to be classified. Entity lookup module 313, which is connected to entity matching module 312; The entity lookup module 313 matches the found entity names with the external database to determine the affected objects pointed to by the entity names and the degree of influence corresponding to the entity names.

[0064] Specifically, in order to achieve a better generation process for the classification criteria, in this embodiment, the data to be classified is first extracted according to the table creation statement, multiple data columns are extracted from the data to be classified, and for each data column, the corresponding field part is obtained, and then the field part is identified and the entity part contained in the field is extracted.

[0065] Based on this, the entity part is input into the industry knowledge base for matching according to the pre-matched industry knowledge base, so that the model can accurately understand the affected object and the degree of impact, thereby generating the classification basis.

[0066] The above are merely preferred embodiments of the present invention and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should recognize that any equivalent substitutions and obvious changes made based on the description and illustrations of the present invention should be included within the protection scope of the present invention.

Claims

1. A smart data hierarchical classification method based on data profiling, characterized in that, include: Step S1: Extract field information from the database to be classified and input it into a large language model to extract semantic features of the fields; Step S2: Determine the classification system based on the semantic features of the fields, and construct the classification definition corresponding to the database to be classified; Step S3: Obtain the table creation statement input by the user, extract the data to be classified from the database to be classified according to the table creation statement, classify the data to be classified according to the classification definition, and generate a classification data table corresponding to the table creation statement.

2. The intelligent data hierarchical classification method according to claim 1, characterized in that, Step S1 includes: Step S11: Extract the field information from the database to be classified, and input it into the large language model for cleaning, labeling fields with disordered naming; Step S12: Add unified field labels to the fields with inconsistent naming to form corrected field information; Step S13: Pre-identify the corrected field information to obtain the domain identification result, and then connect the large language model to the industry knowledge base corresponding to the domain identification result to extract the semantic features of the field from the corrected field information.

3. The intelligent data hierarchical classification method according to claim 1, characterized in that, Step S2 includes: Step S21: Find the corresponding semantic features of the identified fields for each table in the database to be classified, and match the corresponding tag system based on the semantic features of the fields; Step S22: Classify according to the label system to obtain the classification system, and construct classification labels corresponding to the table; Step S23: Generate the classification definition by semantic generation based on all the classification labels.

4. The intelligent data hierarchical classification method according to claim 1, characterized in that, Step S3 includes: Step S31: Extract the data to be classified according to the table creation statement, and identify the table creation statement to determine the affected objects and the degree of influence to generate classification criteria; Step S32: Match the corresponding classification method from the external database according to the classification definition, and generate model prompt words according to the classification definition; Step S33: Input the classification method, classification criteria, and model prompt words into the large language model, so that the large language model generates a hierarchical data table corresponding to the table creation statement.

5. The intelligent data hierarchical classification method according to claim 4, characterized in that, Step S31 includes: Step S311: Extract the data to be classified according to the table creation statement, and extract multiple data columns from the data to be classified; Step S312: For each data column, search for matching entity nouns from the data to be classified; Step S313: Match the external database with the found entity name to determine the affected object pointed to by the entity name, and determine the degree of influence corresponding to the entity name.

6. An intelligent data hierarchical classification system based on data profiling, characterized in that, Used to implement the intelligent data hierarchical classification method as described in any one of claims 1-5; The intelligent data hierarchical classification system includes: The field semantic extraction module extracts field information from the database to be classified and inputs it into a large language model to extract field semantic features; A classification determination module, which is connected to the field semantic extraction module; The classification determination module determines the classification system based on the semantic features of the fields and constructs a classification definition corresponding to the database to be classified. A hierarchical table creation module, which is connected to the classification determination module; The hierarchical table creation module obtains the table creation statement input by the user, extracts the data to be classified from the database to be classified according to the table creation statement, classifies the data to be classified according to the classification definition, and generates a hierarchical data table corresponding to the table creation statement.

7. The intelligent data hierarchical classification system according to claim 6, characterized in that, The field semantic extraction module includes: The field recognition module extracts the field information from the database to be classified and inputs it into the large language model for cleaning, labeling fields with disordered names; A unified field annotation module, which is connected to the field recognition module; The unified field annotation module adds unified field annotations to the fields with disordered naming, forming corrected field information; A semantic extraction module, which is connected to the unified field annotation module; The semantic extraction module pre-identifies the corrected field information to obtain a domain identification result, and then connects the large language model to an industry knowledge base corresponding to the domain identification result to extract the semantic features of the corrected field information.

8. The intelligent data hierarchical classification system according to claim 6, characterized in that, The classification determination module includes: The tag system extraction module searches for the corresponding field semantic features identified in each table of the database to be classified, and matches the corresponding tag system based on the field semantic features. A label annotation module is connected to the label system extraction module; The labeling module classifies according to the label system to obtain the classification system, and constructs classification labels corresponding to the table; A category definition generation module, which is connected to the label annotation module; The classification definition generation module generates the classification definition semantically based on all the classification labels.

9. The intelligent data hierarchical classification system according to claim 6, characterized in that, The hierarchical table creation module includes: The grading basis generation module extracts the data to be graded based on the table creation statement, and identifies the table creation statement to determine the affected objects and the degree of influence to generate grading basis. The prompt word generation module is connected to the hierarchical basis generation module; The prompt word generation module matches the corresponding classification method from an external database according to the classification definition, and generates model prompt words according to the classification definition; The data table hierarchical module inputs the classification method, classification criteria, and model prompt words into the large language model, thereby enabling the large language model to generate hierarchical data tables corresponding to the table creation statements.

10. The intelligent data hierarchical classification system according to claim 9, characterized in that, The classification basis generation module includes: A data column extraction module extracts the data to be classified according to the table creation statement, and extracts multiple data columns from the data to be classified. An entity matching module, which is connected to the data column extraction module; The entity matching module searches for matching entity names in the data to be classified for each data column; An entity lookup module, which is connected to the entity matching module; The entity lookup module matches the found entity names with an external database to determine the affected object pointed to by the entity name, and to determine the degree of influence corresponding to the entity name.

Images